CoRe Optimizer: An All-in-One Solution for Machine Learning

Abstract

The optimization algorithm and its hyperparameters can significantly affect the training speed and resulting model accuracy in machine learning applications. The wish list for an ideal optimizer includes fast and smooth convergence to low error, low computational demand, and general applicability. Our recently introduced continual resilient (CoRe) optimizer has shown superior performance compared to other state-of-the-art first-order gradient-based optimizers for training lifelong machine learning potentials. In this work we provide an extensive performance comparison of the CoRe optimizer and nine other optimization algorithms including the Adam optimizer and resilient backpropagation (RPROP) for diverse machine learning tasks. We analyze the influence of different hyperparameters and provide generally applicable values. The CoRe optimizer yields best or competitive performance in every investigated application, while only one hyperparameter needs to be changed depending on mini-batch or batch learning.

arXiv comment12 pages, 5 figures, 1 table

References in corpus (15)

• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Deep Learning using Rectified Linear Units (ReLU)
• Machine learning for molecular simulation
• Machine Learning Unifies the Modelling of Materials and Molecules
• OpenAI Gym
• Perspective on integrating machine learning into computational chemistry and materials science
• Symbolic Discovery of Optimization Algorithms
• Neural Network Potentials for Chemistry: Concepts, Applications and Prospects
• Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models
• Insights into lithium manganese oxide-water interfaces using machine learning potentials
• Lifelong Machine Learning Potentials
• A Survey of Optimization Methods from a Machine Learning Perspective
• Closing the gap between theory and experiment for lithium manganese oxide spinels using a high-dimensional neural network potential
• Predicting Oxidation and Spin States by High-Dimensional Neural Networks: Applications to Lithium Manganese Oxide Spinels
• Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training

Cited by in corpus (5)

• Machine Learning Enhanced Calculation of Quantum-Classical Binding Free Energies
• Hierarchical quantum embedding by machine learning for large molecular assemblies
• Lifelong Machine Learning Potentials for Chemical Reaction Network Explorations
• Modal Backflow Neural Quantum States for Anharmonic Vibrational Calculations
• Solving intractable chemical problems by tensor decomposition