Gravity Optimizer: a Kinematic Approach on Optimization in Deep Learning
arXiv:2101.09192
Abstract
We introduce Gravity, another algorithm for gradient-based optimization. In this paper, we explain how our novel idea change parameters to reduce the deep learning model's loss. It has three intuitive hyper-parameters that the best values for them are proposed. Also, we propose an alternative to moving average. To compare the performance of the Gravity optimizer with two common optimizers, Adam and RMSProp, five standard datasets were trained on two VGGNet models with a batch size of 128 for 100 epochs. Gravity hyper-parameters did not need to be tuned for different models. As will be explained more in the paper, to investigate the direct impact of the optimizer itself on loss reduction no overfitting prevention technique was used. The obtained results show that the Gravity optimizer has more stable performance than Adam and RMSProp and gives greater values of validation accuracy for datasets with more output classes like CIFAR-100 (Fine).
References in corpus (16)
- Adam: A Method for Stochastic Optimization
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- ADADELTA: An Adaptive Learning Rate Method
- On the Convergence of Adam and Beyond
- The Marginal Value of Adaptive Gradient Methods in Machine Learning
- A Comparison of Optimization Algorithms for Deep Learning
- On Empirical Comparisons of Optimizers for Deep Learning
- How Does Learning Rate Decay Help Modern Neural Networks?
- Adafactor: Adaptive Learning Rates with Sublinear Memory Cost
- Equilibrated adaptive learning rates for non-convex optimization
- Variants of RMSProp and Adagrad with Logarithmic Regret Bounds
- Convergence guarantees for RMSProp and ADAM in non-convex optimization and an empirical comparison to Nesterov acceleration
- Nostalgic Adam: Weighting more of the past gradients when designing the adaptive learning rate
- Eve: A Gradient Based Optimization Method with Locally and Globally Adaptive Learning Rates
- DeepOBS: A Deep Learning Optimizer Benchmark Suite