Gravity Optimizer: a Kinematic Approach on Optimization in Deep Learning
arXiv:2101.09192
Abstract
We introduce Gravity, another algorithm for gradient-based optimization. In this paper, we explain how our novel idea change parameters to reduce the deep learning model's loss. It has three intuitive hyper-parameters that the best values for them are proposed. Also, we propose an alternative to moving average. To compare the performance of the Gravity optimizer with two common optimizers, Adam and RMSProp, five standard datasets were trained on two VGGNet models with a batch size of 128 for 100 epochs. Gravity hyper-parameters did not need to be tuned for different models. As will be explained more in the paper, to investigate the direct impact of the optimizer itself on loss reduction no overfitting prevention technique was used. The obtained results show that the Gravity optimizer has more stable performance than Adam and RMSProp and gives greater values of validation accuracy for datasets with more output classes like CIFAR-100 (Fine).
References in corpus (8)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- ADADELTA: An Adaptive Learning Rate Method
- On the Convergence of Adam and Beyond
- A Comparison of Optimization Algorithms for Deep Learning
- How Does Learning Rate Decay Help Modern Neural Networks?
- Variants of RMSProp and Adagrad with Logarithmic Regret Bounds
- DeepOBS: A Deep Learning Optimizer Benchmark Suite