Mixing ADAM and SGD: a Combined Optimization Method

Abstract

Optimization methods (optimizers) get special attention for the efficient training of neural networks in the field of deep learning. In literature there are many papers that compare neural models trained with the use of different optimizers. Each paper demonstrates that for a particular problem an optimizer is better than the others but as the problem changes this type of result is no longer valid and we have to start from scratch. In our paper we propose to use the combination of two very different optimizers but when used simultaneously they can overcome the performances of the single optimizers in very different problems. We propose a new optimizer called MAS (Mixing ADAM and SGD) that integrates SGD and ADAM simultaneously by weighing the contributions of both through the assignment of constant weights. Rather than trying to improve SGD or ADAM we exploit both at the same time by taking the best of both. We have conducted several experiments on images and text document classification, using various CNNs, and we demonstrated by experiments that the proposed MAS optimizer produces better performance than the single SGD or ADAM optimizers. The source code and all the results of the experiments are available online at the following link https://gitlab.com/nicolalandro/multi\_optimizer

References in corpus (9)

• Adam: A Method for Stochastic Optimization
• ADADELTA: An Adaptive Learning Rate Method
• On the Convergence of Adam and Beyond
• Resnet in Resnet: Generalizing Residual Architectures
• Improving Generalization Performance by Switching from Adam to SGD
• Adaptive Gradient Methods with Dynamic Bound of Learning Rate
• Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks
• Accelerating SGD with momentum for over-parameterized learning
• Evolutionary Stochastic Gradient Descent for Optimization of Deep Neural Networks

Cited by in corpus (5)

• A Framework Based on Symbolic Regression Coupled with eXtended Physics-Informed Neural Networks for Gray-Box Learning of Equations of Motion from Data
• Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers
• New Insight in Cervical Cancer Diagnosis Using Convolution Neural Network Architecture
• Machine Learning in Application to Automatic Noise Processing of Solar Spectrograms
• Weber-Fechner Law in Temporal Difference learning derived from Control as Inference