Painless step size adaptation for SGD

Abstract

Convergence and generalization are two crucial aspects of performance in neural networks. When analyzed separately, these properties may lead to contradictory results. Optimizing a convergence rate yields fast training, but does not guarantee the best generalization error. To avoid the conflict, recent studies suggest adopting a moderately large step size for optimizers, but the added value on the performance remains unclear. We propose the LIGHT function with the four configurations which regulate explicitly an improvement in convergence and generalization on testing. This contribution allows to: 1) improve both convergence and generalization of neural networks with no need to guarantee their stability; 2) build more reliable and explainable network architectures with no need for overparameterization. We refer to it as "painless" step size adaptation.

arXiv commentThis work has been submitted to the IEEE for possible publication

References in corpus (17)

• ADADELTA: An Adaptive Learning Rate Method
• A Convergence Theory for Deep Learning via Over-Parameterization
• The Marginal Value of Adaptive Gradient Methods in Machine Learning
• Train longer, generalize better: closing the generalization gap in large batch training of neural networks
• Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
• Characterizing Implicit Bias in Terms of Optimization Geometry
• In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning
• On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization
• A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks
• Overparameterized Nonlinear Learning: Gradient Descent Takes the Shortest Path?
• Accelerating SGD with momentum for over-parameterized learning
• Stochastic Learning under Random Reshuffling with Constant Step-sizes
• Deep Information Propagation
• Optimal Rates for Averaged Stochastic Gradient Descent under Neural Tangent Kernel Regime
• Agnostic Learning of a Single Neuron with Gradient Descent
• Quantitative Propagation of Chaos for SGD in Wide Neural Networks
• Can Single Neurons Solve MNIST? The Computational Power of Biological Dendritic Trees