Neurons learn slower than they think

Abstract

Recent studies revealed complex convergence dynamics in gradient-based methods, which has been little understood so far. Changing the step size to balance between high convergence rate and small generalization error may not be sufficient: maximizing the test accuracy usually requires a larger learning rate than minimizing the training loss. To explore the dynamic bounds of convergence rate, this study introduces \textit{differential capability} into an optimization process, which measures whether the test accuracy increases as fast as a model approaches the decision boundary in a classification problem. The convergence analysis showed that: 1) a higher convergence rate leads to slower capability growth; 2) a lower convergence rate results in faster capability growth and decay; 3) regulating a convergence rate in either direction reduces differential capability.

arXiv comment7 pages, 3 figures

References in corpus (6)

• Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
• Adaptive Gradient Methods with Dynamic Bound of Learning Rate
• Overparameterized Nonlinear Learning: Gradient Descent Takes the Shortest Path?
• Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability
• Can Single Neurons Solve MNIST? The Computational Power of Biological Dendritic Trees
• ${\beta }^3$-IRT: A New Item Response Model and its Applications