On the distance between two neural networks and the stability of learning
arXiv:2002.03432
Abstract
This paper relates parameter distance to gradient breakdown for a broad class of nonlinear compositional functions. The analysis leads to a new distance function called deep relative trust and a descent lemma for neural networks. Since the resulting learning rule seems to require little to no learning rate tuning, it may unlock a simpler workflow for training deeper and more complex neural networks. The Python code used in this paper is here: https://github.com/jxbz/fromage.
References in corpus (16)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
- On the difficulty of training Recurrent Neural Networks
- Large Scale GAN Training for High Fidelity Natural Image Synthesis
- Spectral Normalization for Generative Adversarial Networks
- HuggingFace's Transformers: State-of-the-art Natural Language Processing
- The Marginal Value of Adaptive Gradient Methods in Machine Learning
- Optimizing Neural Networks with Kronecker-factored Approximate Curvature
- A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks
- No More Pesky Learning Rates
- Path-SGD: Path-Normalized Optimization in Deep Neural Networks
- Optimization for deep learning: theory and algorithms
- Gradient Descent Can Take Exponential Time to Escape Saddle Points
- Competitive Gradient Descent
- Stochastic Mirror Descent on Overparameterized Nonlinear Models: Convergence, Implicit Regularization, and Generalization
- Stochastic Gradient/Mirror Descent: Minimax Optimality and Implicit Regularization
Cited by in corpus (8)
- High-Performance Large-Scale Image Recognition Without Normalization
- AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients
- Why Spectral Normalization Stabilizes GANs: Analysis and Improvements
- Multiplicative update rules for accelerating deep learning training and increasing robustness
- Learning compositional functions via multiplicative weight updates
- Learning by Turning: Neural Architecture Aware Optimisation
- Solving hybrid machine learning tasks by traversing weight space geodesics
- Online-Learning Deep Neuro-Adaptive Dynamic Inversion Controller for Model Free Control