Gradients are Not All You Need
arXiv:2111.05803
Abstract
Differentiable programming techniques are widely used in the community and are responsible for the machine learning renaissance of the past several decades. While these methods are powerful, they have limits. In this short report, we discuss a common chaos based failure mode which appears in a variety of differentiable circumstances, ranging from recurrent neural networks and numerical physics simulation to training learned optimizers. We trace this failure to the spectrum of the Jacobian of the system under study, and provide criteria for when a practitioner might expect this failure to spoil their differentiation based optimization algorithms.
References in corpus (23)
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- On the difficulty of training Recurrent Neural Networks
- Convolutional Sequence to Sequence Learning
- Jamming at Zero Temperature and Zero Applied Stress: the Epitome of Disorder
- Machine learning accelerated computational fluid dynamics
- Evolution Strategies as a Scalable Alternative to Reinforcement Learning
- A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
- Training Deep Nets with Sublinear Memory Cost
- Regularizing and Optimizing LSTM Language Models
- Deep Equilibrium Models
- Neural GPUs Learn Algorithms
- Meta-Learning with Implicit Gradients
- Differentiable Rendering: A Survey
- Learned Optimizers that Scale and Generalize
- Unbiasing Truncated Backpropagation Through Time
- Learning Gradient Descent: Better Generalization and Longer Horizons
- Understanding and correcting pathologies in the training of learned optimizers
- Understanding Short-Horizon Bias in Stochastic Meta-Optimization
- Efficient and Modular Implicit Differentiation
- Fast and Feature-Complete Differentiable Physics for Articulated Rigid Bodies with Contact
- Deep Learning without Poor Local Minima
- Disentangling Trainability and Generalization in Deep Neural Networks
- Deluca -- A Differentiable Control Library: Environments, Methods, and Benchmarking