Nondeterminism and Instability in Neural Network Optimization
arXiv:2103.04514
Abstract
Nondeterminism in neural network optimization produces uncertainty in performance, making small improvements difficult to discern from run-to-run variability. While uncertainty can be reduced by training multiple model copies, doing so is time-consuming, costly, and harms reproducibility. In this work, we establish an experimental protocol for understanding the effect of optimization nondeterminism on model diversity, allowing us to isolate the effects of a variety of sources of nondeterminism. Surprisingly, we find that all sources of nondeterminism have similar effects on measures of model diversity. To explain this intriguing fact, we identify the instability of model training, taken as an end-to-end procedure, as the key determinant. We show that even one-bit changes in initial parameters result in models converging to vastly different values. Last, we propose two approaches for reducing the effects of instability on run-to-run variability.
arXiv commentICML 2021
References in corpus (14)
- Adam: A Method for Stochastic Optimization
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- SGDR: Stochastic Gradient Descent with Warm Restarts
- HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
- cuDNN: Efficient Primitives for Deep Learning
- Regularizing and Optimizing LSTM Language Models
- Similarity of Neural Network Representations Revisited
- Deep Ensembles: A Loss Landscape Perspective
- Deep Reinforcement Learning that Matters
- Quasi-Recurrent Neural Networks
- Averaging Weights Leads to Wider Optima and Better Generalization
- Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control
- Critical Learning Periods in Deep Neural Networks