On Warm-Starting Neural Network Training

Abstract

In many real-world deployments of machine learning systems, data arrive piecemeal. These learning scenarios may be passive, where data arrive incrementally due to structural properties of the problem (e.g., daily financial data) or active, where samples are selected according to a measure of their quality (e.g., experimental design). In both of these cases, we are building a sequence of models that incorporate an increasing amount of data. We would like each of these models in the sequence to be performant and take advantage of all the data that are available to that point. Conventional intuition suggests that when solving a sequence of related optimization problems of this form, it should be possible to initialize using the solution of the previous iterate -- to "warm start" the optimization rather than initialize from scratch -- and see reductions in wall-clock time. However, in practice this warm-starting seems to yield poorer generalization performance than models that have fresh random initializations, even though the final training losses are similar. While it appears that some hyperparameter settings allow a practitioner to close this generalization gap, they seem to only do so in regimes that damage the wall-clock gains of the warm start. Nevertheless, it is highly desirable to be able to warm-start neural network training, as it would dramatically reduce the resource usage associated with the construction of performant deep learning systems. In this work, we take a closer look at this empirical phenomenon and try to understand when and how it occurs. We also provide a surprisingly simple trick that overcomes this pathology in several important situations, and present experiments that elucidate some of its properties.

References in corpus (14)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Prototypical Networks for Few-shot Learning
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Training Very Deep Networks
• Optimizing Neural Networks with Kronecker-factored Approximate Curvature
• Sensitivity and Generalization in Neural Networks: an Empirical Study
• Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Bounds
• Fisher-Rao Metric, Geometry, and Complexity of Neural Networks
• Gradient Surgery for Multi-Task Learning
• Non-Vacuous Generalization Bounds at the ImageNet Scale: A PAC-Bayesian Compression Approach
• Model-Based Active Exploration
• Generalization in Deep Networks: The Role of Distance from Initialization
• An Analysis of Pre-Training on Object Detection
• Unsupervised Domain Adaptation Using Approximate Label Matching

Cited by in corpus (12)

• Diminishing Uncertainty within the Training Pool: Active Learning for Medical Image Segmentation
• Predicting Good Quantum Circuit Compilation Options
• Architectural Vision for Quantum Computing in the Edge-Cloud Continuum
• Warm-Starting and Quantum Computing: A Systematic Mapping Study
• Augmenting GAIL with BC for sample efficient imitation learning
• Cold-start Active Learning through Self-supervised Language Modeling
• A Priori Uncertainty Quantification of Reacting Turbulence Closure Models using Bayesian Neural Networks
• Deep Neural Implicit Representation of Accessibility for Multi-Axis Manufacturing
• Churn Reduction via Distillation
• ZORB: A Derivative-Free Backpropagation Algorithm for Neural Networks
• A study on the plasticity of neural networks
• Reducing the Long Tail Losses in Scientific Emulations with Active Learning