Predicting Training Time Without Training
arXiv:2008.12478
Abstract
We tackle the problem of predicting the number of optimization steps that a pre-trained deep network needs to converge to a given value of the loss function. To do so, we leverage the fact that the training dynamics of a deep network during fine-tuning are well approximated by those of a linearized model. This allows us to approximate the training loss and accuracy at any point during training by solving a low-dimensional Stochastic Differential Equation (SDE) in function space. Using this result, we are able to predict the time it takes for Stochastic Gradient Descent (SGD) to fine-tune a model to a given loss without having to perform any training. In our experiments, we are able to predict training time of a ResNet within a 20% error margin on a variety of datasets and hyper-parameters, at a 30 to 45-fold reduction in cost compared to actual training. We also discuss how to further reduce the computational and memory cost of our method, and in particular we show that by exploiting the spectral properties of the gradients' matrix it is possible predict training time on a large dataset while processing only a subset of the samples.
References in corpus (10)
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- Fine-Grained Visual Classification of Aircraft
- Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
- A Convergence Theory for Deep Learning via Over-Parameterization
- Deep Neural Networks as Gaussian Processes
- Gradient Descent Provably Optimizes Over-parameterized Neural Networks
- Convergence Analysis of Two-layer Neural Networks with ReLU Activation
- Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks
- Predicting the Computational Cost of Deep Learning Models
- Gradients as Features for Deep Representation Learning