How Low Can We Go: Trading Memory for Error in Low-Precision Training

Abstract

Low-precision arithmetic trains deep learning models using less energy, less memory and less time. However, we pay a price for the savings: lower precision may yield larger round-off error and hence larger prediction error. As applications proliferate, users must choose which precision to use to train a new model, and chip manufacturers must decide which precisions to manufacture. We view these precision choices as a hyperparameter tuning problem, and borrow ideas from meta-learning to learn the tradeoff between memory and error. In this paper, we introduce Pareto Estimation to Pick the Perfect Precision (PEPPP). We use matrix factorization to find non-dominated configurations (the Pareto frontier) with a limited number of network evaluations. For any given memory budget, the precision that minimizes error is a point on this frontier. Practitioners can use the frontier to trade memory for error and choose the best precision for their goals.

arXiv commentICLR 2022

References in corpus (12)

• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Mixed Precision Training
• Templates for Convex Cone Problems with Applications to Sparse Signal Recovery
• A Tutorial on Bayesian Optimization
• Training deep neural networks with low precision multiplications
• Meta-Learning: A Survey
• OBOE: Collaborative Filtering for AutoML Model Selection
• High-Accuracy Low-Precision Training
• Probabilistic Matrix Factorization for Automated Machine Learning
• Low-Memory Neural Network Training: A Technical Report
• Missing Not at Random in Matrix Completion: The Effectiveness of Estimating Missingness Probabilities Under a Low Nuclear Norm Assumption
• The ZipML Framework for Training Models with End-to-End Low Precision: The Cans, the Cannots, and a Little Bit of Deep Learning