One Backward from Ten Forward, Subsampling for Large-Scale Deep Learning

Abstract

Deep learning models in large-scale machine learning systems are often continuously trained with enormous data from production environments. The sheer volume of streaming training data poses a significant challenge to real-time training subsystems and ad-hoc sampling is the standard practice. Our key insight is that these deployed ML systems continuously perform forward passes on data instances during inference, but ad-hoc sampling does not take advantage of this substantial computational effort. Therefore, we propose to record a constant amount of information per instance from these forward passes. The extra information measurably improves the selection of which data instances should participate in forward and backward passes. A novel optimization framework is proposed to analyze this problem and we provide an efficient approximation algorithm under the framework of Mini-batch gradient descent as a practical solution. We also demonstrate the effectiveness of our framework and algorithm on several large-scale classification and regression tasks, when compared with competitive baselines widely used in industry.

arXiv comment13 pages

References in corpus (15)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• An overview of gradient descent optimization algorithms
• MobileNetV2: Inverted Residuals and Linear Bottlenecks
• Learning Tree-based Deep Model for Recommender Systems
• Online Batch Selection for Faster Training of Neural Networks
• Coresets for Scalable Bayesian Logistic Regression
• Coresets via Bilevel Optimization for Continual Learning and Streaming
• Variance Reduction in SGD by Distributed Importance Sampling
• Automated Scalable Bayesian Inference via Hilbert Coresets
• Bayesian Batch Active Learning as Sparse Subset Approximation
• Are All Training Examples Created Equal? An Empirical Study
• Accelerating Deep Learning by Focusing on the Biggest Losers
• Fast and Robust Least Squares Estimation in Corrupted Linear Models
• Optimal Subsampling Approaches for Large Sample Linear Regression
• Choosing the Sample with Lowest Loss makes SGD Robust