Online Batch Selection for Faster Training of Neural Networks

Abstract

Deep neural networks are commonly trained using stochastic non-convex optimization procedures, which are driven by gradient information estimated on fractions (batches) of the dataset. While it is commonly accepted that batch size is an important parameter for offline tuning, the benefits of online selection of batches remain poorly understood. We investigate online batch selection strategies for two state-of-the-art methods of stochastic gradient-based optimization, AdaDelta and Adam. As the loss function to be minimized for the whole dataset is an aggregation of loss functions of individual datapoints, intuitively, datapoints with the greatest loss should be considered (selected in a batch) more frequently. However, the limitations of this intuition and the proper control of the selection pressure over time are open questions. We propose a simple strategy where all datapoints are ranked w.r.t. their latest known loss value and the probability to be selected decays exponentially as a function of rank. Our experimental results on the MNIST dataset suggest that selecting batches speeds up both AdaDelta and Adam by a factor of about 5.

Workshop paper at ICLR 2016

References in corpus (8)

• Practical Bayesian Optimization of Machine Learning Algorithms
• ADADELTA: An Adaptive Learning Rate Method
• Prioritized Experience Replay
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• The Loss Surfaces of Multilayer Networks
• Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
• Variance Reduction in SGD by Distributed Importance Sampling
• Stop Wasting My Gradients: Practical SVRG

Cited by in corpus (59)

• Distributed Prioritized Experience Replay
• Recent Advances in Convolutional Neural Networks
• CMA-ES for Hyperparameter Optimization of Deep Neural Networks
• Regularization for Deep Learning: A Taxonomy
• Bridging Category-level and Instance-level Semantic Image Segmentation
• Active Bias: Training More Accurate Neural Networks by Emphasizing High Variance Samples
• Self-supervised learning of a facial attribute embedding from video
• Not All Samples Are Created Equal: Deep Learning with Importance Sampling
• Optimal Client Sampling for Federated Learning
• DynNet: Physics-based neural architecture design for linear and nonlinear structural response modeling and prediction
• High-performance Semantic Segmentation Using Very Deep Fully Convolutional Networks
• Transfer Learning via Contextual Invariants for One-to-Many Cross-Domain Recommendation
• Learning What Data to Learn
• Biased Importance Sampling for Deep Neural Network Training
• FairBatch: Batch Selection for Model Fairness
• Accelerating Deep Learning by Focusing on the Biggest Losers
• ScreenerNet: Learning Self-Paced Curriculum for Deep Neural Networks
• C-RPNs: Promoting Object Detection in real world via a Cascade Structure of Region Proposal Networks
• An Equivalence between Loss Functions and Non-Uniform Sampling in Experience Replay
• Deep Q-Networks for Accelerating the Training of Deep Neural Networks
• Unsupervised Hard Example Mining from Videos for Improved Object Detection
• A Survey on Curriculum Learning
• Adaptive Sketch-and-Project Methods for Solving Linear Systems
• Optimizing Region Selection for Weakly Supervised Object Detection
• Derivative Manipulation for General Example Weighting
• SampleAhead: Online Classifier-Sampler Communication for Learning from Synthesized Data
• A-Fast-RCNN: Hard Positive Generation via Adversary for Object Detection
• Accelerating Minibatch Stochastic Gradient Descent using Typicality Sampling
• Distributionally Robust Deep Learning using Hardness Weighted Sampling
• Robust Face Detection via Learning Small Faces on Hard Images
• BulletTrain: Accelerating Robust Neural Network Training via Boundary Example Mining
• Self-Paced Learning with Adaptive Deep Visual Embeddings
• Adaptive Task Sampling for Meta-Learning
• Training Multi-organ Segmentation Networks with Sample Selection by Relaxed Upper Confident Bound
• Which Strategies Matter for Noisy Label Classification? Insight into Loss and Uncertainty
• Monocular Real-Time Volumetric Performance Capture
• Approximate Newton-based statistical inference using only stochastic gradients
• Stochastic Reweighted Gradient Descent
• Two-Stream Region Convolutional 3D Network for Temporal Activity Detection
• Modern Subsampling Methods for Large-Scale Least Squares Regression
• Locally Adaptive Label Smoothing for Predictive Churn
• STADB: A Self-Thresholding Attention Guided ADB Network for Person Re-identification
• When does loss-based prioritization fail?
• Churn Reduction via Distillation
• Margin-Based Regularization and Selective Sampling in Deep Neural Networks
• Improved Hard Example Mining by Discovering Attribute-based Hard Person Identity
• A Simple Non-i.i.d. Sampling Approach for Efficient Training and Better Generalization
• Skeleton-Based Action Recognition with Synchronous Local and Non-local Spatio-temporal Learning and Frequency Attention
• Selective sampling for accelerating training of deep neural networks
• Using Wavelets to Analyze Similarities in Image-Classification Datasets
• A Tale Of Two Long Tails
• One Backward from Ten Forward, Subsampling for Large-Scale Deep Learning
• Statistical Measures For Defining Curriculum Scoring Function
• Submodular Batch Selection for Training Deep Neural Networks
• Carpe Diem, Seize the Samples Uncertain "At the Moment" for Adaptive Batch Selection
• Unsupervised Difficulty Estimation with Action Scores
• Efficient Reinforcement Learning in Resource Allocation Problems Through Permutation Invariant Multi-task Learning
• Large Batch Experience Replay
• Distractor-Aware Neuron Intrinsic Learning for Generic 2D Medical Image Classifications