Augment your batch: better training with larger batches

Abstract

Large-batch SGD is important for scaling training of deep neural networks. However, without fine-tuning hyperparameter schedules, the generalization of the model may be hampered. We propose to use batch augmentation: replicating instances of samples within the same batch with different data augmentations. Batch augmentation acts as a regularizer and an accelerator, increasing both generalization and performance scaling. We analyze the effect of batch augmentation on gradient variance and show that it empirically improves convergence for a wide variety of deep neural networks and datasets. Our results show that batch augmentation reduces the number of necessary SGD updates to achieve the same accuracy as the state-of-the-art. Overall, this simple yet effective method enables faster training and better generalization by allowing more computational resources to be used concurrently.

References in corpus (20)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• mixup: Beyond Empirical Risk Minimization
• Improved Regularization of Convolutional Neural Networks with Cutout
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Wide Residual Networks
• Data Augmentation Generative Adversarial Networks
• AutoAugment: Learning Augmentation Policies from Data
• Horovod: fast and easy distributed deep learning in TensorFlow
• Large Batch Training of Convolutional Networks
• Regularizing and Optimizing LSTM Language Models
• Train longer, generalize better: closing the generalization gap in large batch training of neural networks
• Revisiting Small Batch Training for Deep Neural Networks
• Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes
• Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis
• Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations
• Measuring the Effects of Data Parallelism on Neural Network Training
• Image Classification at Supercomputer Scale
• Massively Distributed SGD: ImageNet/ResNet-50 Training in a Flash
• Large-Scale Distributed Second-Order Optimization Using Kronecker-Factored Approximate Curvature for Deep Convolutional Neural Networks

Cited by in corpus (22)

• Avoiding Overfitting: A Survey on Regularization Methods for Convolutional Neural Networks
• Early Convolutions Help Transformers See Better
• Hardware and Software Optimizations for Accelerating Deep Neural Networks: Survey of Current Trends, Challenges, and the Road Ahead
• Review: Deep Learning in Electron Microscopy
• ECAPA-TDNN Embeddings for Speaker Diarization
• ResizeMix: Mixing Data with Preserved Object Information and True Labels
• UniformAugment: A Search-free Probabilistic Data Augmentation Approach
• Learning Loss for Test-Time Augmentation
• MultiGrain: a unified image embedding for classes and instances
• Faster Neural Network Training with Data Echoing
• AutoFormer: Searching Transformers for Visual Recognition
• Data augmentation in Bayesian neural networks and the cold posterior effect
• Stochastic Optimization with Laggard Data Pipelines
• Learning by Cheating
• Drawing Multiple Augmentation Samples Per Image During Training Efficiently Decreases Test Error
• Does Data Augmentation Benefit from Split BatchNorms
• Data augmentation and image understanding
• Improved generalization by noise enhancement
• Patient ADE Risk Prediction through Hierarchical Time-Aware Neural Network Using Claim Codes
• Social Influence Prediction with Train and Test Time Augmentation for Graph Neural Networks
• Stochastic Batch Augmentation with An Effective Distilled Dynamic Soft Label Regularizer
• Topologically Densified Distributions