Data Augmentation Revisited: Rethinking the Distribution Gap between Clean and Augmented Data

Abstract

Data augmentation has been widely applied as an effective methodology to improve generalization in particular when training deep neural networks. Recently, researchers proposed a few intensive data augmentation techniques, which indeed improved accuracy, yet we notice that these methods augment data have also caused a considerable gap between clean and augmented data. In this paper, we revisit this problem from an analytical perspective, for which we estimate the upper-bound of expected risk using two terms, namely, empirical risk and generalization error, respectively. We develop an understanding of data augmentation as regularization, which highlights the major features. As a result, data augmentation significantly reduces the generalization error, but meanwhile leads to a slightly higher empirical risk. On the assumption that data augmentation helps models converge to a better region, the model can benefit from a lower empirical risk achieved by a simple method, i.e., using less-augmented data to refine the model trained on fully-augmented data. Our approach achieves consistent accuracy gain on a few standard image classification benchmarks, and the gain transfers to object detection.

References in corpus (16)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Improved Regularization of Convolutional Neural Networks with Cutout
• ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness
• Manifold Mixup: Better Representations by Interpolating Hidden States
• Smart Augmentation - Learning an Optimal Data Augmentation Strategy
• Improving Generalization Performance by Switching from Adam to SGD
• Shake-Shake regularization
• Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet
• Population Based Augmentation: Efficient Learning of Augmentation Policy Schedules
• A Bayesian Data Augmentation Approach for Learning Deep Models
• APAC: Augmented PAttern Classification with Neural Networks
• A Kernel Theory of Modern Data Augmentation
• Critical Learning Periods in Deep Neural Networks
• Implicit Semantic Data Augmentation for Deep Networks
• Does Data Augmentation Lead to Positive Margin?

Cited by in corpus (14)

• FMix: Enhancing Mixed Sample Data Augmentation
• ProCAN: Progressive Growing Channel Attentive Non-Local Network for Lung Nodule Classification
• Affinity and Diversity: Quantifying Mechanisms of Data Augmentation
• Data augmentation and feature selection for automatic model recommendation in computational physics
• An Empirical Analysis of the Impact of Data Augmentation on Knowledge Distillation
• A Fourier-based Framework for Domain Generalization
• Circumventing Outliers of AutoAugment with Knowledge Distillation
• Towards Calibrated Model for Long-Tailed Visual Recognition from Prior Perspective
• Does Data Augmentation Benefit from Split BatchNorms
• Direct Differentiable Augmentation Search
• Mixup Without Hesitation
• Self-paced Data Augmentation for Training Neural Networks
• Octave Mix: Data augmentation using frequency decomposition for activity recognition
• Jitter: Random Jittering Loss Function