WeMix: How to Better Utilize Data Augmentation
arXiv:2010.01267
Abstract
Data augmentation is a widely used training trick in deep learning to improve the network generalization ability. Despite many encouraging results, several recent studies did point out limitations of the conventional data augmentation scheme in certain scenarios, calling for a better theoretical understanding of data augmentation. In this work, we develop a comprehensive analysis that reveals pros and cons of data augmentation. The main limitation of data augmentation arises from the data bias, i.e. the augmented data distribution can be quite different from the original one. This data bias leads to a suboptimal performance of existing data augmentation methods. To this end, we develop two novel algorithms, termed "AugDrop" and "MixLoss", to correct the data bias in the data augmentation. Our theoretical analysis shows that both algorithms are guaranteed to improve the effect of data augmentation through the bias correction, which is further validated by our empirical studies. Finally, we propose a generic algorithm "WeMix" by combining AugDrop and MixLoss, whose effectiveness is observed from extensive empirical evaluations.
References in corpus (22)
- Deep Learning in Neural Networks: An Overview
- ADADELTA: An Adaptive Learning Rate Method
- Wide Residual Networks
- On the Convergence of Adam and Beyond
- Understanding deep learning requires rethinking generalization
- A Convergence Theory for Deep Learning via Over-Parameterization
- Gradient Descent Provably Optimizes Over-parameterized Neural Networks
- Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
- On Lazy Training in Differentiable Programming
- Population Based Augmentation: Efficient Learning of Augmentation Policy Schedules
- A Kernel Theory of Modern Data Augmentation
- Diverse Neural Network Learns True Target Functions
- Complexity and performance of an Augmented Lagrangian algorithm
- Small ReLU networks are powerful memorizers: a tight analysis of memorization capacity
- Understanding and Mitigating the Tradeoff Between Robustness and Accuracy
- On the Complexity of an Augmented Lagrangian Method for Nonconvex Optimization
- The Curious Case of Adversarially Robust Models: More Data Can Help, Double Descend, or Hurt Generalization
- Towards Understanding Label Smoothing
- More Data Can Expand the Generalization Gap Between Adversarially Robust and Standard Models
- Inexact Proximal-Point Penalty Methods for Constrained Non-Convex Optimization
- A Second look at Exponential and Cosine Step Sizes: Simplicity, Adaptivity, and Performance
- Does Data Augmentation Lead to Positive Margin?