How Does Mixup Help With Robustness and Generalization?

Abstract

Mixup is a popular data augmentation technique based on taking convex combinations of pairs of examples and their labels. This simple technique has been shown to substantially improve both the robustness and the generalization of the trained model. However, it is not well-understood why such improvement occurs. In this paper, we provide theoretical analysis to demonstrate how using Mixup in training helps model robustness and generalization. For robustness, we show that minimizing the Mixup loss corresponds to approximately minimizing an upper bound of the adversarial loss. This explains why models obtained by Mixup training exhibits robustness to several kinds of adversarial attacks such as Fast Gradient Sign Method (FGSM). For generalization, we prove that Mixup augmentation corresponds to a specific type of data-adaptive regularization which reduces overfitting. Our analysis provides new insights and a framework to understand Mixup.

arXiv commentAccepted by ICLR 2021, and selected as spotlight (top 6% of submissions)

References in corpus (20)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Wide Residual Networks
• Towards Deep Learning Models Resistant to Adversarial Attacks
• Understanding deep learning requires rethinking generalization
• Interpolation Consistency Training for Semi-Supervised Learning
• Manifold Mixup: Better Representations by Interpolating Hidden States
• Robustness May Be at Odds with Accuracy
• Dropout Training as Adaptive Regularization
• Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks
• Norm-Based Capacity Control in Neural Networks
• Puzzle Mix: Exploiting Saliency and Local Statistics for Optimal Mixup
• On the Implicit Bias of Dropout
• The Implicit and Explicit Regularization Effects of Dropout
• On Adversarial Mixup Resynthesis
• Dropout: Explicit Forms and Capacity Control
• On Mixup Regularization
• When and How Mixup Improves Calibration
• On the Depth of Deep Neural Networks: A Theoretical View
• Interpreting Robust Optimization via Adversarial Influence Functions

Cited by in corpus (9)

• FMix: Enhancing Mixed Sample Data Augmentation
• Deep Learning to Estimate Permeability using Geophysical Data
• DP-InstaHide: Provably Defusing Poisoning and Backdoor Attacks with Differentially Private Data Augmentations
• When and How Mixup Improves Calibration
• Towards Understanding the Data Dependency of Mixup-style Training
• Improving Adversarial Robustness via Unlabeled Out-of-Domain Data
• SMILE: Self-Distilled MIxup for Efficient Transfer LEarning
• k-Mixup Regularization for Deep Learning via Optimal Transport
• Noisy Feature Mixup