Noisy Feature Mixup

Abstract

We introduce Noisy Feature Mixup (NFM), an inexpensive yet effective method for data augmentation that combines the best of interpolation based training and noise injection schemes. Rather than training with convex combinations of pairs of examples and their labels, we use noise-perturbed convex combinations of pairs of data points in both input and feature space. This method includes mixup and manifold mixup as special cases, but it has additional advantages, including better smoothing of decision boundaries and enabling improved model robustness. We provide theory to understand this as well as the implicit regularization effects of NFM. Our theory is supported by empirical results, demonstrating the advantage of NFM, as compared to mixup and manifold mixup. We show that residual networks and vision transformers trained with NFM have favorable trade-offs between predictive accuracy on clean data and robustness with respect to various types of data perturbation across a range of computer vision benchmark datasets.

arXiv comment34 pages

References in corpus (38)

• Explaining and Harnessing Adversarial Examples
• mixup: Beyond Empirical Risk Minimization
• Wide Residual Networks
• Towards Deep Learning Models Resistant to Adversarial Attacks
• Deep Big Simple Neural Nets Excel on Handwritten Digit Recognition
• Theoretically Principled Trade-off between Robustness and Accuracy
• AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty
• Adversarially Robust Generalization Requires More Data
• Manifold Mixup: Better Representations by Interpolating Hidden States
• Robustness May Be at Odds with Accuracy
• Robustness and Regularization of Support Vector Machines
• Escaping the Big Data Paradigm with Compact Transformers
• Sensitivity and Generalization in Neural Networks: an Empirical Study
• Formal Guarantees on the Robustness of a Classifier against Adversarial Manipulation
• Robust Large Margin Deep Neural Networks
• Robustness of classifiers: from adversarial to random noise
• Predicting the Generalization Gap in Deep Networks with Margin Distributions
• Fixing Data Augmentation to Improve Adversarial Robustness
• A Kernel Theory of Modern Data Augmentation
• A Closer Look at Accuracy vs. Robustness
• Puzzle Mix: Exploiting Saliency and Local Statistics for Optimal Mixup
• Adversarial Robustness as a Prior for Learned Representations
• Understanding and Mitigating the Tradeoff Between Robustness and Accuracy
• Robust Learning with Jacobian Regularization
• Explicit Regularisation in Gaussian Noise Injections
• Theoretical evidence for adversarial robustness through randomization
• MaxUp: A Simple Way to Improve Generalization of Neural Network Training
• The Curious Case of Adversarially Robust Models: More Data Can Help, Double Descend, or Hurt Generalization
• Improved Sample Complexities for Deep Networks and Robust Classification via an All-Layer Margin
• How Does Mixup Help With Robustness and Generalization?
• Boundary thickness and robustness in learning models
• Large Margin Deep Networks for Classification
• On Mixup Regularization
• When and How Mixup Improves Calibration
• Noisy Recurrent Neural Networks
• A unified view on differential privacy and robustness to adversarial examples
• k-Mixup Regularization for Deep Learning via Optimal Transport
• Why Mixup Improves the Model Performance