MaxUp: A Simple Way to Improve Generalization of Neural Network Training

Abstract

We propose \emph{MaxUp}, an embarrassingly simple, highly effective technique for improving the generalization performance of machine learning models, especially deep neural networks. The idea is to generate a set of augmented data with some random perturbations or transforms and minimize the maximum, or worst case loss over the augmented data. By doing so, we implicitly introduce a smoothness or robustness regularization against the random perturbations, and hence improve the generation performance. For example, in the case of Gaussian perturbation, \emph{MaxUp} is asymptotically equivalent to using the gradient norm of the loss as a penalty to encourage smoothness. We test \emph{MaxUp} on a range of tasks, including image classification, language modeling, and adversarial certification, on which \emph{MaxUp} consistently outperforms the existing best baseline methods, without introducing substantial computational overhead. In particular, we improve ImageNet classification from the state-of-the-art top-1 accuracy $85.5\%$ without extra data to $85.8\%$. Code will be released soon.

References in corpus (11)

• Neural Architecture Search with Reinforcement Learning
• Improved Regularization of Convolutional Neural Networks with Cutout
• Theoretically Principled Trade-off between Robustness and Accuracy
• AutoAugment: Learning Augmentation Policies from Data
• CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features
• DropBlock: A regularization method for convolutional networks
• Adversarially Robust Generalization Requires More Data
• FRAGE: Frequency-Agnostic Word Representation
• Adversarial Examples Improve Image Recognition
• Fixing the train-test resolution discrepancy
• Optimal Non-Asymptotic Lower Bound on the Minimax Regret of Learning with Expert Advice

Cited by in corpus (14)

• Bootstrap your own latent: A new approach to self-supervised Learning
• High-Performance Large-Scale Image Recognition Without Normalization
• Vision Transformers with Patch Diversification
• Data Augmentation for Meta-Learning
• DP-InstaHide: Provably Defusing Poisoning and Backdoor Attacks with Differentially Private Data Augmentations
• On the Inherent Regularization Effects of Noise Injection During Training
• KeepAugment: A Simple Information-Preserving Data Augmentation Approach
• AttentiveNAS: Improving Neural Architecture Search via Attentive Sampling
• Noisy Recurrent Neural Networks
• Improving Robustness of Learning-based Autonomous Steering Using Adversarial Images
• Encoding Robustness to Image Style via Adversarial Feature Perturbations
• Smoothed Inference for Adversarially-Trained Models
• Architectural Resilience to Foreground-and-Background Adversarial Noise
• Noisy Feature Mixup