Manifold Mixup: Better Representations by Interpolating Hidden States

Abstract

Deep neural networks excel at learning the training data, but often provide incorrect and confident predictions when evaluated on slightly different test examples. This includes distribution shifts, outliers, and adversarial examples. To address these issues, we propose Manifold Mixup, a simple regularizer that encourages neural networks to predict less confidently on interpolations of hidden representations. Manifold Mixup leverages semantic interpolations as additional training signal, obtaining neural networks with smoother decision boundaries at multiple levels of representation. As a result, neural networks trained with Manifold Mixup learn class-representations with fewer directions of variance. We prove theory on why this flattening happens under ideal conditions, validate it on practical situations, and connect it to previous works on information theory and generalization. In spite of incurring no significant computation and being implemented in a few lines of code, Manifold Mixup improves strong baselines in supervised learning, robustness to single-step adversarial attacks, and test log-likelihood.

arXiv commentTo appear in ICML 2019

Cited by in corpus (165)

• Interpolation Consistency Training for Semi-Supervised Learning
• CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features
• Regularized Deep Networks in Intelligent Transportation Systems: A Taxonomy and a Case Study
• A Survey on Data Augmentation for Text Classification
• Embracing Imperfect Datasets: A Review of Deep Learning Solutions for Medical Image Segmentation
• Hard Negative Mixing for Contrastive Learning
• DoFE: Domain-oriented Feature Embedding for Generalizable Fundus Image Segmentation on Unseen Datasets
• Motivating the Rules of the Game for Adversarial Example Research
• InfoGraph: Unsupervised and Semi-supervised Graph-Level Representation Learning via Mutual Information Maximization
• Delving into Deep Imbalanced Regression
• Predicting the Generalization Gap in Deep Networks with Margin Distributions
• FMix: Enhancing Mixed Sample Data Augmentation
• An overview of mixing augmentation methods and augmentation strategies
• Revisiting Training Strategies and Generalization Performance in Deep Metric Learning
• FedMix: Approximation of Mixup under Mean Augmented Federated Learning
• Remix: Rebalanced Mixup
• MixCo: Mix-up Contrastive Learning for Visual Representation
• Feature Fusion from Head to Tail for Long-Tailed Visual Recognition
• Virtual Mixup Training for Unsupervised Domain Adaptation
• Self-Augmentation: Generalizing Deep Networks to Unseen Classes for Few-Shot Learning
• Domain Generalization with MixStyle
• MHFC: Multi-Head Feature Collaboration for Few-Shot Learning
• Data Augmentation Revisited: Rethinking the Distribution Gap between Clean and Augmented Data
• PatchUp: A Feature-Space Block-Level Regularization Technique for Convolutional Neural Networks
• Leveraging the Feature Distribution in Transfer-based Few-Shot Learning
• MixKD: Towards Efficient Distillation of Large-scale Language Models
• Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regularized Fine-Tuning
• Fairness via Representation Neutralization
• Interpolated Adversarial Training: Achieving Robust Neural Networks without Sacrificing Too Much Accuracy
• MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classification
• Improved Automatic Diabetic Retinopathy Severity Classification Using Deep Multimodal Fusion of UWF-CFP and OCTA Images
• REMIND Your Neural Network to Prevent Catastrophic Forgetting
• How Does Mixup Help With Robustness and Generalization?
• A Unified Survey on Anomaly, Novelty, Open-Set, and Out-of-Distribution Detection: Solutions and Future Challenges
• GuidedMixup: An Efficient Mixup Strategy Guided by Saliency Maps
• InstaHide: Instance-hiding Schemes for Private Distributed Learning
• Mixup-CAM: Weakly-supervised Semantic Segmentation via Uncertainty Regularization
• Towards Inheritable Models for Open-Set Domain Adaptation
• ELF: An Early-Exiting Framework for Long-Tailed Classification
• Flow-Mixup: Classifying Multi-labeled Medical Images with Corrupted Labels
• Representation Based Complexity Measures for Predicting Generalization in Deep Learning
• MixStyle Neural Networks for Domain Generalization and Adaptation
• Boosting Discriminative Visual Representation Learning with Scenario-Agnostic Mixup
• Dual Mixup Regularized Learning for Adversarial Domain Adaptation
• Interpolation-based semi-supervised learning for object detection
• GraphMixup: Improving Class-Imbalanced Node Classification on Graphs by Self-supervised Context Prediction
• Fair Mixup: Fairness via Interpolation
• Hyperbolic Deep Neural Networks: A Survey
• Meta-Learning with Fewer Tasks through Task Interpolation
• Tackling Catastrophic Forgetting and Background Shift in Continual Semantic Segmentation
• Compression based bound for non-compressed network: unified generalization error analysis of large compressible deep neural network
• Deep learning in magnetic resonance prostate segmentation: A review and a new perspective
• Exploring Self-Supervised Regularization for Supervised and Semi-Supervised Learning
• Understand and Improve Contrastive Learning Methods for Visual Representation: A Review
• Iterative Graph Self-Distillation
• DALLMi: Domain Adaption for LLM-based Multi-label Classifier
• Improving Calibration for Long-Tailed Recognition
• ReMix: Calibrated Resampling for Class Imbalance in Deep learning
• It Takes Two to Tango: Mixup for Deep Metric Learning
• Towards Domain-Agnostic Contrastive Learning
• Combining Ensembles and Data Augmentation can Harm your Calibration
• On Mixup Regularization
• Fine-grained Synthesis of Unrestricted Adversarial Examples
• MixSiam: A Mixture-based Approach to Self-supervised Representation Learning
• Traces of Class/Cross-Class Structure Pervade Deep Learning Spectra
• When and How Mixup Improves Calibration
• Graph-based Interpolation of Feature Vectors for Accurate Few-Shot Classification
• On-manifold Adversarial Data Augmentation Improves Uncertainty Calibration
• Data Interpolating Prediction: Alternative Interpretation of Mixup
• Improving Robustness of Deep Learning Based Knee MRI Segmentation: Mixup and Adversarial Domain Adaptation
• On Data Augmentation and Adversarial Risk: An Empirical Analysis
• IDM: An Intermediate Domain Module for Domain Adaptive Person Re-ID
• Towards Understanding the Data Dependency of Mixup-style Training
• Calibrated Language Model Fine-Tuning for In- and Out-of-Distribution Data
• MixUp Training Leads to Reduced Overfitting and Improved Calibration for the Transformer Architecture
• FASA: Feature Augmentation and Sampling Adaptation for Long-Tailed Instance Segmentation
• Distributed Heteromodal Split Learning for Vision Aided mmWave Received Power Prediction
• SeqMix: Augmenting Active Sequence Labeling via Sequence Mixup
• Robust Semantic Segmentation with Superpixel-Mix
• Recent Deep Semi-supervised Learning Approaches and Related Works
• Semi-supervised Medical Image Classification with Global Latent Mixing
• Towards Calibrated Model for Long-Tailed Visual Recognition from Prior Perspective
• AlignZeg: Mitigating Objective Misalignment for Zero-shot Semantic Segmentation
• Learning Placeholders for Open-Set Recognition
• Machine learning with limited data
• Shape-Texture Debiased Neural Network Training
• SHOT-VAE: Semi-supervised Deep Generative Models With Label-aware ELBO Approximations
• Center-wise Local Image Mixture For Contrastive Representation Learning
• Improving Contrastive Learning by Visualizing Feature Transformation
• GuidedMix-Net: Learning to Improve Pseudo Masks Using Labeled Images as Reference
• Manifold Graph with Learned Prototypes for Semi-Supervised Image Classification
• Few-Shot Learning by Integrating Spatial and Frequency Representation
• Learning Flat Latent Manifolds with VAEs
• LMT: Longitudinal Mixing Training, a Framework to Predict Disease Progression from a Single Image
• Mixup Regularization for Region Proposal based Object Detectors
• On Calibration of Mixup Training for Deep Neural Networks
• Attribute Mix: Semantic Data Augmentation for Fine Grained Recognition
• Adversarially Optimized Mixup for Robust Classification
• Gradual Domain Adaptation in the Wild:When Intermediate Distributions are Absent
• Robust Pollen Imagery Classification with Generative Modeling and Mixup Training
• On the Importance of Firth Bias Reduction in Few-Shot Classification
• Learning Temporally Invariant and Localizable Features via Data Augmentation for Video Recognition
• Reduced-Lead ECG Classifier Model Trained with DivideMix and Model Ensemble
• StackMix: A complementary Mix algorithm
• Sequence-Level Mixed Sample Data Augmentation
• Causality-inspired Single-source Domain Generalization for Medical Image Segmentation
• Encoding Robustness to Image Style via Adversarial Feature Perturbations
• SMILE: Self-Distilled MIxup for Efficient Transfer LEarning
• ROAM: Random Layer Mixup for Semi-Supervised Learning in Medical Imaging
• Normalization Before Shaking Toward Learning Symmetrically Distributed Representation Without Margin in Speech Emotion Recognition
• Enhancing Mixup-based Semi-Supervised Learning with Explicit Lipschitz Regularization
• TransMix: Attend to Mix for Vision Transformers
• MetaPerturb: Transferable Regularizer for Heterogeneous Tasks and Architectures
• Improving Augmentation and Evaluation Schemes for Semantic Image Synthesis
• Predicting the Accuracy of a Few-Shot Classifier
• Why Mixup Improves the Model Performance
• Neural Function Modules with Sparse Arguments: A Dynamic Approach to Integrating Information across Layers
• Semi-supervised Learning for Identifying the Likelihood of Agitation in People with Dementia
• Improving Classifier Confidence using Lossy Label-Invariant Transformations
• A Unified Mixture-View Framework for Unsupervised Representation Learning
• Energy Aligning for Biased Models
• COUnty aggRegation mixup AuGmEntation (COURAGE) COVID-19 Prediction
• Task Affinity with Maximum Bipartite Matching in Few-Shot Learning
• Conditioning Autoencoder Latent Spaces for Real-Time Timbre Interpolation and Synthesis
• Improving Distinction between ASR Errors and Speech Disfluencies with Feature Space Interpolation
• Data Augmentation via Mixed Class Interpolation using Cycle-Consistent Generative Adversarial Networks Applied to Cross-Domain Imagery
• Mixup Regularized Adversarial Networks for Multi-Domain Text Classification
• Principled learning method for Wasserstein distributionally robust optimization with local perturbations
• GroupMixNorm Layer for Learning Fair Models
• Towards Recognizing New Semantic Concepts in New Visual Domains
• Medical Image Segmentation with Limited Supervision: A Review of Deep Network Models
• Similarity Transfer for Knowledge Distillation
• Gi and Pal Scores: Deep Neural Network Generalization Statistics
• Mixup Without Hesitation
• Contrastive Mixup: Self- and Semi-Supervised learning for Tabular Domain
• Active Crowd Counting with Limited Supervision
• Enhancing Data-Free Adversarial Distillation with Activation Regularization and Virtual Interpolation
• Graph Interpolating Activation Improves Both Natural and Robust Accuracies in Data-Efficient Deep Learning
• Data augmentation by morphological mixup for solving Raven's Progressive Matrices
• Distance Metric-Based Learning with Interpolated Latent Features for Location Classification in Endoscopy Image and Video
• SynFace: Face Recognition with Synthetic Data
• PointManifoldCut: Point-wise Augmentation in the Manifold for Point Clouds
• Controlling Covariate Shift using Balanced Normalization of Weights
• Graphs for deep learning representations
• Noisy Feature Mixup
• Geo-Spatiotemporal Features and Shape-Based Prior Knowledge for Fine-grained Imbalanced Data Classification
• Edge Artificial Intelligence for 6G: Vision, Enabling Technologies, and Applications
• Calibrating Class Activation Maps for Long-Tailed Visual Recognition
• Using Fictitious Class Representations to Boost Discriminative Zero-Shot Learners
• An Empirical Study of the Effects of Sample-Mixing Methods for Efficient Training of Generative Adversarial Networks
• DiffAugment: Diffusion based Long-Tailed Visual Relationship Recognition
• Self-training Improves Pre-training for Few-shot Learning in Task-oriented Dialog Systems
• Topologically Densified Distributions
• Stochastic Batch Augmentation with An Effective Distilled Dynamic Soft Label Regularizer
• Procrustean Training for Imbalanced Deep Learning
• MDFM: Multi-Decision Fusing Model for Few-Shot Learning
• Robust Regularization with Adversarial Labelling of Perturbed Samples
• Mixup of Feature Maps in a Hidden Layer for Training of Convolutional Neural Network
• Suppressing Mislabeled Data via Grouping and Self-Attention
• ShuffleBlock: Shuffle to Regularize Deep Convolutional Neural Networks
• Coded-InvNet for Resilient Prediction Serving Systems
• SketchTransfer: A Challenging New Task for Exploring Detail-Invariance and the Abstractions Learned by Deep Networks
• Counterfactual Adversarial Learning with Representation Interpolation
• Adversarial Mixing Policy for Relaxing Locally Linear Constraints in Mixup
• Out-of-Manifold Regularization in Contextual Embedding Space for Text Classification