MixMatch: A Holistic Approach to Semi-Supervised Learning

Abstract

Semi-supervised learning has proven to be a powerful paradigm for leveraging unlabeled data to mitigate the reliance on large labeled datasets. In this work, we unify the current dominant approaches for semi-supervised learning to produce a new algorithm, MixMatch, that works by guessing low-entropy labels for data-augmented unlabeled examples and mixing labeled and unlabeled data using MixUp. We show that MixMatch obtains state-of-the-art results by a large margin across many datasets and labeled data amounts. For example, on CIFAR-10 with 250 labels, we reduce error rate by a factor of 4 (from 38% to 11%) and by a factor of 2 on STL-10. We also demonstrate how MixMatch can help achieve a dramatically better accuracy-privacy trade-off for differential privacy. Finally, we perform an ablation study to tease apart which components of MixMatch are most important for its success.

References in corpus (19)

• Decoupled Weight Decay Regularization
• Deep Learning with Differential Privacy
• mixup: Beyond Empirical Risk Minimization
• Improved Regularization of Convolutional Neural Networks with Cutout
• Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
• Deep Big Simple Neural Nets Excel on Handwritten Digit Recognition
• Interpolation Consistency Training for Semi-Supervised Learning
• AutoAugment: Learning Augmentation Policies from Data
• Semi-Supervised Learning with Ladder Networks
• Semi-Supervised Learning with Generative Adversarial Networks
• Variational Autoencoder for Deep Learning of Images, Labels and Captions
• Shake-Shake regularization
• Scalable Private Learning with PATE
• Stacked What-Where Auto-encoders
• Semi-Supervised Learning with Context-Conditional Generative Adversarial Networks
• Three Mechanisms of Weight Decay Regularization
• Spike-and-Slab Sparse Coding for Unsupervised Feature Discovery
• On the Generalization Properties of Differential Privacy
• Deep Metric Transfer for Label Propagation with Limited Annotated Data

Cited by in corpus (121)

• Shortcut Learning in Deep Neural Networks
• Unsupervised Data Augmentation for Consistency Training
• Learning Representations by Maximizing Mutual Information Across Views
• Annotation-efficient deep learning for automatic medical image segmentation
• A Comprehensive Survey on Test-Time Adaptation under Distribution Shifts
• Are Labels Required for Improving Adversarial Robustness?
• ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoring
• Semi-supervised semantic segmentation needs strong, varied perturbations
• One Million Scenes for Autonomous Driving: ONCE Dataset
• Improve Unsupervised Domain Adaptation with Mixup Training
• Reinventing 2D Convolutions for 3D Images
• And the Bit Goes Down: Revisiting the Quantization of Neural Networks
• Cycle Self-Training for Domain Adaptation
• A Survey of Deep Learning for Scientific Discovery
• Towards Utilizing Unlabeled Data in Federated Learning: A Survey and Prospective
• High Accuracy and High Fidelity Extraction of Neural Networks
• Structured Consistency Loss for semi-supervised semantic segmentation
• Virtual Mixup Training for Unsupervised Domain Adaptation
• Consistency Regularization for Generative Adversarial Networks
• SemiFL: Semi-Supervised Federated Learning for Unlabeled Clients with Alternate Training
• Class-Imbalanced Semi-Supervised Learning
• Machine Learning with Confidential Computing: A Systematization of Knowledge
• Interpolated Adversarial Training: Achieving Robust Neural Networks without Sacrificing Too Much Accuracy
• GAMIN: An Adversarial Approach to Black-Box Model Inversion
• Towards Discriminability and Diversity: Batch Nuclear-norm Maximization under Label Insufficient Situations
• Consistent Estimators for Learning to Defer to an Expert
• OpenMatch: Open-set Consistency Regularization for Semi-supervised Learning with Outliers
• Graph Random Neural Network for Semi-Supervised Learning on Graphs
• Semi-Conditional Normalizing Flows for Semi-Supervised Learning
• Semi-Supervised Visual Representation Learning for Fashion Compatibility
• Dirichlet-Based Prediction Calibration for Learning with Noisy Labels
• Artificial and beneficial -- Exploiting artificial images for aerial vehicle detection
• Towards Cross-Granularity Few-Shot Learning: Coarse-to-Fine Pseudo-Labeling with Visual-Semantic Meta-Embedding
• A Deep Model for Partial Multi-Label Image Classification with Curriculum Based Disambiguation
• Rethinking Re-Sampling in Imbalanced Semi-Supervised Learning
• Variational Rectification Inference for Learning with Noisy Labels
• SECANT: Self-Expert Cloning for Zero-Shot Generalization of Visual Policies
• Semi-Supervised Semantic Segmentation via Adaptive Equalization Learning
• Robust Mutual Learning for Semi-supervised Semantic Segmentation
• Weakly Supervised Semantic Segmentation by Pixel-to-Prototype Contrast
• Semi-Supervised Models via Data Augmentationfor Classifying Interactive Affective Responses
• MixMatch Domain Adaptaion: Prize-winning solution for both tracks of VisDA 2019 challenge
• Contrastive Learning Improves Model Robustness Under Label Noise
• How Useful is Self-Supervised Pretraining for Visual Tasks?
• Multimodal Semi-Supervised Learning for 3D Objects
• On Mixup Regularization
• Federated Semi-Supervised Learning with Class Distribution Mismatch
• Learning to Impute: A General Framework for Semi-supervised Learning
• SegMatch: A semi-supervised learning method for surgical instrument segmentation
• When and How Mixup Improves Calibration
• Semi-supervised Viewpoint Estimation with Geometry-aware Conditional Generation
• Semi-Supervised Learning with Normalizing Flows
• Learning from Label Proportions with Consistency Regularization
• OpenCoS: Contrastive Semi-supervised Learning for Handling Open-set Unlabeled Data
• Revisiting consistency for semi-supervised semantic segmentation
• A Fusion-Denoising Attack on InstaHide with Data Augmentation
• RNAS-CL: Robust Neural Architecture Search by Cross-Layer Knowledge Distillation
• AdvAug: Robust Adversarial Augmentation for Neural Machine Translation
• Semi-supervised Image Classification with Grad-CAM Consistency
• MetaMixUp: Learning Adaptive Interpolation Policy of MixUp with Meta-Learning
• An analysis of over-sampling labeled data in semi-supervised learning with FixMatch
• S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning
• FedSEAL: Semi-Supervised Federated Learning with Self-Ensemble Learning and Negative Learning
• Robust Training of Social Media Image Classification Models for Rapid Disaster Response
• Recent Deep Semi-supervised Learning Approaches and Related Works
• Improving Image Classification of Knee Radiographs: An Automated Image Labeling Approach
• Training without training data: Improving the generalizability of automated medical abbreviation disambiguation
• Negative sampling in semi-supervised learning
• SHOT-VAE: Semi-supervised Deep Generative Models With Label-aware ELBO Approximations
• Edge Bias in Federated Learning and its Solution by Buffered Knowledge Distillation
• Built Year Prediction from Buddha Face with Heterogeneous Labels
• Adversarial Training Helps Transfer Learning via Better Representations
• Manifold Graph with Learned Prototypes for Semi-Supervised Image Classification
• Training on Test Data with Bayesian Adaptation for Covariate Shift
• Semi-supervised Long-tailed Recognition using Alternate Sampling
• Skin disease diagnosis with deep learning: a review
• Are We Hungry for 3D LiDAR Data for Semantic Segmentation? A Survey and Experimental Study
• Deep Learning Classification With Noisy Labels
• Neural Networks Are More Productive Teachers Than Human Raters: Active Mixup for Data-Efficient Knowledge Distillation from a Blackbox Model
• MixPUL: Consistency-based Augmentation for Positive and Unlabeled Learning
• Bridge the Domain Gap Between Ultra-wide-field and Traditional Fundus Images via Adversarial Domain Adaptation
• Improving Molecular Design by Stochastic Iterative Target Augmentation
• HODGEPODGE: Sound event detection based on ensemble of semi-supervised learning methods
• Learning Invariant Representation with Consistency and Diversity for Semi-supervised Source Hypothesis Transfer
• Adversarial Transformations for Semi-Supervised Learning
• Consistency Regularization for Deep Face Anti-Spoofing
• Two Sides of the Same Coin: Learning the Backdoor to Remove the Backdoor
• ROAM: Random Layer Mixup for Semi-Supervised Learning in Medical Imaging
• StackMix: A complementary Mix algorithm
• Optimally Combining Classifiers for Semi-Supervised Learning
• Local Clustering with Mean Teacher for Semi-supervised Learning
• Data augmentation with Mobius transformations
• Pseudo-Representation Labeling Semi-Supervised Learning
• Pretraining Image Encoders without Reconstruction via Feature Prediction Loss
• Alignment Attention by Matching Key and Query Distributions
• Semi-Supervised Class Discovery
• Learning to Detect Important People in Unlabelled Images for Semi-supervised Important People Detection
• Multimodal Knowledge Expansion
• T${}^2$-Net: A Semi-supervised Deep Model for Turbulence Forecasting
• Improving Distinction between ASR Errors and Speech Disfluencies with Feature Space Interpolation
• RPCL: A Framework for Improving Cross-Domain Detection with Auxiliary Tasks
• Gradient-based Data Augmentation for Semi-Supervised Learning
• A New Distribution on the Simplex with Auto-Encoding Applications
• Sample Prior Guided Robust Model Learning to Suppress Noisy Labels
• Active Crowd Counting with Limited Supervision
• Boosting Semi-Supervised Face Recognition with Noise Robustness
• CoDiM: Learning with Noisy Labels via Contrastive Semi-Supervised Learning
• Hodge and Podge: Hybrid Supervised Sound Event Detection with Multi-Hot MixMatch and Composition Consistence Training
• Efficient data augmentation using graph imputation neural networks
• Semi-supervised learning method based on predefined evenly-distributed class centroids
• BiSTF: Bilateral-Branch Self-Training Framework for Semi-Supervised Large-scale Fine-Grained Recognition
• Mutual-Information Based Few-Shot Classification
• MetaMix: Improved Meta-Learning with Interpolation-based Consistency Regularization
• Improving Limited Labeled Dialogue State Tracking with Self-Supervision
• SketchTransfer: A Challenging New Task for Exploring Detail-Invariance and the Abstractions Learned by Deep Networks
• Scalable Consistency Training for Graph Neural Networks via Self-Ensemble Self-Distillation
• Predictions For Pre-training Language Models
• Epsilon Consistent Mixup: Structural Regularization with an Adaptive Consistency-Interpolation Tradeoff
• VeniBot: Towards Autonomous Venipuncture with Semi-supervised Vein Segmentation from Ultrasound Images
• Domain-Guided Task Decomposition with Self-Training for Detecting Personal Events in Social Media
• End-To-End Graph-based Deep Semi-Supervised Learning