ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoring

Abstract

We improve the recently-proposed "MixMatch" semi-supervised learning algorithm by introducing two new techniques: distribution alignment and augmentation anchoring. Distribution alignment encourages the marginal distribution of predictions on unlabeled data to be close to the marginal distribution of ground-truth labels. Augmentation anchoring feeds multiple strongly augmented versions of an input into the model and encourages each output to be close to the prediction for a weakly-augmented version of the same input. To produce strong augmentations, we propose a variant of AutoAugment which learns the augmentation policy while the model is being trained. Our new algorithm, dubbed ReMixMatch, is significantly more data-efficient than prior work, requiring between $5\times$ and $16\times$ less data to reach the same accuracy. For example, on CIFAR-10 with 250 labeled examples we reach $93.73\%$ accuracy (compared to MixMatch's accuracy of $93.58\%$ with $4,000$ examples) and a median accuracy of $84.92\%$ with just four labels per class. We make our code and data open-source at https://github.com/google-research/remixmatch.

References in corpus (10)

• mixup: Beyond Empirical Risk Minimization
• Unsupervised Data Augmentation for Consistency Training
• Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
• Interpolation Consistency Training for Semi-Supervised Learning
• AutoAugment: Learning Augmentation Policies from Data
• MixMatch: A Holistic Approach to Semi-Supervised Learning
• Learning with Pseudo-Ensembles
• Stacked What-Where Auto-encoders
• Three Mechanisms of Weight Decay Regularization
• Deep Metric Transfer for Label Propagation with Limited Annotated Data

Cited by in corpus (56)

• Rethinking Pre-training and Self-training
• An Overview of Deep Semi-Supervised Learning
• Self-training with Noisy Student improves ImageNet classification
• Semi-supervised semantic segmentation needs strong, varied perturbations
• One Million Scenes for Autonomous Driving: ONCE Dataset
• Unsupervised Sound Separation Using Mixture Invariant Training
• Distribution Aligning Refinery of Pseudo-label for Imbalanced Semi-supervised Learning
• Affinity and Diversity: Quantifying Mechanisms of Data Augmentation
• A critical look at the current train/test split in machine learning
• SemiFL: Semi-Supervised Federated Learning for Unlabeled Clients with Alternate Training
• OpenMatch: Open-set Consistency Regularization for Semi-supervised Learning with Outliers
• Unsupervised Semantic Aggregation and Deformable Template Matching for Semi-Supervised Learning
• Rethinking Re-Sampling in Imbalanced Semi-Supervised Learning
• A New Semi-supervised Learning Benchmark for Classifying View and Diagnosing Aortic Stenosis from Echocardiograms
• The Semi-Supervised iNaturalist-Aves Challenge at FGVC7 Workshop
• SECANT: Self-Expert Cloning for Zero-Shot Generalization of Visual Policies
• Robust Mutual Learning for Semi-supervised Semantic Segmentation
• Federated Semi-Supervised Learning for COVID Region Segmentation in Chest CT using Multi-National Data from China, Italy, Japan
• 3DIoUMatch: Leveraging IoU Prediction for Semi-Supervised 3D Object Detection
• Multimodal Semi-Supervised Learning for 3D Objects
• FeatMatch: Feature-Based Augmentation for Semi-Supervised Learning
• Triple Generative Adversarial Networks
• Self-supervised Mean Teacher for Semi-supervised Chest X-ray Classification
• Learning to Impute: A General Framework for Semi-supervised Learning
• FixBi: Bridging Domain Spaces for Unsupervised Domain Adaptation
• A Realistic Evaluation of Semi-Supervised Learning for Fine-Grained Classification
• Calibrate and Prune: Improving Reliability of Lottery Tickets Through Prediction Calibration
• Robust Semantic Segmentation with Superpixel-Mix
• SHOT-VAE: Semi-supervised Deep Generative Models With Label-aware ELBO Approximations
• Robust Training of Social Media Image Classification Models for Rapid Disaster Response
• Recent Deep Semi-supervised Learning Approaches and Related Works
• Boosting Active Learning for Speech Recognition with Noisy Pseudo-labeled Samples
• Pseudo-labeling for Scalable 3D Object Detection
• Co-matching: Combating Noisy Labels by Augmentation Anchoring
• Evaluating Self and Semi-Supervised Methods for Remote Sensing Segmentation Tasks
• Direct Differentiable Augmentation Search
• A Unified Contrastive Loss for Self-Training
• A Probabilistic Model for Discriminative and Neuro-Symbolic Semi-Supervised Learning
• Supervision Accelerates Pre-training in Contrastive Semi-Supervised Learning of Visual Representations
• Consistency Regularization with Generative Adversarial Networks for Semi-Supervised Learning
• Empirical Perspectives on One-Shot Semi-supervised Learning
• Boosting the Performance of Semi-Supervised Learning with Unsupervised Clustering
• Semi-Supervised Vision Transformers
• X-model: Improving Data Efficiency in Deep Learning with A Minimax Model
• On the Marginal Benefit of Active Learning: Does Self-Supervision Eat Its Cake?
• Multimodal Knowledge Expansion
• AlphaMatch: Improving Consistency for Semi-supervised Learning with Alpha-divergence
• End-To-End Graph-based Deep Semi-Supervised Learning
• On-target Adaptation
• Improve Unsupervised Pretraining for Few-label Transfer
• Predictions For Pre-training Language Models
• Hard Class Rectification for Domain Adaptation
• Adaptive Hierarchical Similarity Metric Learning with Noisy Labels
• Learning the Prediction Distribution for Semi-Supervised Learning with Normalising Flows
• LiDAM: Semi-Supervised Learning with Localized Domain Adaptation and Iterative Matching
• FROST: Faster and more Robust One-shot Semi-supervised Training