FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence

Abstract

Semi-supervised learning (SSL) provides an effective means of leveraging unlabeled data to improve a model's performance. In this paper, we demonstrate the power of a simple combination of two common SSL methods: consistency regularization and pseudo-labeling. Our algorithm, FixMatch, first generates pseudo-labels using the model's predictions on weakly-augmented unlabeled images. For a given image, the pseudo-label is only retained if the model produces a high-confidence prediction. The model is then trained to predict the pseudo-label when fed a strongly-augmented version of the same image. Despite its simplicity, we show that FixMatch achieves state-of-the-art performance across a variety of standard semi-supervised learning benchmarks, including 94.93% accuracy on CIFAR-10 with 250 labels and 88.61% accuracy with 40 -- just 4 labels per class. Since FixMatch bears many similarities to existing SSL methods that achieve worse performance, we carry out an extensive ablation study to tease apart the experimental factors that are most important to FixMatch's success. We make our code available at https://github.com/google-research/fixmatch.

arXiv commentPublished at NeurIPS 2020 as a conference paper

References in corpus (20)

• Decoupled Weight Decay Regularization
• mixup: Beyond Empirical Risk Minimization
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• Improved Regularization of Convolutional Neural Networks with Cutout
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Unsupervised Data Augmentation for Consistency Training
• Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
• A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks
• Interpolation Consistency Training for Semi-Supervised Learning
• Exploring the Limits of Language Modeling
• Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks
• Semi-Supervised Learning with Ladder Networks
• AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty
• Deep Learning Scaling is Predictable, Empirically
• Learning with Pseudo-Ensembles
• On Empirical Comparisons of Optimizers for Deep Learning
• Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks
• Three Mechanisms of Weight Decay Regularization
• Distribution Density, Tails, and Outliers in Machine Learning: Metrics and Applications

Cited by in corpus (245)

• A Simple Framework for Contrastive Learning of Visual Representations
• Bootstrap your own latent: A new approach to self-supervised Learning
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• Recent advances and clinical applications of deep learning in medical image analysis
• Hands-on Bayesian Neural Networks -- a Tutorial for Deep Learning Users
• Big Self-Supervised Models are Strong Semi-Supervised Learners
• Rethinking Pre-training and Self-training
• A Simple Semi-Supervised Learning Framework for Object Detection
• A Comprehensive Survey on Test-Time Adaptation under Distribution Shifts
• An Overview of Deep Semi-Supervised Learning
• Reinforcement Learning with Augmented Data
• Improved Noisy Student Training for Automatic Speech Recognition
• A survey on Semi-, Self- and Unsupervised Learning for Image Classification
• Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition
• SPICE: Semantic Pseudo-labeling for Image Clustering
• Semi-supervised semantic segmentation needs strong, varied perturbations
• One Million Scenes for Autonomous Driving: ONCE Dataset
• Image Augmentations for GAN Training
• Self-Training: A Survey
• Learning Self-Supervised Low-Rank Network for Single-Stage Weakly and Semi-Supervised Semantic Segmentation
• Contrast to Divide: Self-Supervised Pre-Training for Learning with Noisy Labels
• Distribution Aligning Refinery of Pseudo-label for Imbalanced Semi-supervised Learning
• Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Data
• PARSE: Pairwise Alignment of Representations in Semi-Supervised EEG Learning for Emotion Recognition
• In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learning
• Reference-guided Pseudo-Label Generation for Medical Semantic Segmentation
• A Systematic Survey of Regularization and Normalization in GANs
• TimeMatch: Unsupervised Cross-Region Adaptation by Temporal Shift Estimation
• Affinity and Diversity: Quantifying Mechanisms of Data Augmentation
• Dynamic Instance Domain Adaptation
• Conservative-Progressive Collaborative Learning for Semi-supervised Semantic Segmentation
• Iterative pseudo-labeling based adaptive copy-paste supervision for semi-supervised tumor segmentation
• A critical look at the current train/test split in machine learning
• Semi-Supervised Training to Improve Player and Ball Detection in Soccer
• SemiFL: Semi-Supervised Federated Learning for Unlabeled Clients with Alternate Training
• GALA: Graph Diffusion-based Alignment with Jigsaw for Source-free Domain Adaptation
• Contrastive Learning with Stronger Augmentations
• MaxMatch: Semi-Supervised Learning with Worst-Case Consistency
• Pretraining Representations for Data-Efficient Reinforcement Learning
• Boosting Facial Expression Recognition by A Semi-Supervised Progressive Teacher
• Improved Training of Physics-Informed Neural Networks with Model Ensembles
• Unsupervised Domain Adaptation for RF-based Gesture Recognition
• Meta-Learned Confidence for Few-shot Learning
• I2F: A Unified Image-to-Feature Approach for Domain Adaptive Semantic Segmentation
• SelfMatch: Combining Contrastive Self-Supervision and Consistency for Semi-Supervised Learning
• CoDA: Contrast-enhanced and Diversity-promoting Data Augmentation for Natural Language Understanding
• Self-Supervised Relational Reasoning for Representation Learning
• Assessing Post-Disaster Damage from Satellite Imagery using Semi-Supervised Learning Techniques
• GCT: Graph Co-Training for Semi-Supervised Few-Shot Learning
• Heri-Graphs: A Workflow of Creating Datasets for Multi-modal Machine Learning on Graphs of Heritage Values and Attributes with Social Media
• OpenMatch: Open-set Consistency Regularization for Semi-supervised Learning with Outliers
• SemiFed: Semi-supervised Federated Learning with Consistency and Pseudo-Labeling
• Semi-FedSER: Semi-supervised Learning for Speech Emotion Recognition On Federated Learning using Multiview Pseudo-Labeling
• SoftPatch+: Fully Unsupervised Anomaly Classification and Segmentation
• Adaptive Betweenness Clustering for Semi-Supervised Domain Adaptation
• Unsupervised Semantic Aggregation and Deformable Template Matching for Semi-Supervised Learning
• GaitFormer: Learning Gait Representations with Noisy Multi-Task Learning
• Single-Temporal Supervised Learning for Universal Remote Sensing Change Detection
• GraFN: Semi-Supervised Node Classification on Graph with Few Labels via Non-Parametric Distribution Assignment
• Understanding Self-Training for Gradual Domain Adaptation
• Diversity Helps: Unsupervised Few-shot Learning via Distribution Shift-based Data Augmentation
• RETRIEVE: Coreset Selection for Efficient and Robust Semi-Supervised Learning
• Entanglement Verification with Deep Semi-supervised Machine Learning
• PseudoAugment: Learning to Use Unlabeled Data for Data Augmentation in Point Clouds
• On Data-Augmentation and Consistency-Based Semi-Supervised Learning
• Semi-supervised learning using teacher-student models for vocal melody extraction
• Artificial and beneficial -- Exploiting artificial images for aerial vehicle detection
• A Deep Model for Partial Multi-Label Image Classification with Curriculum Based Disambiguation
• Duplex Conversation: Towards Human-like Interaction in Spoken Dialogue Systems
• Improving Semi-supervised Federated Learning by Reducing the Gradient Diversity of Models
• Variational Rectification Inference for Learning with Noisy Labels
• Perturbed and Strict Mean Teachers for Semi-supervised Semantic Segmentation
• Statistical and Algorithmic Insights for Semi-supervised Learning with Self-training
• Rethinking Re-Sampling in Imbalanced Semi-Supervised Learning
• Semi-supervised Models are Strong Unsupervised Domain Adaptation Learners
• Beyond Cats and Dogs: Semi-supervised Classification of fuzzy labels with overclustering
• SECANT: Self-Expert Cloning for Zero-Shot Generalization of Visual Policies
• A New Semi-supervised Learning Benchmark for Classifying View and Diagnosing Aortic Stenosis from Echocardiograms
• RDA: Reciprocal Distribution Alignment for Robust Semi-supervised Learning
• Class Gradient Projection For Continual Learning
• Condition-Invariant Semantic Segmentation
• Active Self-Semi-Supervised Learning for Few Labeled Samples
• Fuzzy Overclustering: Semi-Supervised Classification of Fuzzy Labels with Overclustering and Inverse Cross-Entropy
• SimPLE: Similar Pseudo Label Exploitation for Semi-Supervised Classification
• KeepAugment: A Simple Information-Preserving Data Augmentation Approach
• FedCon: A Contrastive Framework for Federated Semi-Supervised Learning
• Consistency-based Semi-supervised Active Learning: Towards Minimizing Labeling Cost
• Federated Stain Normalization for Computational Pathology
• Monocular 3D Detection with Geometric Constraints Embedding and Semi-supervised Training
• Interpolation-based semi-supervised learning for object detection
• Class-Specific Distribution Alignment for Semi-Supervised Medical Image Classification
• Dynamic Sub-graph Distillation for Robust Semi-supervised Continual Learning
• Semi-supervised Open-World Object Detection
• ICE: Inter-instance Contrastive Encoding for Unsupervised Person Re-identification
• Robust Mutual Learning for Semi-supervised Semantic Segmentation
• Narrowing the Gap: Improved Detector Training with Noisy Location Annotations
• Federated Semi-Supervised Learning for COVID Region Segmentation in Chest CT using Multi-National Data from China, Italy, Japan
• More Photos are All You Need: Semi-Supervised Learning for Fine-Grained Sketch Based Image Retrieval
• 3DIoUMatch: Leveraging IoU Prediction for Semi-Supervised 3D Object Detection
• Robust Semi-Supervised Learning for Self-learning Open-World Classes
• Multimodal Semi-Supervised Learning for 3D Objects
• Dynamic Distillation Network for Cross-Domain Few-Shot Recognition with Unlabeled Data
• Multi-Source domain adaptation via supervised contrastive learning and confident consistency regularization
• Ego-Vehicle Action Recognition based on Semi-Supervised Contrastive Learning
• FedSiam: Towards Adaptive Federated Semi-Supervised Learning
• Combating Noise: Semi-supervised Learning by Region Uncertainty Quantification
• It Takes Two to Tango: Mixup for Deep Metric Learning
• Triple Generative Adversarial Networks
• Geometric Understanding of Discriminability and Transferability for Visual Domain Adaptation
• Transferring Relative Monocular Depth to Surgical Vision with Temporal Consistency
• EC-GAN: Low-Sample Classification using Semi-Supervised Algorithms and GANs
• Augmentation Strategies for Learning with Noisy Labels
• FixBi: Bridging Domain Spaces for Unsupervised Domain Adaptation
• The Rich Get Richer: Disparate Impact of Semi-Supervised Learning
• Contrastive Semi-Supervised Learning for 2D Medical Image Segmentation
• Learning Fast Sample Re-weighting Without Reward Data
• Learning to Impute: A General Framework for Semi-supervised Learning
• Semantic keypoint-based pose estimation from single RGB frames
• Comparing Transfer and Meta Learning Approaches on a Unified Few-Shot Classification Benchmark
• Cycle Self-Training for Semi-Supervised Object Detection with Distribution Consistency Reweighting
• Joint User Association and Power Allocation in Heterogeneous Ultra Dense Network via Semi-Supervised Representation Learning
• Semi-Supervised Medical Image Segmentation with Co-Distribution Alignment
• SST: Self-training with Self-adaptive Thresholding for Semi-supervised Learning
• Exponential Moving Average Normalization for Self-supervised and Semi-supervised Learning
• SegMatch: A semi-supervised learning method for surgical instrument segmentation
• AHA! an 'Artificial Hippocampal Algorithm' for Episodic Machine Learning
• Combining Self-Supervised and Supervised Learning with Noisy Labels
• Pseudolabel guided pixels contrast for domain adaptive semantic segmentation
• OpenCoS: Contrastive Semi-supervised Learning for Handling Open-set Unlabeled Data
• Memory-Efficient Semi-Supervised Continual Learning: The World is its Own Replay Buffer
• Reliable Student: Addressing Noise in Semi-Supervised 3D Object Detection
• An Empirical Investigation of Representation Learning for Imitation
• Revisiting consistency for semi-supervised semantic segmentation
• An analysis of over-sampling labeled data in semi-supervised learning with FixMatch
• Cross Branch Feature Fusion Decoder for Consistency Regularization-based Semi-Supervised Change Detection
• Semi-Supervising Learning, Transfer Learning, and Knowledge Distillation with SimCLR
• Boundary-Refined Prototype Generation: A General End-to-End Paradigm for Semi-Supervised Semantic Segmentation
• Extreme Consistency: Overcoming Annotation Scarcity and Domain Shifts
• Elsa: Energy-based learning for semi-supervised anomaly detection
• Adaptive Early-Learning Correction for Segmentation from Noisy Annotations
• Domain Generalization via Semi-supervised Meta Learning
• ReSmooth: Detecting and Utilizing OOD Samples when Training with Data Augmentation
• Recent Deep Semi-supervised Learning Approaches and Related Works
• Towards Few-Annotation Learning for Object Detection: Are Transformer-based Models More Efficient ?
• Self-training Avoids Using Spurious Features Under Domain Shift
• SHOT-VAE: Semi-supervised Deep Generative Models With Label-aware ELBO Approximations
• Order-Guided Disentangled Representation Learning for Ulcerative Colitis Classification with Limited Labels
• FedSEAL: Semi-Supervised Federated Learning with Self-Ensemble Learning and Negative Learning
• BiPC: Bidirectional Probability Calibration for Unsupervised Domain Adaption
• RankingMatch: Delving into Semi-Supervised Learning with Consistency Regularization and Ranking Loss
• Feature Stylization and Domain-aware Contrastive Learning for Domain Generalization
• Injecting Text in Self-Supervised Speech Pretraining
• Built Year Prediction from Buddha Face with Heterogeneous Labels
• Lesion Localization in OCT by Semi-Supervised Object Detection
• Training on Test Data with Bayesian Adaptation for Covariate Shift
• Estimating Physical Information Consistency of Channel Data Augmentation for Remote Sensing Images
• Two-Stage Adaptive Network for Semi-Supervised Cross-Domain Crater Detection under Varying Scenario Distributions
• APP: Adaptive Prototypical Pseudo-Labeling for Few-shot OOD Detection
• Manifold-Aware Self-Training for Unsupervised Domain Adaptation on Regressing 6D Object Pose
• Semi-supervised Long-tailed Recognition using Alternate Sampling
• Semi-supervised Keypoint Localization
• Pseudo-labeling for Scalable 3D Object Detection
• An Empirical Study and Analysis on Open-Set Semi-Supervised Learning
• Does Data Augmentation Benefit from Split BatchNorms
• Boosting Active Learning for Speech Recognition with Noisy Pseudo-labeled Samples
• High-contrast "gaudy" images improve the training of deep neural network models of visual cortex
• Coarse-to-Fine Domain Adaptive Semantic Segmentation with Photometric Alignment and Category-Center Regularization
• Opportunistic hip fracture risk prediction in Men from X-ray: Findings from the Osteoporosis in Men (MrOS) Study
• Semi-supervised Learning by Latent Space Energy-Based Model of Symbol-Vector Coupling
• Beyond Self-Supervision: A Simple Yet Effective Network Distillation Alternative to Improve Backbones
• Learning Invariant Representation with Consistency and Diversity for Semi-supervised Source Hypothesis Transfer
• Learning Temporally Invariant and Localizable Features via Data Augmentation for Video Recognition
• Gradual Domain Adaptation via Self-Training of Auxiliary Models
• Polyphone Disambiguation in Mandarin Chinese with Semi-Supervised Learning
• Cross-Domain Transfer Learning with CoRTe: Consistent and Reliable Transfer from Black-Box to Lightweight Segmentation Model
• Energy-Based Models with Applications to Speech and Language Processing
• The Mathematical Foundations of Manifold Learning
• Semi-Supervised Learning with Data Augmentation for End-to-End ASR
• Co-matching: Combating Noisy Labels by Augmentation Anchoring
• Why Mixup Improves the Model Performance
• Data augmentation with Mobius transformations
• Supervision Accelerates Pre-training in Contrastive Semi-Supervised Learning of Visual Representations
• Exploiting Cross-Modal Prediction and Relation Consistency for Semi-Supervised Image Captioning
• USCL: Pretraining Deep Ultrasound Image Diagnosis Model through Video Contrastive Representation Learning
• One-bit Supervision for Image Classification: Problem, Solution, and Beyond
• MUSCLE: Strengthening Semi-Supervised Learning Via Concurrent Unsupervised Learning Using Mutual Information Maximization
• Unreliable Partial Label Learning with Recursive Separation
• Transfer Learning with Pre-trained Conditional Generative Models
• Scribble2Label: Scribble-Supervised Cell Segmentation via Self-Generating Pseudo-Labels with Consistency
• Target Consistency for Domain Adaptation: when Robustness meets Transferability
• A Unified Contrastive Loss for Self-Training
• Self-supervised Text-to-SQL Learning with Header Alignment Training
• Multimodal Knowledge Expansion
• Certainty Driven Consistency Loss on Multi-Teacher Networks for Semi-Supervised Learning
• Reliable Label Bootstrapping for Semi-Supervised Learning
• Modular Adaptation for Cross-Domain Few-Shot Learning
• ContraCluster: Learning to Classify without Labels by Contrastive Self-Supervision and Prototype-Based Semi-Supervision
• ReRankMatch: Semi-Supervised Learning with Semantics-Oriented Similarity Representation
• Empirical Perspectives on One-Shot Semi-supervised Learning
• On the Marginal Benefit of Active Learning: Does Self-Supervision Eat Its Cake?
• Uncover and Unlearn Nuisances: Agnostic Fully Test-Time Adaptation
• TumorCP: A Simple but Effective Object-Level Data Augmentation for Tumor Segmentation
• GraSSNet: Graph Soft Sensing Neural Networks
• Boosting the Performance of Semi-Supervised Learning with Unsupervised Clustering
• One to Transfer All: A Universal Transfer Framework for Vision Foundation Model with Few Data
• Point Cloud Instance Segmentation with Semi-supervised Bounding-Box Mining
• Semi-Supervised Vision Transformers
• AlphaMatch: Improving Consistency for Semi-supervised Learning with Alpha-divergence
• Train a One-Million-Way Instance Classifier for Unsupervised Visual Representation Learning
• Learning a Discriminant Latent Space with Neural Discriminant Analysis
• Contrastive Vicinal Space for Unsupervised Domain Adaptation
• FROST: Faster and more Robust One-shot Semi-supervised Training
• Active Learning at the ImageNet Scale
• LiDAM: Semi-Supervised Learning with Localized Domain Adaptation and Iterative Matching
• STraTA: Self-Training with Task Augmentation for Better Few-shot Learning
• Peak Detection On Data Independent Acquisition Mass Spectrometry Data With Semisupervised Convolutional Transformers
• Adaptive Hierarchical Similarity Metric Learning with Noisy Labels
• Explicit homography estimation improves contrastive self-supervised learning
• Semi-supervised Learning for Dense Object Detection in Retail Scenes
• Semi-supervised Sound Event Detection using Random Augmentation and Consistency Regularization
• MULTIMODAL ANALYSIS: Informed content estimation and audio source separation
• High-Dimensional Bayesian Optimization via Semi-Supervised Learning with Optimized Unlabeled Data Sampling
• On sensitivity of meta-learning to support data
• Generation and Simulation of Yeast Microscopy Imagery with Deep Learning
• Leveraging Out-of-Distribution Unlabeled Images: Semi-Supervised Semantic Segmentation with an Open-Vocabulary Model
• BiSTF: Bilateral-Branch Self-Training Framework for Semi-Supervised Large-scale Fine-Grained Recognition
• Scalable Consistency Training for Graph Neural Networks via Self-Ensemble Self-Distillation
• CERM: Context-aware Literature-based Discovery via Sentiment Analysis
• Predictions For Pre-training Language Models
• An empirical study of domain-agnostic semi-supervised learning via energy-based models: joint-training and pre-training
• Matching the Clinical Reality: Accurate OCT-Based Diagnosis From Few Labels
• MetaMix: Improved Meta-Learning with Interpolation-based Consistency Regularization
• Revisiting Self-Training for Few-Shot Learning of Language Model
• Semi-supervised learning objectives as log-likelihoods in a generative model of data curation
• Consistency Training of Multi-exit Architectures for Sensor Data
• Covariate Distribution Aware Meta-learning
• Angular Gap: Reducing the Uncertainty of Image Difficulty through Model Calibration
• Attribute Guided Sparse Tensor-Based Model for Person Re-Identification
• FLiText: A Faster and Lighter Semi-Supervised Text Classification with Convolution Networks
• Improve Unsupervised Pretraining for Few-label Transfer
• On-target Adaptation
• Boosting Unconstrained Face Recognition with Auxiliary Unlabeled Data
• Negative Pseudo Labeling using Class Proportion for Semantic Segmentation in Pathology
• Weakly-supervised fire segmentation by visualizing intermediate CNN layers
• TAGLETS: A System for Automatic Semi-Supervised Learning with Auxiliary Data