Temporal Ensembling for Semi-Supervised Learning

Abstract

In this paper, we present a simple and efficient method for training deep neural networks in a semi-supervised setting where only a small portion of training data is labeled. We introduce self-ensembling, where we form a consensus prediction of the unknown labels using the outputs of the network-in-training on different epochs, and most importantly, under different regularization and input augmentation conditions. This ensemble prediction can be expected to be a better predictor for the unknown labels than the output of the network at the most recent training epoch, and can thus be used as a target for training. Using our method, we set new records for two standard semi-supervised learning benchmarks, reducing the (non-augmented) classification error rate from 18.44% to 7.05% in SVHN with 500 labels and from 18.63% to 16.55% in CIFAR-10 with 4000 labels, and further to 5.12% and 12.16% by enabling the standard augmentations. We additionally obtain a clear improvement in CIFAR-100 classification accuracy by using random images from the Tiny Images dataset as unlabeled extra inputs during training. Finally, we demonstrate good tolerance to incorrect labels.

Final ICLR 2017 version. Includes new results for CIFAR-100 with additional unlabeled data from Tiny Images dataset

References in corpus (14)

• Adam: A Method for Stochastic Optimization
• Distilling the Knowledge in a Neural Network
• Striving for Simplicity: The All Convolutional Net
• Improved Techniques for Training GANs
• Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
• Training Deep Neural Networks on Noisy Labels with Bootstrapping
• Semi-Supervised Learning with Generative Adversarial Networks
• Training Convolutional Networks with Noisy Labels
• Fractional Max-Pooling
• Deep Networks with Stochastic Depth
• Auxiliary Deep Generative Models
• Swapout: Learning an ensemble of deep architectures
• Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approach
• Generalizing Pooling Functions in Convolutional Neural Networks: Mixed, Gated, and Tree

Cited by in corpus (252)

• Bootstrap your own latent: A new approach to self-supervised Learning
• Ensemble deep learning: A review
• Diffusion Models in Vision: A Survey
• Unsupervised Data Augmentation for Consistency Training
• ADVENT: Adversarial Entropy Minimization for Domain Adaptation in Semantic Segmentation
• Interpolation Consistency Training for Semi-Supervised Learning
• AutoAugment: Learning Augmentation Policies from Data
• Big Self-Supervised Models are Strong Semi-Supervised Learners
• Shape-aware Semi-supervised 3D Semantic Segmentation for Medical Images
• A DIRT-T Approach to Unsupervised Domain Adaptation
• A Comprehensive Survey on Deep Graph Representation Learning
• A Simple Semi-Supervised Learning Framework for Object Detection
• Uncertainty-aware multi-view co-training for semi-supervised medical image segmentation and domain adaptation
• Good Semi-supervised Learning that Requires a Bad GAN
• Triple Generative Adversarial Nets
• A survey on Semi-, Self- and Unsupervised Learning for Image Classification
• Rethinking the Value of Labels for Improving Class-Imbalanced Learning
• SPICE: Semantic Pseudo-labeling for Image Clustering
• InfoGraph: Unsupervised and Semi-supervised Graph-Level Representation Learning via Mutual Information Maximization
• One Million Scenes for Autonomous Driving: ONCE Dataset
• Image Augmentations for GAN Training
• Semi-supervised Learning with GANs: Manifold Invariance with Improved Inference
• Memory-aware curriculum federated learning for breast cancer classification
• Unsupervised Scene Adaptation with Memory Regularization in vivo
• Model Adaptation: Historical Contrastive Learning for Unsupervised Domain Adaptation without Source Data
• LE-UDA: Label-efficient unsupervised domain adaptation for medical image segmentation
• Selfie: Self-supervised Pretraining for Image Embedding
• Uncertainty-Aware Consistency Regularization for Cross-Domain Semantic Segmentation
• Towards Label-efficient Automatic Diagnosis and Analysis: A Comprehensive Survey of Advanced Deep Learning-based Weakly-supervised, Semi-supervised and Self-supervised Techniques in Histopathological Image Analysis
• PARSE: Pairwise Alignment of Representations in Semi-Supervised EEG Learning for Emotion Recognition
• Robust Self-Ensembling Network for Hyperspectral Image Classification
• Unlabeled Data Improves Adversarial Robustness
• Self-paced and self-consistent co-training for semi-supervised image segmentation
• FDA: Fourier Domain Adaptation for Semantic Segmentation
• Towards Utilizing Unlabeled Data in Federated Learning: A Survey and Prospective
• A Survey of Deep Learning for Scientific Discovery
• PMU measurements based short-term voltage stability assessment of power systems via deep transfer learning
• SELF: Learning to Filter Noisy Labels with Self-Ensembling
• OSLNet: Deep Small-Sample Classification with an Orthogonal Softmax Layer
• Unsupervised Person Re-identification with Stochastic Training Strategy
• Improving the Improved Training of Wasserstein GANs: A Consistency Term and Its Dual Effect
• Data-Efficient Reinforcement Learning with Self-Predictive Representations
• MutexMatch: Semi-Supervised Learning with Mutex-Based Consistency Regularization
• Open-World Semi-Supervised Learning
• Consistency Regularization for Generative Adversarial Networks
• Structured Consistency Loss for semi-supervised semantic segmentation
• Denoising Multi-Source Weak Supervision for Neural Text Classification
• Generative Artificial Intelligence Meets Synthetic Aperture Radar: A Survey
• Enhancing Diversity in Teacher-Student Networks via Asymmetric branches for Unsupervised Person Re-identification
• MT-UDA: Towards Unsupervised Cross-modality Medical Image Segmentation with Limited Source Labels
• Class-Imbalanced Semi-Supervised Learning
• Continual Semi-Supervised Learning through Contrastive Interpolation Consistency
• SemiFL: Semi-Supervised Federated Learning for Unlabeled Clients with Alternate Training
• CReST: A Class-Rebalancing Self-Training Framework for Imbalanced Semi-Supervised Learning
• GraphXCOVID: Explainable Deep Graph Diffusion Pseudo-Labelling for Identifying COVID-19 on Chest X-rays
• Exploring Feature Representation Learning for Semi-supervised Medical Image Segmentation
• KGNN: Harnessing Kernel-based Networks for Semi-supervised Graph Classification
• Towards Better Accuracy-efficiency Trade-offs: Divide and Co-training
• Boosting Facial Expression Recognition by A Semi-Supervised Progressive Teacher
• Feature Fusion for Online Mutual Knowledge Distillation
• Self-Ensembling with GAN-based Data Augmentation for Domain Adaptation in Semantic Segmentation
• Deep Two-path Semi-supervised Learning for Fake News Detection
• Open-set Label Noise Can Improve Robustness Against Inherent Label Noise
• Boosting RGB-D Saliency Detection by Leveraging Unlabeled RGB Images
• Improved Consistency Regularization for GANs
• Learning Invariances in Neural Networks
• GAR: An efficient and scalable Graph-based Activity Regularization for semi-supervised learning
• Semi-Supervised Building Footprint Generation with Feature and Output Consistency Training
• Smooth Neighbors on Teacher Graphs for Semi-supervised Learning
• Structured Generative Adversarial Networks
• Towards Discriminability and Diversity: Batch Nuclear-norm Maximization under Label Insufficient Situations
• Self-Ensembling GAN for Cross-Domain Semantic Segmentation
• MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classification
• A Prototype-Oriented Framework for Unsupervised Domain Adaptation
• Efficient Semi-Supervised Gross Target Volume of Nasopharyngeal Carcinoma Segmentation via Uncertainty Rectified Pyramid Consistency
• Quantitative Phase Imaging and Artificial Intelligence: A Review
• OpenMatch: Open-set Consistency Regularization for Semi-supervised Learning with Outliers
• Regularizing Deep Networks with Semantic Data Augmentation
• SemiCurv: Semi-Supervised Curvilinear Structure Segmentation
• Learning Kernel for Conditional Moment-Matching Discrepancy-based Image Classification
• SALUDA: Surface-based Automotive Lidar Unsupervised Domain Adaptation
• Semi-supervised Left Atrium Segmentation with Mutual Consistency Training
• Unsupervised Semantic Aggregation and Deformable Template Matching for Semi-Supervised Learning
• Connecting Low-Loss Subspace for Personalized Federated Learning
• Cluster Alignment with a Teacher for Unsupervised Domain Adaptation
• Macroscale fracture surface segmentation via semi-supervised learning considering the structural similarity
• Uncertain Label Correction via Auxiliary Action Unit Graphs for Facial Expression Recognition
• Semi-supervised 3D Object Detection via Adaptive Pseudo-Labeling
• RDIS: Random Drop Imputation with Self-Training for Incomplete Time Series Data
• Momentum Contrastive Voxel-wise Representation Learning for Semi-supervised Volumetric Medical Image Segmentation
• On Data-Augmentation and Consistency-Based Semi-Supervised Learning
• EnAET: A Self-Trained framework for Semi-Supervised and Supervised Learning with Ensemble Transformations
• Beyond Cats and Dogs: Semi-supervised Classification of fuzzy labels with overclustering
• Dense Supervision Propagation for Weakly Supervised Semantic Segmentation on 3D Point Clouds
• Auxiliary Task Reweighting for Minimum-data Learning
• Cross-Referencing Self-Training Network for Sound Event Detection in Audio Mixtures
• A Comparative Review of Recent Few-Shot Object Detection Algorithms
• Adaptive Self-Training for Object Detection
• Fuzzy Overclustering: Semi-Supervised Classification of Fuzzy Labels with Overclustering and Inverse Cross-Entropy
• Multi-organ segmentation: a progressive exploration of learning paradigms under scarce annotation
• Interpolation-based semi-supervised learning for object detection
• Contrastive and Generative Graph Convolutional Networks for Graph-based Semi-Supervised Learning
• Face Detection in the Operating Room: Comparison of State-of-the-art Methods and a Self-supervised Approach
• Deep Metric Transfer for Label Propagation with Limited Annotated Data
• Deep Co-Training for Semi-Supervised Image Segmentation
• Robust Mutual Learning for Semi-supervised Semantic Segmentation
• One Thing One Click: A Self-Training Approach for Weakly Supervised 3D Semantic Segmentation
• A Multi-grained based Attention Network for Semi-supervised Sound Event Detection
• Semi-Supervised Brain Lesion Segmentation with an Adapted Mean Teacher Model
• Proposal Learning for Semi-Supervised Object Detection
• A Semi-Supervised Two-Stage Approach to Learning from Noisy Labels
• Robust Semi-Supervised Learning for Self-learning Open-World Classes
• Neural Semi-supervised Learning for Text Classification Under Large-Scale Pretraining
• 3DIoUMatch: Leveraging IoU Prediction for Semi-Supervised 3D Object Detection
• Ego-Vehicle Action Recognition based on Semi-Supervised Contrastive Learning
• ACT-Net: Asymmetric Co-Teacher Network for Semi-supervised Memory-efficient Medical Image Segmentation
• FedSiam: Towards Adaptive Federated Semi-Supervised Learning
• Run Away From your Teacher: Understanding BYOL by a Novel Self-Supervised Approach
• Triple Generative Adversarial Networks
• Self-supervised Mean Teacher for Semi-supervised Chest X-ray Classification
• Split Batch Normalization: Improving Semi-Supervised Learning under Domain Shift
• Federated Semi-Supervised Learning with Class Distribution Mismatch
• FeatMatch: Feature-Based Augmentation for Semi-Supervised Learning
• Advances in Medical Image Segmentation: A Comprehensive Survey with a Focus on Lumbar Spine Applications
• Self-Supervised Learning for Semi-Supervised Temporal Action Proposal
• SST: Self-training with Self-adaptive Thresholding for Semi-supervised Learning
• Supervision by Denoising for Medical Image Segmentation
• Adaptive Consistency Regularization for Semi-Supervised Transfer Learning
• Learning the Precise Feature for Cluster Assignment
• Leveraging Unlabelled Data in Multiple-Instance Learning Problems for Improved Detection of Parkinsonian Tremor in Free-Living Conditions
• Deep Semi-Supervised Learning for Time Series Classification
• CORE-ReID: Comprehensive Optimization and Refinement through Ensemble fusion in Domain Adaptation for person re-identification
• Learning from Label Proportions with Consistency Regularization
• A Two-Stream Mutual Attention Network for Semi-supervised Biomedical Segmentation with Noisy Labels
• Semi-Supervised Learning with Normalizing Flows
• Every Node Counts: Self-Ensembling Graph Convolutional Networks for Semi-Supervised Learning
• MetaMixUp: Learning Adaptive Interpolation Policy of MixUp with Meta-Learning
• An analysis of over-sampling labeled data in semi-supervised learning with FixMatch
• Knowledge Distillation with Adaptive Asymmetric Label Sharpening for Semi-supervised Fracture Detection in Chest X-rays
• Extreme Consistency: Overcoming Annotation Scarcity and Domain Shifts
• Boundary-Refined Prototype Generation: A General End-to-End Paradigm for Semi-Supervised Semantic Segmentation
• Recent Deep Semi-supervised Learning Approaches and Related Works
• From Synthetic to Real: Image Dehazing Collaborating with Unlabeled Real Data
• Manifold Adversarial Learning
• Towards Few-Annotation Learning for Object Detection: Are Transformer-based Models More Efficient ?
• AVT: Unsupervised Learning of Transformation Equivariant Representations by Autoencoding Variational Transformations
• Data-Uncertainty Guided Multi-Phase Learning for Semi-Supervised Object Detection
• Exploiting Shared Knowledge from Non-COVID Lesions for Annotation-Efficient COVID-19 CT Lung Infection Segmentation
• Robust Training of Social Media Image Classification Models for Rapid Disaster Response
• Bridge the Domain Gap Between Ultra-wide-field and Traditional Fundus Images via Adversarial Domain Adaptation
• Self-mentoring: a new deep learning pipeline to train a self-supervised U-net for few-shot learning of bio-artificial capsule segmentation
• RankingMatch: Delving into Semi-Supervised Learning with Consistency Regularization and Ranking Loss
• Semi-Supervised Learning Enabled by Multiscale Deep Neural Network Inversion
• Semi-Supervised and Active Few-Shot Learning with Prototypical Networks
• Semi-supervised Long-tailed Recognition using Alternate Sampling
• 3D Human Shape and Pose from a Single Low-Resolution Image with Self-Supervised Learning
• Pseudo-labeling for Scalable 3D Object Detection
• One-bit Supervision for Image Classification
• 3D Human Pose, Shape and Texture from Low-Resolution Images and Videos
• Optimizing machine learning methods to discover strong gravitational lenses in the Deep Lens Survey
• Boosting Active Learning for Speech Recognition with Noisy Pseudo-labeled Samples
• Improving Adversarial Robustness via Unlabeled Out-of-Domain Data
• CADA: Multi-scale Collaborative Adversarial Domain Adaptation for Unsupervised Optic Disc and Cup Segmentation
• Semi-supervised Learning by Latent Space Energy-Based Model of Symbol-Vector Coupling
• How Out-of-Distribution Data Hurts Semi-Supervised Learning
• Semi-supervised Learning using Adversarial Training with Good and Bad Samples
• Co-matching: Combating Noisy Labels by Augmentation Anchoring
• Semi-Supervised Learning for Fetal Brain MRI Quality Assessment with ROI consistency
• ERNIE-Tiny : A Progressive Distillation Framework for Pretrained Transformer Compression
• DS-Net++: Dynamic Weight Slicing for Efficient Inference in CNNs and Transformers
• Learning Invariant Representation with Consistency and Diversity for Semi-supervised Source Hypothesis Transfer
• Efficient Bidirectional Neural Machine Translation
• A Simple yet Effective Baseline for Robust Deep Learning with Noisy Labels
• CUDA: Contradistinguisher for Unsupervised Domain Adaptation
• Explore the Potential Performance of Vision-and-Language Navigation Model: a Snapshot Ensemble Method
• Auto-Annotation Quality Prediction for Semi-Supervised Learning with Ensembles
• Energy-Based Models with Applications to Speech and Language Processing
• Cell Type Identification from Single-Cell Transcriptomic Data via Semi-supervised Learning
• Robust Deep Learning with Active Noise Cancellation for Spatial Computing
• Optimally Combining Classifiers for Semi-Supervised Learning
• SSMD: Semi-Supervised Medical Image Detection with Adaptive Consistency and Heterogeneous Perturbation
• One-bit Supervision for Image Classification: Problem, Solution, and Beyond
• Automatic CT Segmentation from Bounding Box Annotations using Convolutional Neural Networks
• Deep Recurrent Semi-Supervised EEG Representation Learning for Emotion Recognition
• Progressive Representative Labeling for Deep Semi-Supervised Learning
• Semi-supervised learning for medical image classification using imbalanced training data
• Multi-Feature Semi-Supervised Learning for COVID-19 Diagnosis from Chest X-ray Images
• A*HAR: A New Benchmark towards Semi-supervised learning for Class-imbalanced Human Activity Recognition
• Learning to Detect Important People in Unlabelled Images for Semi-supervised Important People Detection
• Metric learning by Similarity Network for Deep Semi-Supervised Learning
• Non-technical Loss Detection with Statistical Profile Images Based on Semi-supervised Learning
• Target Consistency for Domain Adaptation: when Robustness meets Transferability
• Temporal Self-Ensembling Teacher for Semi-Supervised Object Detection
• Consistency Regularization with Generative Adversarial Networks for Semi-Supervised Learning
• Local Clustering with Mean Teacher for Semi-supervised Learning
• Semi-Supervised Learning for In-Game Expert-Level Music-to-Dance Translation
• Mixup-breakdown: a consistency training method for improving generalization of speech separation models
• SSAH: Semi-supervised Adversarial Deep Hashing with Self-paced Hard Sample Generation
• Reliable Semi-Supervised Learning when Labels are Missing at Random
• A Comprehensive Approach to Unsupervised Embedding Learning based on AND Algorithm
• SaaS: Speed as a Supervisor for Semi-supervised Learning
• Label Prediction Framework for Semi-Supervised Cross-Modal Retrieval
• ContraCluster: Learning to Classify without Labels by Contrastive Self-Supervision and Prototype-Based Semi-Supervision
• Temporal Knowledge Consistency for Unsupervised Visual Representation Learning
• Improving Localization for Semi-Supervised Object Detection
• Contrastive Learning and Self-Training for Unsupervised Domain Adaptation in Semantic Segmentation
• Semi-Supervised Learning with IPM-based GANs: an Empirical Study
• Hierarchical Deep Network with Uncertainty-aware Semi-supervised Learning for Vessel Segmentation
• MP2: A Momentum Contrast Approach for Recommendation with Pointwise and Pairwise Learning
• Improving Object Detection with Selective Self-supervised Self-training
• Boosting Semi-Supervised Face Recognition with Noise Robustness
• Active Crowd Counting with Limited Supervision
• Patch-level Neighborhood Interpolation: A General and Effective Graph-based Regularization Strategy
• Medical Image Segmentation with Limited Supervision: A Review of Deep Network Models
• Learning Temporal Action Proposals With Fewer Labels
• Consistent Posterior Distributions under Vessel-Mixing: A Regularization for Cross-Domain Retinal Artery/Vein Classification
• Semi-supervised Learning for COVID-19 Image Classification via ResNet
• Attribute-Induced Bias Eliminating for Transductive Zero-Shot Learning
• AlphaMatch: Improving Consistency for Semi-supervised Learning with Alpha-divergence
• Matching Distributions via Optimal Transport for Semi-Supervised Learning
• Ensemble Learning with Manifold-Based Data Splitting for Noisy Label Correction
• Weakly Supervised Person Re-Identification
• A Review on Semi-Supervised Relation Extraction
• Point Cloud Instance Segmentation with Semi-supervised Bounding-Box Mining
• Unsupervised End-to-End Training with a Self-Defined Target
• Robust Collaborative Learning with Noisy Labels
• A Self-Supervised Bootstrap Method for Single-Image 3D Face Reconstruction
• Pairwise Teacher-Student Network for Semi-Supervised Hashing
• Domain Constraint Approximation based Semi Supervision
• Dual-Teacher: Integrating Intra-domain and Inter-domain Teachers for Annotation-efficient Cardiac Segmentation
• Semi-Supervised Learning for Bone Mineral Density Estimation in Hip X-ray Images
• OXnet: Omni-supervised Thoracic Disease Detection from Chest X-rays
• Scalable Semi-supervised Landmark Localization for X-ray Images using Few-shot Deep Adaptive Graph
• Learn to Resolve Conversational Dependency: A Consistency Training Framework for Conversational Question Answering
• Dual-Teacher++: Exploiting Intra-domain and Inter-domain Knowledge with Reliable Transfer for Cardiac Segmentation
• Understanding Classifier Mistakes with Generative Models
• On Model Robustness Against Adversarial Examples
• Snowball: Iterative Model Evolution and Confident Sample Discovery for Semi-Supervised Learning on Very Small Labeled Datasets
• Unsupervised Image Segmentation using Mutual Mean-Teaching
• LiDAM: Semi-Supervised Learning with Localized Domain Adaptation and Iterative Matching
• Investigating Semi-Supervised Learning Algorithms in Text Datasets
• CFEA: Collaborative Feature Ensembling Adaptation for Domain Adaptation in Unsupervised Optic Disc and Cup Segmentation
• On-target Adaptation
• Self Training with Ensemble of Teacher Models
• Skeptical Deep Learning with Distribution Correction
• BiSTF: Bilateral-Branch Self-Training Framework for Semi-Supervised Large-scale Fine-Grained Recognition
• Long Short-Term Sample Distillation
• Mitigating Memorization in Sample Selection for Learning with Noisy Labels
• AMC-Loss: Angular Margin Contrastive Loss for Improved Explainability in Image Classification
• Teacher-Student Competition for Unsupervised Domain Adaptation
• High-Dimensional Bayesian Optimization via Semi-Supervised Learning with Optimized Unlabeled Data Sampling
• VeniBot: Towards Autonomous Venipuncture with Semi-supervised Vein Segmentation from Ultrasound Images