There Are Many Consistent Explanations of Unlabeled Data: Why You Should Average

Abstract

Presently the most successful approaches to semi-supervised learning are based on consistency regularization, whereby a model is trained to be robust to small perturbations of its inputs and parameters. To understand consistency regularization, we conceptually explore how loss geometry interacts with training procedures. The consistency loss dramatically improves generalization performance over supervised-only training; however, we show that SGD struggles to converge on the consistency loss and continues to make large steps that lead to changes in predictions on the test data. Motivated by these observations, we propose to train consistency-based methods with Stochastic Weight Averaging (SWA), a recent approach which averages weights along the trajectory of SGD with a modified learning rate schedule. We also propose fast-SWA, which further accelerates convergence by averaging multiple points within each cycle of a cyclical learning rate schedule. With weight averaging, we achieve the best known semi-supervised results on CIFAR-10 and CIFAR-100, over many different quantities of labeled training data. For example, we achieve 5.0% error on CIFAR-10 with only 4000 labels, compared to the previous best result in the literature of 6.3%.

Appears at ICLR 2019

Cited by in corpus (57)

• Unsupervised Data Augmentation for Consistency Training
• Interpolation Consistency Training for Semi-Supervised Learning
• Rethinking Pre-training and Self-training
• An Overview of Deep Semi-Supervised Learning
• Self-training with Noisy Student improves ImageNet classification
• A survey on Semi-, Self- and Unsupervised Learning for Image Classification
• PSLA: Improving Audio Tagging with Pretraining, Sampling, Labeling, and Aggregation
• Semi-supervised semantic segmentation needs strong, varied perturbations
• Adversarially Robust Generalization Just Requires More Unlabeled Data
• High-Quality Self-Supervised Deep Image Denoising
• Parting with Illusions about Deep Active Learning
• Learning Invariances in Neural Networks
• A Review of Single-Source Deep Unsupervised Visual Domain Adaptation
• Dual Student: Breaking the Limits of the Teacher in Semi-supervised Learning
• Semi-Supervised Learning by Label Gradient Alignment
• EnAET: A Self-Trained framework for Semi-Supervised and Supervised Learning with Ensemble Transformations
• Guided Collaborative Training for Pixel-wise Semi-Supervised Learning
• Stochastic Weight Averaging in Parallel: Large-Batch Training that Generalizes Well
• FedCon: A Contrastive Framework for Federated Semi-Supervised Learning
• Energy Models for Better Pseudo-Labels: Improving Semi-Supervised Classification with the 1-Laplacian Graph Energy
• Consistency-based Semi-supervised Active Learning: Towards Minimizing Labeling Cost
• Proposal Learning for Semi-Supervised Object Detection
• Run Away From your Teacher: Understanding BYOL by a Novel Self-Supervised Approach
• Exploring Self-Supervised Regularization for Supervised and Semi-Supervised Learning
• FedSiam: Towards Adaptive Federated Semi-Supervised Learning
• Triple Generative Adversarial Networks
• Learning to Impute: A General Framework for Semi-supervised Learning
• Exponential Moving Average Normalization for Self-supervised and Semi-supervised Learning
• Self-supervised Semi-supervised Learning for Data Labeling and Quality Evaluation
• Semi-Supervised Learning with Normalizing Flows
• Dangers of Bayesian Model Averaging under Covariate Shift
• Density-Aware Graph for Deep Semi-Supervised Visual Recognition
• Countering Noisy Labels By Learning From Auxiliary Clean Labels
• Consistency Regularization for Cross-Lingual Fine-Tuning
• Learning to Learn in a Semi-Supervised Fashion
• SHOT-VAE: Semi-supervised Deep Generative Models With Label-aware ELBO Approximations
• Recent Deep Semi-supervised Learning Approaches and Related Works
• Manifold Graph with Learned Prototypes for Semi-Supervised Image Classification
• SSMD: Semi-Supervised Medical Image Detection with Adaptive Consistency and Heterogeneous Perturbation
• Are you Struggling? Dataset and Baselines for Struggle Determination in Assembly Videos
• Semixup: In- and Out-of-Manifold Regularization for Deep Semi-Supervised Knee Osteoarthritis Severity Grading from Plain Radiographs
• Gradient-based Data Augmentation for Semi-Supervised Learning
• Learning with Multiplicative Perturbations
• Out-distribution aware Self-training in an Open World Setting
• Detecting and Understanding Generalization Barriers for Neural Machine Translation
• Certainty Driven Consistency Loss on Multi-Teacher Networks for Semi-Supervised Learning
• ROI Regularization for Semi-supervised and Supervised Learning
• Semi-supervised Learning of Fetal Anatomy from Ultrasound
• MvSR-NAT: Multi-view Subset Regularization for Non-Autoregressive Machine Translation
• Robust Temporal Ensembling for Learning with Noisy Labels
• Balanced Masked and Standard Face Recognition
• End-To-End Graph-based Deep Semi-Supervised Learning
• Custom Object Detection via Multi-Camera Self-Supervised Learning
• MisMatch: Calibrated Segmentation via Consistency on Differential Morphological Feature Perturbations with Limited Labels
• ARCH: Efficient Adversarial Regularized Training with Caching
• Regularization with Latent Space Virtual Adversarial Training
• Semi-Supervised Learning with Meta-Gradient