Realistic Evaluation of Deep Semi-Supervised Learning Algorithms

Abstract

Semi-supervised learning (SSL) provides a powerful framework for leveraging unlabeled data when labels are limited or expensive to obtain. SSL algorithms based on deep neural networks have recently proven successful on standard benchmark tasks. However, we argue that these benchmarks fail to address many issues that these algorithms would face in real-world applications. After creating a unified reimplementation of various widely-used SSL techniques, we test them in a suite of experiments designed to address these issues. We find that the performance of simple baselines which do not use unlabeled data is often underreported, that SSL methods differ in sensitivity to the amount of labeled and unlabeled data, and that performance can degrade substantially when the unlabeled dataset contains out-of-class examples. To help guide SSL research towards real-world applicability, we make our unified reimplemention and evaluation platform publicly available.

References in corpus (9)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• How transferable are features in deep neural networks?
• Improved Regularization of Convolutional Neural Networks with Cutout
• Layer Normalization
• Regularizing Neural Networks by Penalizing Confident Output Distributions
• Variational Autoencoder for Deep Learning of Images, Labels and Captions
• A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets
• Learning with Pseudo-Ensembles
• Shake-Shake regularization

Cited by in corpus (45)

• AutoAugment: Learning Augmentation Policies from Data
• Multi-task Self-Supervised Learning for Human Activity Detection
• True Few-Shot Learning with Language Models
• Unsupervised Learning via Meta-Learning
• TensorFuzz: Debugging Neural Networks with Coverage-Guided Fuzzing
• Self-Training: A Survey
• Artificial Neural Networks for Photonic Applications: From Algorithms to Implementation
• Towards Utilizing Unlabeled Data in Federated Learning: A Survey and Prospective
• Zero-shot Knowledge Transfer via Adversarial Belief Matching
• Mitigating Overfitting in Supervised Classification from Two Unlabeled Datasets: A Consistent Risk Correction Approach
• Generalized Jensen-Shannon Divergence Loss for Learning with Noisy Labels
• Lifting 2D Human Pose to 3D with Domain Adapted 3D Body Concept
• A Framework for Cluster and Classifier Evaluation in the Absence of Reference Labels
• Semi-supervised Rare Disease Detection Using Generative Adversarial Network
• Multi-class Classification without Multi-class Labels
• Semi-supervised classification of radiology images with NoTeacher: A Teacher that is not Mean
• Hitting the Target: Stopping Active Learning at the Cost-Based Optimum
• Dynamic Sub-graph Distillation for Robust Semi-supervised Continual Learning
• Multimodal Semi-Supervised Learning for 3D Objects
• Improving Fingerprint Pore Detection with a Small FCN
• LAMDA-SSL: Semi-Supervised Learning in Python
• Revisiting consistency for semi-supervised semantic segmentation
• OpenCoS: Contrastive Semi-supervised Learning for Handling Open-set Unlabeled Data
• An analysis of over-sampling labeled data in semi-supervised learning with FixMatch
• Recent Deep Semi-supervised Learning Approaches and Related Works
• Negative sampling in semi-supervised learning
• Manifold Graph with Learned Prototypes for Semi-Supervised Image Classification
• Gotta Adapt 'Em All: Joint Pixel and Feature-Level Domain Adaptation for Recognition in the Wild
• StackMix: A complementary Mix algorithm
• Automatic Dataset Annotation to Learn CNN Pore Description for Fingerprint Recognition
• Adversarial Transformations for Semi-Supervised Learning
• Multi-Feature Semi-Supervised Learning for COVID-19 Diagnosis from Chest X-ray Images
• The Effectiveness of Multitask Learning for Phenotyping with Electronic Health Records Data
• Deep Recurrent Semi-Supervised EEG Representation Learning for Emotion Recognition
• Consistency Regularization Can Improve Robustness to Label Noise
• Reliable Label Bootstrapping for Semi-Supervised Learning
• Towards Robust Pattern Recognition: A Review
• Credal Self-Supervised Learning
• Learning Consistent Deep Generative Models from Sparse Data via Prediction Constraints
• Industry Scale Semi-Supervised Learning for Natural Language Understanding
• Detecting Anomalies Through Contrast in Heterogeneous Data
• Efficient Semi-Supervised Learning for Natural Language Understanding by Optimizing Diversity
• Mixed integer programming formulation of unsupervised learning
• End-To-End Graph-based Deep Semi-Supervised Learning
• TzK: Flow-Based Conditional Generative Model