In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learning

Abstract

The recent research in semi-supervised learning (SSL) is mostly dominated by consistency regularization based methods which achieve strong performance. However, they heavily rely on domain-specific data augmentations, which are not easy to generate for all data modalities. Pseudo-labeling (PL) is a general SSL approach that does not have this constraint but performs relatively poorly in its original formulation. We argue that PL underperforms due to the erroneous high confidence predictions from poorly calibrated models; these predictions generate many incorrect pseudo-labels, leading to noisy training. We propose an uncertainty-aware pseudo-label selection (UPS) framework which improves pseudo labeling accuracy by drastically reducing the amount of noise encountered in the training process. Furthermore, UPS generalizes the pseudo-labeling process, allowing for the creation of negative pseudo-labels; these negative pseudo-labels can be used for multi-label classification as well as negative learning to improve the single-label classification. We achieve strong performance when compared to recent SSL methods on the CIFAR-10 and CIFAR-100 datasets. Also, we demonstrate the versatility of our method on the video dataset UCF-101 and the multi-label dataset Pascal VOC.

arXiv commentICLR 2021

References in corpus (12)

• A Simple Framework for Contrastive Learning of Visual Representations
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Prototypical Networks for Few-shot Learning
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Improved Regularization of Convolutional Neural Networks with Cutout
• FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
• Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
• Semi-Supervised Learning with Deep Generative Models
• DropBlock: A regularization method for convolutional networks
• Variational Autoencoder for Deep Learning of Images, Labels and Captions
• Shake-Shake regularization
• Structured and Efficient Variational Deep Learning with Matrix Gaussian Posteriors

Cited by in corpus (6)

• Energy Models for Better Pseudo-Labels: Improving Semi-Supervised Classification with the 1-Laplacian Graph Energy
• Learning from Temporal Gradient for Semi-supervised Action Recognition
• Semi-Supervised Hypothesis Transfer for Source-Free Domain Adaptation
• Self-training Converts Weak Learners to Strong Learners in Mixture Models
• One-bit Supervision for Image Classification: Problem, Solution, and Beyond
• Combining Diverse Feature Priors