Self-Training: A Survey

Abstract

Semi-supervised algorithms aim to learn prediction functions from a small set of labeled observations and a large set of unlabeled observations. Because this framework is relevant in many applications, they have received a lot of interest in both academia and industry. Among the existing techniques, self-training methods have undoubtedly attracted greater attention in recent years. These models are designed to find the decision boundary on low density regions without making additional assumptions about the data distribution, and use the unsigned output score of a learned classifier, or its margin, as an indicator of confidence. The working principle of self-training algorithms is to learn a classifier iteratively by assigning pseudo-labels to the set of unlabeled training samples with a margin greater than a certain threshold. The pseudo-labeled examples are then used to enrich the labeled training data and to train a new classifier in conjunction with the labeled training set. In this paper, we present self-training methods for binary and multi-class classification; as well as their variants and two related approaches, namely consistency-based approaches and transductive learning. We examine the impact of significant self-training features on various methods, using different general and image classification benchmarks, and we discuss our ideas for future research in self-training. To the best of our knowledge, this is the first thorough and complete survey on this subject.

43 pages, 1 figure

References in corpus (29)

• Semi-Supervised Learning with Deep Generative Models
• A Survey on Deep Semi-supervised Learning
• A Comprehensive Survey of Image Augmentation Techniques for Deep Learning
• FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling
• Recovery Guarantees for One-hidden-layer Neural Networks
• Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approach
• Generalization error bounds in semi-supervised classification under the cluster assumption
• Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Data
• Semi-Supervised Semantic Segmentation with Cross Pseudo Supervision
• Explicit Learning Curves for Transduction and Application to Clustering and Compression Algorithms
• Predicting What You Already Know Helps: Provable Self-Supervised Learning
• Corrupted Image Modeling for Self-Supervised Visual Pre-Training
• Learning from Future: A Novel Self-Training Framework for Semantic Segmentation
• Understanding Self-Training for Gradual Domain Adaptation
• MIST: Multiple Instance Self-Training Framework for Video Anomaly Detection
• Self-training and Pre-training are Complementary for Speech Recognition
• Know Your Self-supervised Learning: A Survey on Image-based Generative and Discriminative Training
• DTG-SSOD: Dense Teacher Guidance for Semi-Supervised Object Detection
• Text Classification Using Label Names Only: A Language Model Self-Training Approach
• Multi-Task Self-Training for Learning General Representations
• BoxTeacher: Exploring High-Quality Pseudo Labels for Weakly Supervised Instance Segmentation
• Self-training Converts Weak Learners to Strong Learners in Mixture Models
• Unbiased Multiple Instance Learning for Weakly Supervised Video Anomaly Detection
• Making More of Little Data: Improving Low-Resource Automatic Speech Recognition Using Data Augmentation
• STAD: Self-Training with Ambiguous Data for Low-Resource Relation Extraction
• Self-Training of Halfspaces with Generalization Guarantees under Massart Mislabeling Noise Model
• Cost-Sensitive Self-Training for Optimizing Non-Decomposable Metrics
• Leveraging Ensemble Diversity for Robust Self-Training in the Presence of Sample Selection Bias
• Joint Speech Transcription and Translation: Pseudo-Labeling with Out-of-Distribution Data