One-bit Supervision for Image Classification: Problem, Solution, and Beyond

Abstract

This paper presents one-bit supervision, a novel setting of learning with fewer labels, for image classification. Instead of training model using the accurate label of each sample, our setting requires the model to interact with the system by predicting the class label of each sample and learn from the answer whether the guess is correct, which provides one bit (yes or no) of information. An intriguing property of the setting is that the burden of annotation largely alleviates in comparison to offering the accurate label. There are two keys to one-bit supervision, which are (i) improving the guess accuracy and (ii) making good use of the incorrect guesses. To achieve these goals, we propose a multi-stage training paradigm and incorporate negative label suppression into an off-the-shelf semi-supervised learning algorithm. Theoretical analysis shows that one-bit annotation is more efficient than full-bit annotation in most cases and gives the conditions of combining our approach with active learning. Inspired by this, we further integrate the one-bit supervision framework into the self-supervised learning algorithm which yields an even more efficient training schedule. Different from training from scratch, when self-supervised learning is used for initialization, both hard example mining and class balance are verified effective in boosting the learning performance. However, these two frameworks still need full-bit labels in the initial stage. To cast off this burden, we utilize unsupervised domain adaptation to train the initial model and conduct pure one-bit annotations on the target dataset. In multiple benchmarks, the learning efficiency of the proposed approach surpasses that using full-bit, semi-supervised supervision.

arXiv commentACM TOMM. arXiv admin note: text overlap with arXiv:2009.06168

References in corpus (24)

• Distilling the Knowledge in a Neural Network
• A Simple Framework for Contrastive Learning of Visual Representations
• FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
• Improved Baselines with Momentum Contrastive Learning
• FitNets: Hints for Thin Deep Nets
• Unsupervised Data Augmentation for Consistency Training
• Unsupervised Representation Learning by Predicting Image Rotations
• Temporal Ensembling for Semi-Supervised Learning
• Active Learning for Convolutional Neural Networks: A Core-Set Approach
• Cost-Effective Active Learning for Deep Image Classification
• Deep Bayesian Active Learning with Image Data
• Bayesian Active Learning for Classification and Preference Learning
• Big Self-Supervised Models are Strong Semi-Supervised Learners
• Born Again Neural Networks
• Shake-Shake regularization
• BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learning
• Unsupervised Domain Adaptation through Self-Supervision
• In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learning
• Knowledge Distillation in Generations: More Tolerant Teachers Educate Better Students
• Bayesian Batch Active Learning as Sparse Subset Approximation
• Creating Something from Nothing: Unsupervised Knowledge Distillation for Cross-Modal Hashing
• Curriculum Labeling: Revisiting Pseudo-Labeling for Semi-Supervised Learning
• Negative sampling in semi-supervised learning
• One-bit Supervision for Image Classification