Data Distillation: Towards Omni-Supervised Learning

Abstract

We investigate omni-supervised learning, a special regime of semi-supervised learning in which the learner exploits all available labeled data plus internet-scale sources of unlabeled data. Omni-supervised learning is lower-bounded by performance on existing labeled datasets, offering the potential to surpass state-of-the-art fully supervised methods. To exploit the omni-supervised setting, we propose data distillation, a method that ensembles predictions from multiple transformations of unlabeled data, using a single model, to automatically generate new training annotations. We argue that visual recognition models have recently become accurate enough that it is now possible to apply classic ideas about self-training to challenging real-world data. Our experimental results show that in the cases of human keypoint detection and general object detection, state-of-the-art models trained with data distillation surpass the performance of using labeled data from the COCO dataset alone.

tech report

References in corpus (16)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• Deep Residual Learning for Image Recognition
• FitNets: Hints for Thin Deep Nets
• Going Deeper with Convolutions
• OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks
• Fully Convolutional Networks for Semantic Segmentation
• Training Deep Neural Networks on Noisy Labels with Bootstrapping
• Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learning
• Rich feature hierarchies for accurate object detection and semantic segmentation
• Aggregated Residual Transformations for Deep Neural Networks
• Learning with Pseudo-Ensembles
• Scalable, High-Quality Object Detection
• Revisiting Unreasonable Effectiveness of Data in Deep Learning Era
• Unifying distillation and privileged information
• Cross Modal Distillation for Supervision Transfer

Cited by in corpus (16)

• Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks
• Annotation-efficient deep learning for automatic medical image segmentation
• Localizing Objects with Self-Supervised Transformers and no Labels
• Learning Efficient Detector with Semi-supervised Adaptive Distillation
• Attention-Guided Answer Distillation for Machine Reading Comprehension
• Deep Co-Training for Semi-Supervised Image Segmentation
• Improving Multi-Person Pose Estimation using Label Correction
• OMNIA Faster R-CNN: Detection in the wild through dataset merging and soft distillation
• AffinityNet: semi-supervised few-shot learning for disease type prediction
• Universal Lesion Detection by Learning from Multiple Heterogeneously Labeled Datasets
• X-model: Improving Data Efficiency in Deep Learning with A Minimax Model
• RDPD: Rich Data Helps Poor Data via Imitation
• CHEER: Rich Model Helps Poor Model via Knowledge Infusion
• Path-Invariant Map Networks
• A Self-Supervised Bootstrap Method for Single-Image 3D Face Reconstruction
• Dataset Culling: Towards Efficient Training Of Distillation-Based Domain Specific Models