Multi-Task Self-Training for Learning General Representations
arXiv:2108.11353
Abstract
Despite the fast progress in training specialized models for various tasks, learning a single general model that works well for many tasks is still challenging for computer vision. Here we introduce multi-task self-training (MuST), which harnesses the knowledge in independent specialized teacher models (e.g., ImageNet model on classification) to train a single general student model. Our approach has three steps. First, we train specialized teachers independently on labeled datasets. We then use the specialized teachers to label an unlabeled dataset to create a multi-task pseudo labeled dataset. Finally, the dataset, which now contains pseudo labels from teacher models trained on different datasets/tasks, is then used to train a student model with multi-task learning. We evaluate the feature representations of the student model on 6 vision tasks including image recognition (classification, detection, segmentation)and 3D geometry estimation (depth and surface normal estimation). MuST is scalable with unlabeled or partially labeled datasets and outperforms both specialized supervised models and self-supervised models when training on large scale datasets. Lastly, we show MuST can improve upon already strong checkpoints trained with billions of examples. The results suggest self-training is a promising direction to aggregate labeled and unlabeled training data for learning general feature representations.
arXiv commentICCV 2021
References in corpus (15)
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Distilling the Knowledge in a Neural Network
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- An Overview of Multi-Task Learning in Deep Neural Networks
- Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
- Data-Efficient Image Recognition with Contrastive Predictive Coding
- The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale
- Billion-scale semi-supervised learning for image classification
- Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition
- Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding
- Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biases
- Gradient Surgery for Multi-Task Learning
- DIODE: A Dense Indoor and Outdoor DEpth Dataset
- Self-training Improves Pre-training for Natural Language Understanding
- An Analysis of Pre-Training on Object Detection