Rethinking Pre-training and Self-training

Abstract

Pre-training is a dominant paradigm in computer vision. For example, supervised ImageNet pre-training is commonly used to initialize the backbones of object detection and segmentation models. He et al., however, show a surprising result that ImageNet pre-training has limited impact on COCO object detection. Here we investigate self-training as another method to utilize additional data on the same setup and contrast it against ImageNet pre-training. Our study reveals the generality and flexibility of self-training with three additional insights: 1) stronger data augmentation and more labeled data further diminish the value of pre-training, 2) unlike pre-training, self-training is always helpful when using stronger data augmentation, in both low-data and high-data regimes, and 3) in the case that pre-training is helpful, self-training improves upon pre-training. For example, on the COCO object detection dataset, pre-training benefits when we use one fifth of the labeled data, and hurts accuracy when we use all labeled data. Self-training, on the other hand, shows positive improvements from +1.3 to +3.4AP across all dataset sizes. In other words, self-training works well exactly on the same setup that pre-training does not work (using ImageNet to help COCO). On the PASCAL segmentation dataset, which is a much smaller dataset than COCO, though pre-training does help significantly, self-training improves upon the pre-trained model. On COCO object detection, we achieve 54.3AP, an improvement of +1.5AP over the strongest SpineNet model. On PASCAL segmentation, we achieve 90.5 mIOU, an improvement of +1.5% mIOU over the previous state-of-the-art result by DeepLabv3+.

arXiv commentAccepted for publication at the Thirty-fourth Conference on Neural Information Processing Systems (NeurIPS 2020)

References in corpus (27)

• Deep Learning in Neural Networks: An Overview
• Rethinking Atrous Convolution for Semantic Image Segmentation
• A Simple Framework for Contrastive Learning of Visual Representations
• Representation Learning with Contrastive Predictive Coding
• Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation
• DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
• FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
• Unsupervised Data Augmentation for Consistency Training
• Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
• Interpolation Consistency Training for Semi-Supervised Learning
• DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs
• AutoAugment: Learning Augmentation Policies from Data
• Semi-Supervised Learning with Ladder Networks
• Contrastive Multiview Coding
• DropBlock: A regularization method for convolutional networks
• Semi-supervised Sequence Learning
• Fast-SCNN: Fast Semantic Segmentation Network
• Learning with Pseudo-Ensembles
• Billion-scale semi-supervised learning for image classification
• A Simple Semi-Supervised Learning Framework for Object Detection
• Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning
• Self-Training for End-to-End Speech Recognition
• ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoring
• There Are Many Consistent Explanations of Unlabeled Data: Why You Should Average
• An Analysis of Pre-Training on Object Detection
• Bridging the domain gap in cross-lingual document classification
• Naive-Student: Leveraging Semi-Supervised Learning in Video Sequences for Urban Scene Segmentation

Cited by in corpus (72)

• Self-Supervised Representation Learning: Introduction, Advances and Challenges
• A Survey on Deep Learning-based Architectures for Semantic Segmentation on 2D images
• Revisiting ResNets: Improved Training and Scaling Strategies
• PseudoSeg: Designing Pseudo Labels for Semantic Segmentation
• Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition
• Gait Recognition Based on Deep Learning: A Survey
• Learning 3D Mineral Prospectivity from 3D Geological Models Using Convolutional Neural Networks: Application to a Structure-controlled Hydrothermal Gold Deposit
• Deep Long-Tailed Learning: A Survey
• Semi-Supervised Semantic Segmentation with Cross Pseudo Supervision
• Cycle Self-Training for Domain Adaptation
• Few-Shot Named Entity Recognition: A Comprehensive Study
• Self-training Improves Pre-training for Natural Language Understanding
• A Comprehensive Review of Modern Object Segmentation Approaches
• Semi-Supervised Training to Improve Player and Ball Detection in Soccer
• Exploring the Limits of Large Scale Pre-training
• Adaptive Self-training for Few-shot Neural Sequence Labeling
• A Study of the Generalizability of Self-Supervised Representations
• Exploiting Shape Cues for Weakly Supervised Semantic Segmentation
• Self-Training Vision Language BERTs with a Unified Conditional Model
• ST++: Make Self-training Work Better for Semi-supervised Semantic Segmentation
• Effect of Pre-Training Scale on Intra- and Inter-Domain Full and Few-Shot Transfer Learning for Natural and Medical X-Ray Chest Images
• PseudoAugment: Learning to Use Unlabeled Data for Data Augmentation in Point Clouds
• RPT: Toward Transferable Model on Heterogeneous Researcher Data via Pre-Training
• Effective End-to-End Vision Language Pretraining with Semantic Visual Loss
• Robust Mutual Learning for Semi-supervised Semantic Segmentation
• Distribution Consistency based Self-Training for Graph Neural Networks with Sparse Labels
• STAL3D: Unsupervised Domain Adaptation for 3D Object Detection via Collaborating Self-Training and Adversarial Learning
• Neural Semi-supervised Learning for Text Classification Under Large-Scale Pretraining
• A linearized framework and a new benchmark for model selection for fine-tuning
• A Survey on Deep Learning Methods for Semantic Image Segmentation in Real-Time
• In-N-Out: Pre-Training and Self-Training using Auxiliary Information for Out-of-Distribution Robustness
• Learning Curves for Analysis of Deep Networks
• Benchmarking the Robustness of Instance Segmentation Models
• A Simple Baseline for Semi-supervised Semantic Segmentation with Strong Data Augmentation
• Combining Self-Supervised and Supervised Learning with Noisy Labels
• Aerial Imagery Pixel-level Segmentation
• EfficientSeg: An Efficient Semantic Segmentation Network
• An analysis of over-sampling labeled data in semi-supervised learning with FixMatch
• Dilated SpineNet for Semantic Segmentation
• Re-distributing Biased Pseudo Labels for Semi-supervised Semantic Segmentation: A Baseline Investigation
• Data-Uncertainty Guided Multi-Phase Learning for Semi-Supervised Object Detection
• Non-binary deep transfer learning for image classification
• A Scaling Law for Synthetic-to-Real Transfer: How Much Is Your Pre-training Effective?
• Distilling Ensemble of Explanations for Weakly-Supervised Pre-Training of Image Segmentation Models
• Data Transfer Approaches to Improve Seq-to-Seq Retrosynthesis
• EBJR: Energy-Based Joint Reasoning for Adaptive Inference
• A new semi-supervised self-training method for lung cancer prediction
• Pseudo-labeling for Scalable 3D Object Detection
• Pointly-Supervised Instance Segmentation
• Can Pretext-Based Self-Supervised Learning Be Boosted by Downstream Data? A Theoretical Analysis
• Does Data Augmentation Benefit from Split BatchNorms
• Fashionpedia: Ontology, Segmentation, and an Attribute Localization Dataset
• Gradual Domain Adaptation in the Wild:When Intermediate Distributions are Absent
• Data-Efficient Deep Learning Method for Image Classification Using Data Augmentation, Focal Cosine Loss, and Ensemble
• Recursive Training for Zero-Shot Semantic Segmentation
• Semi-supervised Semantic Segmentation with Directional Context-aware Consistency
• Ranking Neural Checkpoints
• A Brief Summary of Interactions Between Meta-Learning and Self-Supervised Learning
• Quantifying Topology In Pancreatic Tubular Networks From Live Imaging 3D Microscopy
• Selection for short-term empowerment accelerates the evolution of homeostatic neural cellular automata
• On the Marginal Benefit of Active Learning: Does Self-Supervision Eat Its Cake?
• Task-adaptive Pre-training and Self-training are Complementary for Natural Language Understanding
• Object-Aware Cropping for Self-Supervised Learning
• An empirical study of domain-agnostic semi-supervised learning via energy-based models: joint-training and pre-training
• SLADE: A Self-Training Framework For Distance Metric Learning
• Revisiting Self-Training for Few-Shot Learning of Language Model
• Network representation learning: A macro and micro view
• Learning from Multiple Noisy Augmented Data Sets for Better Cross-Lingual Spoken Language Understanding
• Semi-Supervised Learning with Taxonomic Labels
• Using Psuedolabels for training Sentiment Classifiers makes the model generalize better across datasets
• Predictions For Pre-training Language Models
• Multi-Object Tracking with Hallucinated and Unlabeled Videos