The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models

Abstract

The computer vision world has been re-gaining enthusiasm in various pre-trained models, including both classical ImageNet supervised pre-training and recently emerged self-supervised pre-training such as simCLR and MoCo. Pre-trained weights often boost a wide range of downstream tasks including classification, detection, and segmentation. Latest studies suggest that pre-training benefits from gigantic model capacity. We are hereby curious and ask: after pre-training, does a pre-trained model indeed have to stay large for its downstream transferability? In this paper, we examine supervised and self-supervised pre-trained models through the lens of the lottery ticket hypothesis (LTH). LTH identifies highly sparse matching subnetworks that can be trained in isolation from (nearly) scratch yet still reach the full models' performance. We extend the scope of LTH and question whether matching subnetworks still exist in pre-trained computer vision models, that enjoy the same downstream transfer performance. Our extensive experiments convey an overall positive message: from all pre-trained weights obtained by ImageNet classification, simCLR, and MoCo, we are consistently able to locate such matching subnetworks at 59.04% to 96.48% sparsity that transfer universally to multiple downstream tasks, whose performance see no degradation compared to using full pre-trained weights. Further analyses reveal that subnetworks found from different pre-training tend to yield diverse mask structures and perturbation sensitivities. We conclude that the core LTH observations remain generally relevant in the pre-training paradigm of computer vision, but more delicate discussions are needed in some cases. Codes and pre-trained models will be made available at: https://github.com/VITA-Group/CV_LTH_Pre-training.

CVPR 2021

References in corpus (33)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• YOLOv4: Optimal Speed and Accuracy of Object Detection
• A Simple Framework for Contrastive Learning of Visual Representations
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• An overview of gradient descent optimization algorithms
• DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
• Bootstrap your own latent: A new approach to self-supervised Learning
• Improved Baselines with Momentum Contrastive Learning
• Learning deep representations by mutual information estimation and maximization
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• VisDA: The Visual Domain Adaptation Challenge
• Contrastive Multiview Coding
• Big Self-Supervised Models are Strong Semi-Supervised Learners
• The State of Sparsity in Deep Neural Networks
• What makes ImageNet good for transfer learning?
• Sparse Networks from Scratch: Faster Training without Losing Performance
• Comparing Rewinding and Fine-tuning in Neural Network Pruning
• Picking Winning Tickets Before Training by Preserving Gradient Flow
• Stabilizing the Lottery Ticket Hypothesis
• Linear Mode Connectivity and the Lottery Ticket Hypothesis
• Playing the lottery with rewards and multiple languages: lottery tickets in RL and NLP
• Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?
• Local Aggregation for Unsupervised Learning of Visual Embeddings
• Intriguing Properties of Contrastive Losses
• The Early Phase of Neural Network Training
• When BERT Plays the Lottery, All Tickets Are Winning
• TinyTL: Reduce Activations, Not Trainable Parameters for Efficient On-Device Learning
• The Difficulty of Training Sparse Neural Networks
• A Unified Lottery Ticket Hypothesis for Graph Neural Networks
• Sparse Transfer Learning via Winning Lottery Tickets
• Shift Equivariance in Object Detection
• GANs Can Play Lottery Tickets Too

Cited by in corpus (16)

• Pre-Trained Image Processing Transformer
• Chasing Sparsity in Vision Transformers: An End-to-End Exploration
• A Unified Lottery Ticket Hypothesis for Graph Neural Networks
• EarlyBERT: Efficient BERT Training via Early-bird Lottery Tickets
• Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsity
• PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognition
• Data-Efficient GAN Training Beyond (Just) Augmentations: A Lottery Ticket Perspective
• You are caught stealing my winning lottery ticket! Making a lottery ticket claim its ownership
• GANs Can Play Lottery Tickets Too
• The Elastic Lottery Ticket Hypothesis
• Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalization
• On Lottery Tickets and Minimal Task Representations in Deep Reinforcement Learning
• Spending Your Winning Lottery Better After Drawing It
• On the Robustness of Pretraining and Self-Supervision for a Deep Learning-based Analysis of Diabetic Retinopathy
• SmartDeal: Re-Modeling Deep Network Weights for Efficient Inference and Training
• Algorithm to Compilation Co-design: An Integrated View of Neural Network Sparsity