Exploring the Limits of Large Scale Pre-training

Abstract

Recent developments in large-scale machine learning suggest that by scaling up data, model size and training time properly, one might observe that improvements in pre-training would transfer favorably to most downstream tasks. In this work, we systematically study this phenomena and establish that, as we increase the upstream accuracy, the performance of downstream tasks saturates. In particular, we investigate more than 4800 experiments on Vision Transformers, MLP-Mixers and ResNets with number of parameters ranging from ten million to ten billion, trained on the largest scale of available image data (JFT, ImageNet21K) and evaluated on more than 20 downstream image recognition tasks. We propose a model for downstream performance that reflects the saturation phenomena and captures the nonlinear relationship in performance of upstream and downstream tasks. Delving deeper to understand the reasons that give rise to these phenomena, we show that the saturation behavior we observe is closely related to the way that representations evolve through the layers of the models. We showcase an even more extreme scenario where performance on upstream and downstream are at odds with each other. That is, to have a better downstream performance, we need to hurt upstream accuracy.

References in corpus (24)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Learning Transferable Visual Models From Natural Language Supervision
• How transferable are features in deep neural networks?
• Language Models are Few-Shot Learners
• Scaling Laws for Neural Language Models
• MLP-Mixer: An all-MLP Architecture for Vision
• Transfusion: Understanding Transfer Learning for Medical Imaging
• Rethinking Pre-training and Self-training
• WILDS: A Benchmark of in-the-Wild Distribution Shifts
• Revisiting ResNets: Improved Training and Scaling Strategies
• A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark
• Self-supervised Pretraining of Visual Features in the Wild
• Carbon Emissions and Large Neural Network Training
• Do CIFAR-10 Classifiers Generalize to CIFAR-10?
• Predicting the Generalization Gap in Deep Networks with Margin Distributions
• Domain Adaptive Transfer Learning with Specialist Models
• Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers
• Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalization
• Scalable Transfer Learning with Expert Models
• Scaling Laws for Transfer
• Supervised Transfer Learning at Scale for Medical Imaging
• TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?
• OmniNet: Omnidirectional Representations from Transformers
• Comparing Transfer and Meta Learning Approaches on a Unified Few-Shot Classification Benchmark

Cited by in corpus (6)

• Data efficiency and extrapolation trends in neural network interatomic potentials
• Under the Cover Infant Pose Estimation using Multimodal Data
• A vision transformer-based framework for knowledge transfer from multi-modal to mono-modal lymphoma subtyping models
• SCENIC: A JAX Library for Computer Vision Research and Beyond
• Better with Less: A Data-Active Perspective on Pre-Training Graph Neural Networks
• Why Do Better Loss Functions Lead to Less Transferable Features?