Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers

Abstract

There remain many open questions pertaining to the scaling behaviour of Transformer architectures. These scaling decisions and findings can be critical, as training runs often come with an associated computational cost which have both financial and/or environmental impact. The goal of this paper is to present scaling insights from pretraining and finetuning Transformers. While Kaplan et al. presents a comprehensive study of the scaling behaviour of Transformer language models, the scope is only on the upstream (pretraining) loss. Therefore, it is still unclear if these set of findings transfer to downstream task within the context of the pretrain-finetune paradigm. The key findings of this paper are as follows: (1) we show that aside from only the model size, model shape matters for downstream fine-tuning, (2) scaling protocols operate differently at different compute regions, (3) widely adopted T5-base and T5-large sizes are Pareto-inefficient. To this end, we present improved scaling protocols whereby our redesigned models achieve similar downstream fine-tuning quality while having 50\% fewer parameters and training 40\% faster compared to the widely adopted T5-base model. We publicly release over 100 pretrained checkpoints of different T5 configurations to facilitate future research and analysis.

ICLR 2022 + Updated Checkpoint Release

References in corpus (20)

• Distilling the Knowledge in a Neural Network
• Sequence to Sequence Learning with Neural Networks
• Language Models are Few-Shot Learners
• Scaling Laws for Neural Language Models
• MLP-Mixer: An all-MLP Architecture for Vision
• Reformer: The Efficient Transformer
• Revisiting ResNets: Improved Training and Scaling Strategies
• Perceiver IO: A General Architecture for Structured Inputs & Outputs
• The Evolved Transformer
• Scaling Laws for Autoregressive Generative Modeling
• Explaining Neural Scaling Laws
• Carbon Emissions and Large Neural Network Training
• Charformer: Fast Character Transformers via Gradient-based Subword Tokenization
• Sparse Sinkhorn Attention
• M6: A Chinese Multimodal Pretrainer
• Beyond Fully-Connected Layers with Quaternions: Parameterization of Hypercomplex Multiplications with $1/n$ Parameters
• Are Pre-trained Convolutions Better than Pre-trained Transformers?
• Scaling Laws for Transfer
• Lightweight and Efficient Neural Natural Language Processing with Quaternion Networks
• The Benchmark Lottery

Cited by in corpus (10)

• Reproducible scaling laws for contrastive language-image learning
• Exploring the Limits of Large Scale Pre-training
• Energy and Carbon Considerations of Fine-Tuning BERT
• Scaling Up Vision-Language Pre-training for Image Captioning
• RSTeller: Scaling Up Visual Language Modeling in Remote Sensing with Rich Linguistic Semantics from Openly Available Data and Large Language Models
• Jack and Masters of all Trades: One-Pass Learning Sets of Model Sets From Large Pre-Trained Models
• NLP From Scratch Without Large-Scale Pretraining: A Simple and Efficient Framework
• ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning
• SCENIC: A JAX Library for Computer Vision Research and Beyond
• The Efficiency Misnomer