Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
arXiv:2010.07003
Abstract
Despite transformers' impressive accuracy, their computational cost is often prohibitive to use with limited computational resources. Most previous approaches to improve inference efficiency require a separate model for each possible computational budget. In this paper, we extend PoWER-BERT (Goyal et al., 2020) and propose Length-Adaptive Transformer that can be used for various inference scenarios after one-shot training. We train a transformer with LengthDrop, a structural variant of dropout, which stochastically determines a sequence length at each layer. We then conduct a multi-objective evolutionary search to find a length configuration that maximizes the accuracy and minimizes the efficiency metric under any given computational budget. Additionally, we significantly extend the applicability of PoWER-BERT beyond sequence-level classification into token-level classification with Drop-and-Restore process that drops word-vectors temporarily in intermediate layers and restores at the last layer if necessary. We empirically verify the utility of the proposed approach by demonstrating the superior accuracy-efficiency trade-off under various setups, including span-based question answering and text classification. Code is available at https://github.com/clovaai/length-adaptive-transformer.
arXiv commentACL 2021; 11 pages, 4 figures
References in corpus (30)
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Distilling the Knowledge in a Neural Network
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
- Language Models are Few-Shot Learners
- Longformer: The Long-Document Transformer
- Scaling Laws for Neural Language Models
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
- SQuAD: 100,000+ Questions for Machine Comprehension of Text
- DeBERTa: Decoding-enhanced BERT with Disentangled Attention
- Conformer: Convolution-augmented Transformer for Speech Recognition
- Reducing Transformer Depth on Demand with Structured Dropout
- Big Bird: Transformers for Longer Sequences
- Slimmable Neural Networks
- Long Range Arena: A Benchmark for Efficient Transformers
- Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learning
- Rethinking Attention with Performers
- DynaBERT: Dynamic BERT with Adaptive Width and Depth
- NSML: Meet the MLaaS platform with a real-world case study
- Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing
- NSML: A Machine Learning Platform That Enables You to Focus on Your Models
- Depth-Adaptive Transformer
- FastBERT: a Self-distilling BERT with Adaptive Inference Time
- Learning Ordered Representations with Nested Dropout
- PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
- ETC: Encoding Long and Structured Inputs in Transformers
- FLOPs as a Direct Optimization Objective for Learning Sparse Neural Networks
- Anytime Neural Prediction via Slicing Networks Vertically
- TR-BERT: Dynamic Token Reduction for Accelerating BERT Inference
- Accelerating BERT Inference for Sequence Labeling via Early-Exit
- Towards Accurate and Reliable Energy Measurement of NLP Models