Lite Transformer with Long-Short Range Attention

Abstract

Transformer has become ubiquitous in natural language processing (e.g., machine translation, question answering); however, it requires enormous amount of computations to achieve high performance, which makes it not suitable for mobile applications that are tightly constrained by the hardware resources and battery. In this paper, we present an efficient mobile NLP architecture, Lite Transformer to facilitate deploying mobile NLP applications on edge devices. The key primitive is the Long-Short Range Attention (LSRA), where one group of heads specializes in the local context modeling (by convolution) while another group specializes in the long-distance relationship modeling (by attention). Such specialization brings consistent improvement over the vanilla transformer on three well-established language tasks: machine translation, abstractive summarization, and language modeling. Under constrained resources (500M/100M MACs), Lite Transformer outperforms transformer on WMT'14 English-French by 1.2/1.7 BLEU, respectively. Lite Transformer reduces the computation of transformer base model by 2.5x with 0.3 BLEU score degradation. Combining with pruning and quantization, we further compressed the model size of Lite Transformer by 18.2x. For language modeling, Lite Transformer achieves 1.8 lower perplexity than the transformer at around 500M MACs. Notably, Lite Transformer outperforms the AutoML-based Evolved Transformer by 0.5 higher BLEU for the mobile NLP setting without the costly architecture search that requires more than 250 GPU years. Code has been made available at https://github.com/mit-han-lab/lite-transformer.

ICLR 2020. The first two authors contributed equally to this work

Cited by in corpus (30)

• A Survey on Visual Transformer
• LocalViT: Analyzing Locality in Vision Transformers
• Machine Learning for Microcontroller-Class Hardware: A Review
• HAT: Hardware-Aware Transformers for Efficient Natural Language Processing
• Bringing AI To Edge: From Deep Learning's Perspective
• A Practical Survey on Faster and Lighter Transformers
• ConvBERT: Improving BERT with Span-based Dynamic Convolution
• Random Feature Attention
• Lightweight Self-Attentive Sequential Recommendation
• ConvTransformer: A Convolutional Transformer Network for Video Frame Synthesis
• Long-Short Transformer: Efficient Transformers for Language and Vision
• DeLighT: Deep and Light-weight Transformer
• Transformer-based End-to-End Speech Recognition with Local Dense Synthesizer Attention
• Learning Graph Structures with Transformer for Multivariate Time Series Anomaly Detection in IoT
• Conformer-Kernel with Query Term Independence for Document Retrieval
• Exploiting Temporal Contexts with Strided Transformer for 3D Human Pose Estimation
• Convolution-enhanced Evolving Attention Networks
• Pipeline Parallelism for Inference on Heterogeneous Edge Computing
• ODE Transformer: An Ordinary Differential Equation-Inspired Model for Neural Machine Translation
• Normalized Attention Without Probability Cage
• Dynamic Multi-Branch Layers for On-Device Neural Machine Translation
• Language Modeling using LMUs: 10x Better Data Efficiency or Improved Scaling Compared to Transformers
• Sub-Linear Memory: How to Make Performers SLiM
• Locally Enhanced Self-Attention: Combining Self-Attention and Convolution as Local and Context Terms
• Hi-Transformer: Hierarchical Interactive Transformer for Efficient and Effective Long Document Modeling
• GroupBERT: Enhanced Transformer Architecture with Efficient Grouped Structures
• On the Importance of Local Information in Transformer Based Models
• Subformer: Exploring Weight Sharing for Parameter Efficiency in Generative Transformers
• Input-independent Attention Weights Are Expressive Enough: A Study of Attention in Self-supervised Audio Transformers
• Layer Reduction: Accelerating Conformer-Based Self-Supervised Model via Layer Consistency