DeLighT: Deep and Light-weight Transformer

Abstract

We introduce a deep and light-weight transformer, DeLighT, that delivers similar or better performance than standard transformer-based models with significantly fewer parameters. DeLighT more efficiently allocates parameters both (1) within each Transformer block using the DeLighT transformation, a deep and light-weight transformation, and (2) across blocks using block-wise scaling, which allows for shallower and narrower DeLighT blocks near the input and wider and deeper DeLighT blocks near the output. Overall, DeLighT networks are 2.5 to 4 times deeper than standard transformer models and yet have fewer parameters and operations. Experiments on benchmark machine translation and language modeling tasks show that DeLighT matches or improves the performance of baseline Transformers with 2 to 3 times fewer parameters on average. Our source code is available at: \url{https://github.com/sacmehta/delight}

arXiv commentAccepted at ICLR 2021

References in corpus (15)

• Distilling the Knowledge in a Neural Network
• Improving neural networks by preventing co-adaptation of feature detectors
• Convolutional Sequence to Sequence Learning
• Longformer: The Long-Document Transformer
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• Generating Long Sequences with Sparse Transformers
• Reformer: The Efficient Transformer
• Pay Less Attention with Lightweight and Dynamic Convolutions
• Synthesizer: Rethinking Self-Attention in Transformer Models
• The Evolved Transformer
• An Analysis of Neural Language Modeling at Multiple Scales
• Lite Transformer with Long-Short Range Attention
• Improving Neural Language Models with a Continuous Cache
• Benefits of depth in neural networks
• Latent Alignment and Variational Attention

Cited by in corpus (9)

• Filter-enhanced MLP is All You Need for Sequential Recommendation
• LocalViT: Analyzing Locality in Vision Transformers
• MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer
• ODE Transformer: An Ordinary Differential Equation-Inspired Model for Neural Machine Translation
• Defending Against Backdoor Attacks in Natural Language Generation
• Optimizing Deep Transformers for Chinese-Thai Low-Resource Translation
• Current Limitations of Language Models: What You Need is Retrieval
• GroupBERT: Enhanced Transformer Architecture with Efficient Grouped Structures
• The Efficiency Misnomer