GroupBERT: Enhanced Transformer Architecture with Efficient Grouped Structures

Abstract

Attention based language models have become a critical component in state-of-the-art natural language processing systems. However, these models have significant computational requirements, due to long training times, dense operations and large parameter count. In this work we demonstrate a set of modifications to the structure of a Transformer layer, producing a more efficient architecture. First, we add a convolutional module to complement the self-attention module, decoupling the learning of local and global interactions. Secondly, we rely on grouped transformations to reduce the computational cost of dense feed-forward layers and convolutions, while preserving the expressivity of the model. We apply the resulting architecture to language representation learning and demonstrate its superior performance compared to BERT models of different scales. We further highlight its improved efficiency, both in terms of floating-point operations (FLOPs) and time-to-train.

References in corpus (24)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Distilling the Knowledge in a Neural Network
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Decoupled Weight Decay Regularization
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Language Models are Few-Shot Learners
• Scaling Laws for Neural Language Models
• Searching for Activation Functions
• Conformer: Convolution-augmented Transformer for Speech Recognition
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• Pay Less Attention with Lightweight and Dynamic Convolutions
• Deep Roots: Improving CNN Efficiency with Hierarchical Filter Groups
• Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping
• A Practical Survey on Faster and Lighter Transformers
• Lite Transformer with Long-Short Range Attention
• Rigging the Lottery: Making All Tickets Winners
• On Layer Normalization in the Transformer Architecture
• Learning Deep Transformer Models for Machine Translation
• Dissecting the Graphcore IPU Architecture via Microbenchmarking
• Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth
• DeLighT: Deep and Light-weight Transformer
• Understanding the Difficulty of Training Transformers
• Making EfficientNet More Efficient: Exploring Batch-Independent Normalization, Group Convolutions and Reduced Resolution Training