GroupBERT: Enhanced Transformer Architecture with Efficient Grouped Structures

Abstract

Attention based language models have become a critical component in state-of-the-art natural language processing systems. However, these models have significant computational requirements, due to long training times, dense operations and large parameter count. In this work we demonstrate a set of modifications to the structure of a Transformer layer, producing a more efficient architecture. First, we add a convolutional module to complement the self-attention module, decoupling the learning of local and global interactions. Secondly, we rely on grouped transformations to reduce the computational cost of dense feed-forward layers and convolutions, while preserving the expressivity of the model. We apply the resulting architecture to language representation learning and demonstrate its superior performance compared to BERT models of different scales. We further highlight its improved efficiency, both in terms of floating-point operations (FLOPs) and time-to-train.

References in corpus (13)

• Distilling the Knowledge in a Neural Network
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Language Models are Few-Shot Learners
• Scaling Laws for Neural Language Models
• Searching for Activation Functions
• Conformer: Convolution-augmented Transformer for Speech Recognition
• Pay Less Attention with Lightweight and Dynamic Convolutions
• Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping
• Lite Transformer with Long-Short Range Attention
• Learning Deep Transformer Models for Machine Translation
• Dissecting the Graphcore IPU Architecture via Microbenchmarking
• Making EfficientNet More Efficient: Exploring Batch-Independent Normalization, Group Convolutions and Reduced Resolution Training