Demystifying BERT: Implications for Accelerator Design

Abstract

Transfer learning in natural language processing (NLP), as realized using models like BERT (Bi-directional Encoder Representation from Transformer), has significantly improved language representation with models that can tackle challenging language problems. Consequently, these applications are driving the requirements of future systems. Thus, we focus on BERT, one of the most popular NLP transfer learning algorithms, to identify how its algorithmic behavior can guide future accelerator design. To this end, we carefully profile BERT training and identify key algorithmic behaviors which are worthy of attention in accelerator design. We observe that while computations which manifest as matrix multiplication dominate BERT's overall runtime, as in many convolutional neural networks, memory-intensive computations also feature prominently. We characterize these computations, which have received little attention so far. Further, we also identify heterogeneity in compute-intensive BERT computations and discuss software and possible hardware mechanisms to further optimize these computations. Finally, we discuss implications of these behaviors as networks get larger and use distributed training environments, and how techniques such as micro-batching and mixed-precision training scale. Overall, our analysis identifies holistic solutions to optimize systems for BERT-like models.

References in corpus (16)

• An overview of gradient descent optimization algorithms
• Gaussian Error Linear Units (GELUs)
• Language Models are Few-Shot Learners
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• Well-Read Students Learn Better: On the Importance of Pre-training Compact Models
• Fathom: Reference Workloads for Modern Deep Learning Methods
• Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
• TensorDash: Exploiting Sparsity to Accelerate Deep Neural Network Training and Inference
• Optimizing Performance of Recurrent Neural Networks on GPUs
• Scale MLPerf-0.6 models on Google TPU-v3 Pods
• Data Movement Is All You Need: A Case Study on Optimizing Transformers
• Exploring the limits of Concurrency in ML Training on Google TPUs
• Accelerating Bandwidth-Bound Deep Learning Inference with Main-Memory Accelerators
• Demystifying the MLPerf Benchmark Suite
• Automatic Horizontal Fusion for GPU Kernels
• FlexSA: Flexible Systolic Array Architecture for Efficient Pruned DNN Model Training