Real-Time Execution of Large-scale Language Models on Mobile
arXiv:2009.06823
Abstract
Pre-trained large-scale language models have increasingly demonstrated high accuracy on many natural language processing (NLP) tasks. However, the limited weight storage and computational speed on hardware platforms have impeded the popularity of pre-trained models, especially in the era of edge computing. In this paper, we seek to find the best model structure of BERT for a given computation size to match specific devices. We propose the first compiler-aware neural architecture optimization framework. Our framework can guarantee the identified model to meet both resource and real-time specifications of mobile devices, thus achieving real-time execution of large transformer-based models like BERT variants. We evaluate our model on several NLP tasks, achieving competitive results on well-known benchmarks with lower latency on mobile devices. Specifically, our model is 5.2x faster on CPU and 4.1x faster on GPU with 0.5-2% accuracy loss compared with BERT-base. Our overall framework achieves up to 7.8x speedup compared with TensorFlow-Lite with only minor accuracy loss.
References in corpus (11)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
- HuggingFace's Transformers: State-of-the-art Natural Language Processing
- XLNet: Generalized Autoregressive Pretraining for Language Understanding
- Learning Structured Sparsity in Deep Neural Networks
- Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books
- ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware
- TVM: An Automated End-to-End Optimizing Compiler for Deep Learning
- TinyBERT: Distilling BERT for Natural Language Understanding
- FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search
- Reweighted Proximal Pruning for Large-Scale Language Representation