MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices

Abstract

Natural Language Processing (NLP) has recently achieved great success by using huge pre-trained models with hundreds of millions of parameters. However, these models suffer from heavy model sizes and high latency such that they cannot be deployed to resource-limited mobile devices. In this paper, we propose MobileBERT for compressing and accelerating the popular BERT model. Like the original BERT, MobileBERT is task-agnostic, that is, it can be generically applied to various downstream NLP tasks via simple fine-tuning. Basically, MobileBERT is a thin version of BERT_LARGE, while equipped with bottleneck structures and a carefully designed balance between self-attentions and feed-forward networks. To train MobileBERT, we first train a specially designed teacher model, an inverted-bottleneck incorporated BERT_LARGE model. Then, we conduct knowledge transfer from this teacher to MobileBERT. Empirical studies show that MobileBERT is 4.3x smaller and 5.5x faster than BERT_BASE while achieving competitive results on well-known benchmarks. On the natural language inference tasks of GLUE, MobileBERT achieves a GLUEscore o 77.7 (0.6 lower than BERT_BASE), and 62 ms latency on a Pixel 4 phone. On the SQuAD v1.1/v2.0 question answering task, MobileBERT achieves a dev F1 score of 90.0/79.2 (1.5/2.1 higher than BERT_BASE).

Accepted to ACL 2020

References in corpus (12)

• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• FitNets: Hints for Thin Deep Nets
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• Well-Read Students Learn Better: On the Importance of Pre-training Compact Models
• Distilling Task-Specific Knowledge from BERT into Simple Neural Networks
• Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books
• Multi-Task Deep Neural Networks for Natural Language Understanding
• Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
• Sequence-Level Knowledge Distillation
• Factorization tricks for LSTM networks
• Small and Practical BERT Models for Sequence Labeling

Cited by in corpus (35)

• Pre-trained Models for Natural Language Processing: A Survey
• Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better
• Bringing AI To Edge: From Deep Learning's Perspective
• Keyword Transformer: A Self-Attention Model for Keyword Spotting
• TinyBERT: Distilling BERT for Natural Language Understanding
• ConvBERT: Improving BERT with Span-based Dynamic Convolution
• SEED: Self-supervised Distillation For Visual Representation
• Structured Pruning of a BERT-based Question Answering Model
• Two Stage Transformer Model for COVID-19 Fake News Detection and Fact Checking
• MixKD: Towards Efficient Distillation of Large-scale Language Models
• Optimal Subarchitecture Extraction For BERT
• I-BERT: Integer-only BERT Quantization
• GOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inference
• NLP From Scratch Without Large-Scale Pretraining: A Simple and Efficient Framework
• DeFormer: Decomposing Pre-trained Transformers for Faster Question Answering
• ALP-KD: Attention-Based Layer Projection for Knowledge Distillation
• Automated essay scoring using efficient transformer-based language models
• BERT-EMD: Many-to-Many Layer Mapping for BERT Compression with Earth Mover's Distance
• NxMTransformer: Semi-Structured Sparsification for Natural Language Understanding via ADMM
• Contrastive Distillation on Intermediate Representations for Language Model Compression
• LRC-BERT: Latent-representation Contrastive Knowledge Distillation for Natural Language Understanding
• Revisiting Adversarial Robustness Distillation: Robust Soft Labels Make Student Better
• ML-EXray: Visibility into ML Deployment on the Edge
• Why Skip If You Can Combine: A Simple Knowledge Distillation Technique for Intermediate Layers
• Vis-TOP: Visual Transformer Overlay Processor
• Compressed Deep Networks: Goodbye SVD, Hello Robust Low-Rank Approximation
• Towards Accurate and Reliable Energy Measurement of NLP Models
• Open, Sesame! Introducing Access Control to Voice Services
• Improving Task-Agnostic BERT Distillation with Layer Mapping Search
• EmpLite: A Lightweight Sequence Labeling Model for Emphasis Selection of Short Texts
• Cost-effective Deployment of BERT Models in Serverless Environment
• Why Can You Lay Off Heads? Investigating How BERT Heads Transfer
• Designing a Minimal Retrieve-and-Read System for Open-Domain Question Answering
• AutoFL: Enabling Heterogeneity-Aware Energy Efficient Federated Learning
• Adversarially Robust and Explainable Model Compression with On-Device Personalization for Text Classification