MC-BERT: Efficient Language Pre-Training via a Meta Controller
arXiv:2006.05744
Abstract
Pre-trained contextual representations (e.g., BERT) have become the foundation to achieve state-of-the-art results on many NLP tasks. However, large-scale pre-training is computationally expensive. ELECTRA, an early attempt to accelerate pre-training, trains a discriminative model that predicts whether each input token was replaced by a generator. Our studies reveal that ELECTRA's success is mainly due to its reduced complexity of the pre-training task: the binary classification (replaced token detection) is more efficient to learn than the generation task (masked language modeling). However, such a simplified task is less semantically informative. To achieve better efficiency and effectiveness, we propose a novel meta-learning framework, MC-BERT. The pre-training task is a multi-choice cloze test with a reject option, where a meta controller network provides training input and candidates. Results over GLUE natural language understanding benchmark demonstrate that our proposed method is both efficient and effective: it outperforms baselines on GLUE semantic tasks given the same computational budget.
References in corpus (6)
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
- XLNet: Generalized Autoregressive Pretraining for Language Understanding
- ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
- GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
- Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books
- Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
Cited by in corpus (5)
- COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining
- Building Chinese Biomedical Language Models via Multi-Level Text Discrimination
- Model-Generated Pretraining Signals Improves Zero-Shot Generalization of Text-to-Text Transformers
- Training ELECTRA Augmented with Multi-word Selection
- CoRe: An Efficient Coarse-refined Training Framework for BERT