Distilling Knowledge Learned in BERT for Text Generation

Abstract

Large-scale pre-trained language model such as BERT has achieved great success in language understanding tasks. However, it remains an open question how to utilize BERT for language generation. In this paper, we present a novel approach, Conditional Masked Language Modeling (C-MLM), to enable the finetuning of BERT on target generation tasks. The finetuned BERT (teacher) is exploited as extra supervision to improve conventional Seq2Seq models (student) for better text generation performance. By leveraging BERT's idiosyncratic bidirectional nature, distilling knowledge learned in BERT can encourage auto-regressive Seq2Seq models to plan ahead, imposing global sequence-level supervision for coherent text generation. Experiments show that the proposed approach significantly outperforms strong Transformer baselines on multiple language generation tasks such as machine translation and text summarization. Our proposed model also achieves new state of the art on IWSLT German-English and English-Vietnamese MT datasets. Code is available at https://github.com/ChenRocks/Distill-BERT-Textgen.

arXiv commentACL 2020

References in corpus (17)

• Distributed Representations of Words and Phrases and their Compositionality
• Distilling the Knowledge in a Neural Network
• Sequence to Sequence Learning with Neural Networks
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Convolutional Sequence to Sequence Learning
• Cross-lingual Language Model Pretraining
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• MASS: Masked Sequence to Sequence Pre-training for Language Generation
• Pay Less Attention with Lightweight and Dynamic Convolutions
• Faithful to the Original: Fact Aware Neural Abstractive Summarization
• BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model
• Multilingual Neural Machine Translation with Knowledge Distillation
• Sequence-Level Knowledge Distillation
• OpenNMT: Neural Machine Translation Toolkit
• Twin Networks: Matching the Future for Sequence Generation
• Towards Making the Most of BERT in Neural Machine Translation

Cited by in corpus (5)

• Big Bird: Transformers for Longer Sequences
• HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training
• Incorporating BERT into Parallel Sequence Decoding with Adapters
• A new hope for network model generalization
• POINTER: Constrained Progressive Text Generation via Insertion-based Generative Pre-training