Incorporating BERT into Parallel Sequence Decoding with Adapters

Abstract

While large scale pre-trained language models such as BERT have achieved great success on various natural language understanding tasks, how to efficiently and effectively incorporate them into sequence-to-sequence models and the corresponding text generation tasks remains a non-trivial problem. In this paper, we propose to address this problem by taking two different BERT models as the encoder and decoder respectively, and fine-tuning them by introducing simple and lightweight adapter modules, which are inserted between BERT layers and tuned on the task-specific dataset. In this way, we obtain a flexible and efficient model which is able to jointly leverage the information contained in the source-side and target-side BERT models, while bypassing the catastrophic forgetting problem. Each component in the framework can be considered as a plug-in unit, making the framework flexible and task agnostic. Our framework is based on a parallel sequence decoding algorithm named Mask-Predict considering the bi-directional and conditional independent nature of BERT, and can be adapted to traditional autoregressive decoding easily. We conduct extensive experiments on neural machine translation tasks where the proposed method consistently outperforms autoregressive baselines while reducing the inference latency by half, and achieves $36.49$/$33.57$ BLEU scores on IWSLT14 German-English/WMT14 German-English translation. When adapted to autoregressive decoding, the proposed method achieves $30.60$/$43.56$ BLEU scores on WMT14 English-German/English-French translation, on par with the state-of-the-art baseline models.

NeurIPS 2020

References in corpus (19)

• Distilling the Knowledge in a Neural Network
• Sequence to Sequence Learning with Neural Networks
• Cross-lingual Language Model Pretraining
• MASS: Masked Sequence to Sequence Pre-training for Language Generation
• Learning multiple visual domains with residual adapters
• Non-Autoregressive Neural Machine Translation
• Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond
• Levenshtein Transformer
• Insertion Transformer: Flexible Sequence Generation via Insertion Operations
• Parameter-Efficient Transfer Learning for NLP
• K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
• Sequence-Level Knowledge Distillation
• Understanding Back-Translation at Scale
• Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Models
• Joint Training for Neural Machine Translation Models with Monolingual Data
• Towards Making the Most of BERT in Neural Machine Translation
• Distilling Knowledge Learned in BERT for Text Generation
• Simple, Scalable Adaptation for Neural Machine Translation
• Acquiring Knowledge from Pre-trained Model to Neural Machine Translation

Cited by in corpus (6)

• Task-Oriented Dialogue System as Natural Language Generation
• Energy-Based Reranking: Improving Neural Machine Translation Using Energy-Based Models
• BANG: Bridging Autoregressive and Non-autoregressive Generation with Large Scale Pretraining
• Parameter-Efficient Abstractive Question Answering over Tables or Text
• Non-autoregressive Personalized Bundle Generation
• Neighbors Are Not Strangers: Improving Non-Autoregressive Translation under Low-Frequency Lexical Constraints