Incorporating BERT into Neural Machine Translation

Abstract

The recently proposed BERT has shown great power on a variety of natural language understanding tasks, such as text classification, reading comprehension, etc. However, how to effectively apply BERT to neural machine translation (NMT) lacks enough exploration. While BERT is more commonly used as fine-tuning instead of contextual embedding for downstream language understanding tasks, in NMT, our preliminary exploration of using BERT as contextual embedding is better than using for fine-tuning. This motivates us to think how to better leverage BERT for NMT along this direction. We propose a new algorithm named BERT-fused model, in which we first use BERT to extract representations for an input sequence, and then the representations are fused with each layer of the encoder and decoder of the NMT model through attention mechanisms. We conduct experiments on supervised (including sentence-level and document-level translations), semi-supervised and unsupervised machine translation, and achieve state-of-the-art results on seven benchmark datasets. Our code is available at \url{https://github.com/bert-nmt/bert-nmt}.

Accepted to ICLR-2020

References in corpus (4)

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• MASS: Masked Sequence to Sequence Pre-training for Language Generation
• Pay Less Attention with Lightweight and Dynamic Convolutions
• The Evolved Transformer

Cited by in corpus (23)

• R-Drop: Regularized Dropout for Neural Networks
• Prefix-Tuning: Optimizing Continuous Prompts for Generation
• Empowering Things with Intelligence: A Survey of the Progress, Challenges, and Opportunities in Artificial Intelligence of Things
• ODE Transformer: An Ordinary Differential Equation-Inspired Model for Neural Machine Translation
• Neural Machine Translation: Challenges, Progress and Future
• Encoder-Decoder Models Can Benefit from Pre-trained Masked Language Models in Grammatical Error Correction
• Diverse Pretrained Context Encodings Improve Document Translation
• Reward Optimization for Neural Machine Translation with Learned Metrics
• Prevent the Language Model from being Overconfident in Neural Machine Translation
• Recent Trends in the Use of Deep Learning Models for Grammar Error Handling
• Bridging Subword Gaps in Pretrain-Finetune Paradigm for Natural Language Generation
• BERT, mBERT, or BiBERT? A Study on Contextualized Embeddings for Neural Machine Translation
• Speech Recognition by Simply Fine-tuning BERT
• On the Copying Behaviors of Pre-Training for Neural Machine Translation
• UniDrop: A Simple yet Effective Technique to Improve Transformer without Extra Cost
• Verb Sense Clustering using Contextualized Word Representations for Semantic Frame Induction
• Single Model Ensemble using Pseudo-Tags and Distinct Vectors
• Dynamic Curriculum Learning for Low-Resource Neural Machine Translation
• HRKD: Hierarchical Relational Knowledge Distillation for Cross-domain Language Model Compression
• On the Complementarity between Pre-Training and Back-Translation for Neural Machine Translation
• Multilingual Translation via Grafting Pre-trained Language Models
• The Cross-Lingual Arabic Information REtrieval (CLAIRE) System
• Two-Headed Monster And Crossed Co-Attention Networks