Incorporating BERT into Neural Machine Translation

Abstract

The recently proposed BERT has shown great power on a variety of natural language understanding tasks, such as text classification, reading comprehension, etc. However, how to effectively apply BERT to neural machine translation (NMT) lacks enough exploration. While BERT is more commonly used as fine-tuning instead of contextual embedding for downstream language understanding tasks, in NMT, our preliminary exploration of using BERT as contextual embedding is better than using for fine-tuning. This motivates us to think how to better leverage BERT for NMT along this direction. We propose a new algorithm named BERT-fused model, in which we first use BERT to extract representations for an input sequence, and then the representations are fused with each layer of the encoder and decoder of the NMT model through attention mechanisms. We conduct experiments on supervised (including sentence-level and document-level translations), semi-supervised and unsupervised machine translation, and achieve state-of-the-art results on seven benchmark datasets. Our code is available at \url{https://github.com/bert-nmt/bert-nmt}.

Accepted to ICLR-2020

References in corpus (9)

• Distributed Representations of Words and Phrases and their Compositionality
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• MASS: Masked Sequence to Sequence Pre-training for Language Generation
• Semi-supervised Sequence Learning
• Pay Less Attention with Lightweight and Dynamic Convolutions
• The Evolved Transformer
• Learning to Teach with Dynamic Loss Functions
• Latent Alignment and Variational Attention

Cited by in corpus (36)

• Pre-trained Models for Natural Language Processing: A Survey
• R-Drop: Regularized Dropout for Neural Networks
• Prefix-Tuning: Optimizing Continuous Prompts for Generation
• Revisiting Few-sample BERT Fine-tuning
• Empowering Things with Intelligence: A Survey of the Progress, Challenges, and Opportunities in Artificial Intelligence of Things
• Exploring the State of the Art in Legal QA Systems
• Data Diversification: A Simple Strategy For Neural Machine Translation
• Energy-Based Reranking: Improving Neural Machine Translation Using Energy-Based Models
• Artificial intelligence for topic modelling in Hindu philosophy: mapping themes between the Upanishads and the Bhagavad Gita
• Ad Headline Generation using Self-Critical Masked Language Model
• On Learning Universal Representations Across Languages
• ODE Transformer: An Ordinary Differential Equation-Inspired Model for Neural Machine Translation
• Neural Machine Translation: Challenges, Progress and Future
• ChineseBERT: Chinese Pretraining Enhanced by Glyph and Pinyin Information
• Diverse Pretrained Context Encodings Improve Document Translation
• Encoder-Decoder Models Can Benefit from Pre-trained Masked Language Models in Grammatical Error Correction
• SJTU-NICT's Supervised and Unsupervised Neural Machine Translation Systems for the WMT20 News Translation Task
• Reward Optimization for Neural Machine Translation with Learned Metrics
• Recent Trends in the Use of Deep Learning Models for Grammar Error Handling
• Prevent the Language Model from being Overconfident in Neural Machine Translation
• Bridging Subword Gaps in Pretrain-Finetune Paradigm for Natural Language Generation
• Rapid Domain Adaptation for Machine Translation with Monolingual Data
• Speech Recognition by Simply Fine-tuning BERT
• BERT, mBERT, or BiBERT? A Study on Contextualized Embeddings for Neural Machine Translation
• ChrEn: Cherokee-English Machine Translation for Endangered Language Revitalization
• UniDrop: A Simple yet Effective Technique to Improve Transformer without Extra Cost
• On the Copying Behaviors of Pre-Training for Neural Machine Translation
• Resurrecting Submodularity for Neural Text Generation
• Single Model Ensemble using Pseudo-Tags and Distinct Vectors
• Dynamic Curriculum Learning for Low-Resource Neural Machine Translation
• HRKD: Hierarchical Relational Knowledge Distillation for Cross-domain Language Model Compression
• On the Complementarity between Pre-Training and Back-Translation for Neural Machine Translation
• Multilingual Translation via Grafting Pre-trained Language Models
• The Cross-Lingual Arabic Information REtrieval (CLAIRE) System
• Verb Sense Clustering using Contextualized Word Representations for Semantic Frame Induction
• Two-Headed Monster And Crossed Co-Attention Networks