Diverse Pretrained Context Encodings Improve Document Translation

Abstract

We propose a new architecture for adapting a sentence-level sequence-to-sequence transformer by incorporating multiple pretrained document context signals and assess the impact on translation performance of (1) different pretraining approaches for generating these signals, (2) the quantity of parallel data for which document context is available, and (3) conditioning on source, target, or source and target contexts. Experiments on the NIST Chinese-English, and IWSLT and WMT English-German tasks support four general conclusions: that using pretrained context representations markedly improves sample efficiency, that adequate parallel data resources are crucial for learning to use document context, that jointly conditioning on multiple context representations outperforms any single representation, and that source context is more valuable for translation performance than target side context. Our best multi-context model consistently outperforms the best existing context-aware transformers.

References in corpus (18)

• Neural Machine Translation by Jointly Learning to Align and Translate
• Cross-lingual Language Model Pretraining
• PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
• Unified Language Model Pre-training for Natural Language Understanding and Generation
• MASS: Masked Sequence to Sequence Pre-training for Language Generation
• Learned in Translation: Contextualized Word Vectors
• Incorporating BERT into Neural Machine Translation
• Does Neural Machine Translation Benefit from Larger Context?
• Understanding Back-Translation at Scale
• Evaluating Discourse Phenomena in Neural Machine Translation
• Exploiting Cross-Sentence Context for Neural Machine Translation
• Pre-trained Language Model Representations for Language Generation
• An Embarrassingly Simple Approach for Transfer Learning from Pretrained Language Models
• Improving the Transformer Translation Model with Document-Level Context
• Multi-branch Attentive Transformer
• Document-Level Neural Machine Translation with Hierarchical Attention Networks
• Repurposing Decoder-Transformer Language Models for Abstractive Summarization
• Context-Aware Monolingual Repair for Neural Machine Translation

Cited by in corpus (1)

• Learn To Remember: Transformer with Recurrent Memory for Document-Level Machine Translation