Freezing Subnetworks to Analyze Domain Adaptation in Neural Machine Translation

Abstract

To better understand the effectiveness of continued training, we analyze the major components of a neural machine translation system (the encoder, decoder, and each embedding space) and consider each component's contribution to, and capacity for, domain adaptation. We find that freezing any single component during continued training has minimal impact on performance, and that performance is surprisingly good when a single component is adapted while holding the rest of the model fixed. We also find that continued training does not move the model very far from the out-of-domain model, compared to a sensitivity analysis metric, suggesting that the out-of-domain model can provide a good generic initialization for the new domain.

arXiv commentpresented at WMT 2018. Please cite using the bib entry from here: http://www.statmt.org/wmt18/bib/WMT013.bib

References in corpus (5)

• Adam: A Method for Stochastic Optimization
• Neural Machine Translation by Jointly Learning to Align and Translate
• Sockeye: A Toolkit for Neural Machine Translation
• Fast Domain Adaptation for Neural Machine Translation
• Transfer Learning across Low-Resource, Related Languages for Neural Machine Translation

Cited by in corpus (14)

• Omnidirectional Video Super-Resolution using Deep Learning
• Recipes for Adapting Pre-trained Monolingual and Multilingual Models to Machine Translation
• Controlling the Output Length of Neural Machine Translation
• Robust Neural Machine Translation for Clean and Noisy Speech Transcripts
• Simple, Scalable Adaptation for Neural Machine Translation
• Multilingual Multi-Domain Adaptation Approaches for Neural Machine Translation
• Learning a Multi-Domain Curriculum for Neural Machine Translation
• Neural Machine Translation: A Review and Survey
• Dynamically Composing Domain-Data Selection with Clean-Data Selection by "Co-Curricular Learning" for Neural Machine Translation
• Efficiently Reusing Old Models Across Languages via Transfer Learning
• Investigating Catastrophic Forgetting During Continual Training for Neural Machine Translation
• Quantifying and Maximizing the Benefits of Back-End Noise Adaption on Attention-Based Speech Recognition Models
• Personalizing Grammatical Error Correction: Adaptation to Proficiency Level and L1
• More Parameters? No Thanks!