Exploiting Linguistic Resources for Neural Machine Translation Using Multi-task Learning

Abstract

Linguistic resources such as part-of-speech (POS) tags have been extensively used in statistical machine translation (SMT) frameworks and have yielded better performances. However, usage of such linguistic annotations in neural machine translation (NMT) systems has been left under-explored. In this work, we show that multi-task learning is a successful and a easy approach to introduce an additional knowledge into an end-to-end neural attentional model. By jointly training several natural language processing (NLP) tasks in one system, we are able to leverage common information and improve the performance of the individual task. We analyze the impact of three design decisions in multi-task learning: the tasks used in training, the training schedule, and the degree of parameter sharing across the tasks, which is defined by the network architecture. The experiments are conducted for an German to English translation task. As additional linguistic resources, we exploit POS information and named-entities (NE). Experiments show that the translation quality can be improved by up to 1.5 BLEU points under the low-resource condition. The performance of the POS tagger is also improved using the multi-task learning scheme.

9 pages, Second Conference on Machine Translation(WMT17)

References in corpus (6)

• Neural Machine Translation by Jointly Learning to Align and Translate
• Natural Language Processing (almost) from Scratch
• Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder
• Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation
• Multi-task Sequence to Sequence Learning
• Zero-Resource Translation with Multi-Lingual Neural Machine Translation

Cited by in corpus (9)

• Syntax-Infused Transformer and BERT models for Machine Translation and Natural Language Understanding
• AutoLoss: Learning Discrete Schedules for Alternate Optimization
• Analyzing the Forgetting Problem in the Pretrain-Finetuning of Dialogue Response Models
• Recurrent Graph Syntax Encoder for Neural Machine Translation
• Natural Language Multitasking: Analyzing and Improving Syntactic Saliency of Hidden Representations
• Learning to Multi-Task Learn for Better Neural Machine Translation
• MetaXT: Meta Cross-Task Transfer between Disparate Label Spaces
• Toward a Standardized and More Accurate Indonesian Part-of-Speech Tagging
• Improving Multilingual Translation by Representation and Gradient Regularization