Multi-task Sequence to Sequence Learning

Abstract

Sequence to sequence learning has recently emerged as a new paradigm in supervised learning. To date, most of its applications focused on only one task and not much work explored this framework for multiple tasks. This paper examines three multi-task learning (MTL) settings for sequence to sequence models: (a) the oneto-many setting - where the encoder is shared between several tasks such as machine translation and syntactic parsing, (b) the many-to-one setting - useful when only the decoder can be shared, as in the case of translation and image caption generation, and (c) the many-to-many setting - where multiple encoders and decoders are shared, which is the case with unsupervised objectives and translation. Our results show that training on a small amount of parsing and image caption data can improve the translation quality between English and German by up to 1.5 BLEU points over strong single-task baselines on the WMT benchmarks. Furthermore, we have established a new state-of-the-art result in constituent parsing with 93.0 F1. Lastly, we reveal interesting properties of the two unsupervised learning objectives, autoencoder and skip-thought, in the MTL context: autoencoder helps less in terms of perplexities but more on BLEU scores compared to skip-thought.

10 pages, 4 figures, ICLR 2016 camera-ready, added parsing SOTA results

Cited by in corpus (22)

• Multi-task learning for natural language processing in the 2020s: where are we going?
• Search Engine Guided Non-Parametric Neural Machine Translation
• Encoding Sentences with Graph Convolutional Networks for Semantic Role Labeling
• Embedding API Dependency Graph for Neural Code Generation
• Robust Neural Abstractive Summarization Systems and Evaluation against Adversarial Information
• Doubly Attentive Transformer Machine Translation
• Unsupervised Pretraining for Sequence to Sequence Learning
• Exploiting Multi-typed Treebanks for Parsing with Deep Multi-task Learning
• Exploiting Linguistic Resources for Neural Machine Translation Using Multi-task Learning
• Multitask Learning Strengthens Adversarial Robustness
• Enhancing Machine Translation with Dependency-Aware Self-Attention
• Improving Interpretability of Word Embeddings by Generating Definition and Usage
• Correct-and-Memorize: Learning to Translate from Interactive Revisions
• A Study of Multilingual Neural Machine Translation
• Dependency Parsing as Head Selection
• Neural Argument Generation Augmented with Externally Retrieved Evidence
• Investigating Meta-Learning Algorithms for Low-Resource Natural Language Understanding Tasks
• Shift-Reduce Constituent Parsing with Neural Lookahead Features
• Neural Machine Translation: A Review and Survey
• Exploiting Spatio-Temporal Structure with Recurrent Winner-Take-All Networks
• Prior Attention for Style-aware Sequence-to-Sequence Models
• Constructing Multiple Tasks for Augmentation: Improving Neural Image Classification With K-means Features