Multi-Task Learning in Natural Language Processing: An Overview

Abstract

Deep learning approaches have achieved great success in the field of Natural Language Processing (NLP). However, directly training deep neural models often suffer from overfitting and data scarcity problems that are pervasive in NLP tasks. In recent years, Multi-Task Learning (MTL), which can leverage useful information of related tasks to achieve simultaneous performance improvement on these tasks, has been used to handle these problems. In this paper, we give an overview of the use of MTL in NLP tasks. We first review MTL architectures used in NLP tasks and categorize them into four classes, including parallel architecture, hierarchical architecture, modular architecture, and generative adversarial architecture. Then we present optimization techniques on loss construction, gradient regularization, data sampling, and task scheduling to properly train a multi-task model. After presenting applications of MTL in a variety of NLP tasks, we introduce some benchmark datasets. Finally, we make a conclusion and discuss several possible research directions in this field.

Accepted by ACM Computing Surveys

References in corpus (10)

• XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization
• BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning
• All-in-one: Multi-task Learning for Rumour Verification
• Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models
• Implicit Discourse Relation Classification via Multi-Task Neural Networks
• Multi-Task Neural Models for Translating Between Styles Within and Across Languages
• Multi-Task Learning for Sequence Tagging: An Empirical Study
• Multi-task dialog act and sentiment recognition on Mastodon
• Meta Multi-Task Learning for Sequence Modeling
• Mutlitask Learning for Cross-Lingual Transfer of Semantic Dependencies