Multi-task learning for natural language processing in the 2020s: where are we going?

Abstract

Multi-task learning (MTL) significantly pre-dates the deep learning era, and it has seen a resurgence in the past few years as researchers have been applying MTL to deep learning solutions for natural language tasks. While steady MTL research has always been present, there is a growing interest driven by the impressive successes published in the related fields of transfer learning and pre-training, such as BERT, and the release of new challenge problems, such as GLUE and the NLP Decathlon (decaNLP). These efforts place more focus on how weights are shared across networks, evaluate the re-usability of network components and identify use cases where MTL can significantly outperform single-task solutions. This paper strives to provide a comprehensive survey of the numerous recent MTL contributions to the field of natural language processing and provide a forum to focus efforts on the hardest unsolved problems in the next decade. While novel models that improve performance on NLP benchmarks are continually produced, lasting MTL challenges remain unsolved which could hold the key to better language understanding, knowledge discovery and natural language interfaces.

12 pages, 2 figures. Published in Elsevier Pattern Recognition Letters Volume 136. Accepted manuscript published here

References in corpus (15)

• Distilling the Knowledge in a Neural Network
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• An Overview of Multi-Task Learning in Deep Neural Networks
• SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
• GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
• The Natural Language Decathlon: Multitask Learning as Question Answering
• Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks
• Which Tasks Should Be Learned Together in Multi-task Learning?
• Reverse Curriculum Generation for Reinforcement Learning
• BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning
• Multi-task Sequence to Sequence Learning
• AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence
• Not Enough Data? Deep Learning to the Rescue!
• When is multitask learning effective? Semantic sequence prediction under varying data conditions
• Seq2Seq and Multi-Task Learning for joint intent and content extraction for domain specific interpreters

Cited by in corpus (10)

• MEDIC: A Multi-Task Learning Dataset for Disaster Image Classification
• Finetuned Language Models Are Zero-Shot Learners
• AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing
• Greedy-layer Pruning: Speeding up Transformer Models for Natural Language Processing
• A Domain-adaptive Pre-training Approach for Language Bias Detection in News
• Transfer Learning Study of Motion Transformer-based Trajectory Predictions
• Learning to Transfer for Traffic Forecasting via Multi-task Learning
• Exceeding the Limits of Visual-Linguistic Multi-Task Learning
• WASSA@IITK at WASSA 2021: Multi-task Learning and Transformer Finetuning for Emotion Classification and Empathy Prediction
• Toward a multimodal multitask model for neurodegenerative diseases diagnosis and progression prediction