Multi-Task Learning for Sequence Tagging: An Empirical Study

Abstract

We study three general multi-task learning (MTL) approaches on 11 sequence tagging tasks. Our extensive empirical results show that in about 50% of the cases, jointly learning all 11 tasks improves upon either independent or pairwise learning of the tasks. We also show that pairwise MTL can inform us what tasks can benefit others or what tasks can be benefited if they are learned jointly. In particular, we identify tasks that can always benefit others as well as tasks that can always be harmed by others. Interestingly, one of our MTL approaches yields embeddings of the tasks that reveal the natural clustering of semantic and syntactic tasks. Our inquiries have opened the doors to further utilization of MTL in NLP.

arXiv commentIn Proceedings of the 27th International Conference on Computational Linguistics (COLING 2018)

References in corpus (8)

• Sequence to Sequence Learning with Neural Networks
• Natural Language Processing (almost) from Scratch
• On the difficulty of training Recurrent Neural Networks
• An Overview of Multi-Task Learning in Deep Neural Networks
• Transfer Learning for Sequence Tagging with Hierarchical Recurrent Networks
• Empower Sequence Labeling with Task-Aware Neural Language Model
• Semantic Tagging with Deep Residual Networks
• Deep Semi-Supervised Learning with Linguistically Motivated Sequence Labeling Task Hierarchies

Cited by in corpus (18)

• Structured Prediction as Translation between Augmented Natural Languages
• Multi-Task Learning in Natural Language Processing: An Overview
• An Empirical Study of Multi-Task Learning on BERT for Biomedical Text Mining
• Exploring and Predicting Transferability across NLP Tasks
• Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling
• Multi-head Sequence Tagging Model for Grammatical Error Correction
• Extracting Symptoms and their Status from Clinical Conversations
• Deconstructing and reconstructing word embedding algorithms
• Latent-Optimized Adversarial Neural Transfer for Sarcasm Detection
• Learning Functions to Study the Benefit of Multitask Learning
• Low Resource Multi-Task Sequence Tagging -- Revisiting Dynamic Conditional Random Fields
• Augmented Natural Language for Generative Sequence Labeling
• Word Embedding Algorithms as Generalized Low Rank Models and their Canonical Form
• Multi-Task Learning with Contextualized Word Representations for Extented Named Entity Recognition
• How to best use Syntax in Semantic Role Labelling
• A Framework For Identifying Group Behavior Of Wild Animals
• MetaXT: Meta Cross-Task Transfer between Disparate Label Spaces
• Learning Efficient Task-Specific Meta-Embeddings with Word Prisms