LIMIT-BERT : Linguistic Informed Multi-Task BERT

Abstract

In this paper, we present a Linguistic Informed Multi-Task BERT (LIMIT-BERT) for learning language representations across multiple linguistic tasks by Multi-Task Learning (MTL). LIMIT-BERT includes five key linguistic syntax and semantics tasks: Part-Of-Speech (POS) tags, constituent and dependency syntactic parsing, span and dependency semantic role labeling (SRL). Besides, LIMIT-BERT adopts linguistics mask strategy: Syntactic and Semantic Phrase Masking which mask all of the tokens corresponding to a syntactic/semantic phrase. Different from recent Multi-Task Deep Neural Networks (MT-DNN) (Liu et al., 2019), our LIMIT-BERT is linguistically motivated and learning in a semi-supervised method which provides large amounts of linguistic-task data as same as BERT learning corpus. As a result, LIMIT-BERT not only improves linguistic tasks performance but also benefits from a regularization effect and linguistic information that leads to more general representations to help adapt to new tasks and domains. LIMIT-BERT obtains new state-of-the-art or competitive results on both span and dependency semantic parsing on Propbank benchmarks and both dependency and constituent syntactic parsing on Penn Treebank.

arXiv commentEMNLP 2020, ACL Findings

References in corpus (5)

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Semantic Sentence Matching with Densely-connected Recurrent and Co-attentive Information
• Hierarchical Pointer Net Parsing
• Explicit Contextual Semantics for Text Comprehension
• Discourse-Aware Semantic Self-Attention for Narrative Reading Comprehension

Cited by in corpus (3)

• Machine Reading Comprehension: The Role of Contextualized Language Models and Beyond
• SG-Net: Syntax-Guided Machine Reading Comprehension
• KgPLM: Knowledge-guided Language Model Pre-training via Generative and Discriminative Learning