Semantics-aware BERT for Language Understanding

Abstract

The latest work on language representations carefully integrates contextualized features into language model training, which enables a series of success especially in various machine reading comprehension and natural language inference tasks. However, the existing language representation models including ELMo, GPT and BERT only exploit plain context-sensitive features such as character or word embeddings. They rarely consider incorporating structured semantic information which can provide rich semantics for language representation. To promote natural language understanding, we propose to incorporate explicit contextual semantics from pre-trained semantic role labeling, and introduce an improved language representation model, Semantics-aware BERT (SemBERT), which is capable of explicitly absorbing contextual semantics over a BERT backbone. SemBERT keeps the convenient usability of its BERT precursor in a light fine-tuning way without substantial task-specific modifications. Compared with BERT, semantics-aware BERT is as simple in concept but more powerful. It obtains new state-of-the-art or substantially improves results on ten reading comprehension and language inference tasks.

Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI-2020)

References in corpus (4)

• Distributed Representations of Words and Phrases and their Compositionality
• Modeling Multi-turn Conversation with Deep Utterance Aggregation
• U-Net: Machine Reading Comprehension with Unanswerable Questions
• Explicit Contextual Semantics for Text Comprehension

Cited by in corpus (10)

• TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
• Establishing Baselines for Text Classification in Low-Resource Languages
• Reference Language based Unsupervised Neural Machine Translation
• High-order Semantic Role Labeling
• Probing Contextualized Sentence Representations with Visual Awareness
• Probing the Natural Language Inference Task with Automated Reasoning Tools
• Analyzing Multi-Task Learning for Abstractive Text Summarization
• Subjective Question Answering: Deciphering the inner workings of Transformers in the realm of subjectivity
• Evaluating German Transformer Language Models with Syntactic Agreement Tests
• Multi-Head Self-Attention with Role-Guided Masks