StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding

Abstract

Recently, the pre-trained language model, BERT (and its robustly optimized version RoBERTa), has attracted a lot of attention in natural language understanding (NLU), and achieved state-of-the-art accuracy in various NLU tasks, such as sentiment classification, natural language inference, semantic textual similarity and question answering. Inspired by the linearization exploration work of Elman [8], we extend BERT to a new model, StructBERT, by incorporating language structures into pre-training. Specifically, we pre-train StructBERT with two auxiliary tasks to make the most of the sequential order of words and sentences, which leverage language structures at the word and sentence levels, respectively. As a result, the new model is adapted to different levels of language understanding required by downstream tasks. The StructBERT with structural pre-training gives surprisingly good empirical results on a variety of downstream tasks, including pushing the state-of-the-art on the GLUE benchmark to 89.0 (outperforming all published models), the F1 score on SQuAD v1.1 question answering to 93.0, the accuracy on SNLI to 91.7.

10 Pages

References in corpus (4)

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Snorkel: Rapid Training Data Creation with Weak Supervision
• Multi-Task Deep Neural Networks for Natural Language Understanding
• Neural Sentence Ordering

Cited by in corpus (3)

• Hardware Accelerator for Multi-Head Attention and Position-Wise Feed-Forward in the Transformer
• BET: A Backtranslation Approach for Easy Data Augmentation in Transformer-based Paraphrase Identification Context
• Probabilistically Masked Language Model Capable of Autoregressive Generation in Arbitrary Word Order