StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding

Abstract

Recently, the pre-trained language model, BERT (and its robustly optimized version RoBERTa), has attracted a lot of attention in natural language understanding (NLU), and achieved state-of-the-art accuracy in various NLU tasks, such as sentiment classification, natural language inference, semantic textual similarity and question answering. Inspired by the linearization exploration work of Elman [8], we extend BERT to a new model, StructBERT, by incorporating language structures into pre-training. Specifically, we pre-train StructBERT with two auxiliary tasks to make the most of the sequential order of words and sentences, which leverage language structures at the word and sentence levels, respectively. As a result, the new model is adapted to different levels of language understanding required by downstream tasks. The StructBERT with structural pre-training gives surprisingly good empirical results on a variety of downstream tasks, including pushing the state-of-the-art on the GLUE benchmark to 89.0 (outperforming all published models), the F1 score on SQuAD v1.1 question answering to 93.0, the accuracy on SNLI to 91.7.

10 Pages

References in corpus (10)

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Gaussian Error Linear Units (GELUs)
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• Snorkel: Rapid Training Data Creation with Weak Supervision
• A large annotated corpus for learning natural language inference
• QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension
• Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks
• Multi-Task Deep Neural Networks for Natural Language Understanding
• Neural Sentence Ordering

Cited by in corpus (32)

• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Pre-trained Models for Natural Language Processing: A Survey
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• DeBERTa: Decoding-enhanced BERT with Disentangled Attention
• COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining
• M6: A Chinese Multimodal Pretrainer
• AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing
• Are Pre-trained Convolutions Better than Pre-trained Transformers?
• SemVLP: Vision-Language Pre-training by Aligning Semantics at Multiple Levels
• CAPT: Contrastive Pre-Training for Learning Denoised Sequence Representations
• Hardware Accelerator for Multi-Head Attention and Position-Wise Feed-Forward in the Transformer
• AMBERT: A Pre-trained Language Model with Multi-Grained Tokenization
• Progressively Stacking 2.0: A Multi-stage Layerwise Training Method for BERT Training Speedup
• StructuralLM: Structural Pre-training for Form Understanding
• E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learning
• Symmetric Regularization based BERT for Pair-wise Semantic Reasoning
• MVP-BERT: Redesigning Vocabularies for Chinese BERT and Multi-Vocab Pretraining
• BET: A Backtranslation Approach for Easy Data Augmentation in Transformer-based Paraphrase Identification Context
• Learning Compositional Representation for Few-shot Visual Question Answering
• Shaking Syntactic Trees on the Sesame Street: Multilingual Probing with Controllable Perturbations
• CMV-BERT: Contrastive multi-vocab pretraining of BERT
• Improving Self-supervised Pre-training via a Fully-Explored Masked Language Model
• Meta Fine-Tuning Neural Language Models for Multi-Domain Text Mining
• Language Model Evaluation in Open-ended Text Generation
• Hierarchical Multitask Learning Approach for BERT
• Achieving Human Parity on Visual Question Answering
• Probabilistically Masked Language Model Capable of Autoregressive Generation in Arbitrary Word Order
• MNet-Sim: A Multi-layered Semantic Similarity Network to Evaluate Sentence Similarity
• A Survey on Green Deep Learning
• ICDAR 2021 Competition on Document VisualQuestion Answering
• K-AID: Enhancing Pre-trained Language Models with Domain Knowledge for Question Answering
• Memorization in Deep Neural Networks: Does the Loss Function matter?