MPNet: Masked and Permuted Pre-training for Language Understanding
arXiv:2004.09297
Abstract
BERT adopts masked language modeling (MLM) for pre-training and is one of the most successful pre-training models. Since BERT neglects dependency among predicted tokens, XLNet introduces permuted language modeling (PLM) for pre-training to address this problem. However, XLNet does not leverage the full position information of a sentence and thus suffers from position discrepancy between pre-training and fine-tuning. In this paper, we propose MPNet, a novel pre-training method that inherits the advantages of BERT and XLNet and avoids their limitations. MPNet leverages the dependency among predicted tokens through permuted language modeling (vs. MLM in BERT), and takes auxiliary position information as input to make the model see a full sentence and thus reducing the position discrepancy (vs. PLM in XLNet). We pre-train MPNet on a large-scale dataset (over 160GB text corpora) and fine-tune on a variety of down-streaming tasks (GLUE, SQuAD, etc). Experimental results show that MPNet outperforms MLM and PLM by a large margin, and achieves better results on these tasks compared with previous state-of-the-art pre-trained methods (e.g., BERT, XLNet, RoBERTa) under the same model setting. The code and the pre-trained models are available at: https://github.com/microsoft/MPNet.
References in corpus (4)
Cited by in corpus (32)
- TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up
- OGB-LSC: A Large-Scale Challenge for Machine Learning on Graphs
- Machine Reading Comprehension: The Role of Contextualized Language Models and Beyond
- Cross-Modal and Uni-Modal Soft-Label Alignment for Image-Text Retrieval
- Making LLMs Worth Every Penny: Resource-Limited Text Classification in Banking
- CrisisTransformers: Pre-trained language models and sentence encoders for crisis-related social media texts
- Exploiting Simulated User Feedback for Conversational Search: Ranking, Rewriting, and Beyond
- DeTiME: Diffusion-Enhanced Topic Modeling using Encoder-decoder based LLM
- Dynamic GNNs for Precise Seizure Detection and Classification from EEG Data
- Analyzing and Mitigating Interference in Neural Architecture Search
- Zero-shot Bilingual App Reviews Mining with Large Language Models
- Personalizing Task-oriented Dialog Systems via Zero-shot Generalizable Reward Function
- SCARF: Self-Supervised Contrastive Learning using Random Feature Corruption
- Explainable Fact-checking through Question Answering
- ERNIE-Gram: Pre-Training with Explicitly N-Gram Masked Language Modeling for Natural Language Understanding
- Proactive Prioritization of App Issues via Contrastive Learning
- Investigating Content-Aware Neural Text-To-Speech MOS Prediction Using Prosodic and Linguistic Features
- PARK: Personalized academic retrieval with knowledge-graphs
- Evaluating the Knowledge Dependency of Questions
- Relational Programming with Foundation Models
- FrameFinder: Explorative Multi-Perspective Framing Extraction from News Headlines
- This Reads Like That: Deep Learning for Interpretable Natural Language Processing
- The Strong Pull of Prior Knowledge in Large Language Models and Its Impact on Emotion Recognition
- NLP4Gov: A Comprehensive Library for Computational Policy Analysis
- From Pixels to Words: Leveraging Explainability in Face Recognition through Interactive Natural Language Processing
- Analysing State-Backed Propaganda Websites: a New Dataset and Linguistic Study
- PEFA: Parameter-Free Adapters for Large-scale Embedding-based Retrieval Models
- Hubness Reduction Improves Sentence-BERT Semantic Spaces
- Prosody Analysis of Audiobooks
- Vector Symbolic Open Source Information Discovery
- Transductive Learning for Textual Few-Shot Classification in API-based Embedding Models
- Intelligent Semantic Matching (ISM) for Video Tutorial Search using Transformer Models