COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining

Abstract

We present a self-supervised learning framework, COCO-LM, that pretrains Language Models by COrrecting and COntrasting corrupted text sequences. Following ELECTRA-style pretraining, COCO-LM employs an auxiliary language model to corrupt text sequences, upon which it constructs two new tasks for pretraining the main model. The first token-level task, Corrective Language Modeling, is to detect and correct tokens replaced by the auxiliary model, in order to better capture token-level semantics. The second sequence-level task, Sequence Contrastive Learning, is to align text sequences originated from the same source input while ensuring uniformity in the representation space. Experiments on GLUE and SQuAD demonstrate that COCO-LM not only outperforms recent state-of-the-art pretrained models in accuracy, but also improves pretraining efficiency. It achieves the MNLI accuracy of ELECTRA with 50% of its pretraining GPU hours. With the same pretraining steps of standard base/large-sized models, COCO-LM outperforms the previous best models by 1+ GLUE average points.

arXiv commentNeurIPS 2021. (Code and Models: https://github.com/microsoft/COCO-LM)

References in corpus (21)

• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• Language Models are Few-Shot Learners
• Scaling Laws for Neural Language Models
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• REALM: Retrieval-Augmented Language Model Pre-Training
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• A Simple Method for Commonsense Reasoning
• Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books
• Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
• CLEAR: Contrastive Learning for Sentence Representation
• UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training
• Visualizing and Measuring the Geometry of BERT
• StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding
• Pre-training via Paraphrasing
• InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
• Representation Degeneration Problem in Training Natural Language Generation Models
• Knowledge-Aware Language Model Pretraining
• SimCSE: Simple Contrastive Learning of Sentence Embeddings
• CoDA: Contrast-enhanced and Diversity-promoting Data Augmentation for Natural Language Understanding
• Studying Strategically: Learning to Mask for Closed-book QA
• MC-BERT: Efficient Language Pre-Training via a Meta Controller

Cited by in corpus (17)

• Contrastive Learning for Representation Degeneration Problem in Sequential Recommendation
• DeBERTa: Decoding-enhanced BERT with Disentangled Attention
• DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing
• ESimCSE: Enhanced Sample Building Method for Contrastive Learning of Unsupervised Sentence Embedding
• SimCSE: Simple Contrastive Learning of Sentence Embeddings
• Improving Contrastive Learning of Sentence Embeddings with Case-Augmented Positives and Retrieved Negatives
• Building Chinese Biomedical Language Models via Multi-Level Text Discrimination
• Supporting Clustering with Contrastive Learning
• KFCNet: Knowledge Filtering and Contrastive Learning Network for Generative Commonsense Reasoning
• Unsupervised Dense Retrieval Training with Web Anchors
• Model-Generated Pretraining Signals Improves Zero-Shot Generalization of Text-to-Text Transformers
• Distantly-Supervised Named Entity Recognition with Noise-Robust Learning and Language Model Augmented Self-Training
• DialogueCSE: Dialogue-based Contrastive Learning of Sentence Embeddings
• Pairwise Supervised Contrastive Learning of Sentence Representations
• Efficient Contrastive Learning via Novel Data Augmentation and Curriculum Learning
• Contrastive Document Representation Learning with Graph Attention Networks
• Not All Negatives are Equal: Label-Aware Contrastive Loss for Fine-grained Text Classification