CMV-BERT: Contrastive multi-vocab pretraining of BERT
arXiv:2012.14763
Abstract
In this work, we represent CMV-BERT, which improves the pretraining of a language model via two ingredients: (a) contrastive learning, which is well studied in the area of computer vision; (b) multiple vocabularies, one of which is fine-grained and the other is coarse-grained. The two methods both provide different views of an original sentence, and both are shown to be beneficial. Downstream tasks demonstrate our proposed CMV-BERT are effective in improving the pretrained language models.
arXiv commentwill add more detailed technical contents, and more detailed experiments
References in corpus (10)
- XLNet: Generalized Autoregressive Pretraining for Language Understanding
- ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
- ERNIE: Enhanced Representation through Knowledge Integration
- A large annotated corpus for learning natural language inference
- Exploring Simple Siamese Representation Learning
- Big Bird: Transformers for Longer Sequences
- XNLI: Evaluating Cross-lingual Sentence Representations
- StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding
- ERNIE 2.0: A Continual Pre-training Framework for Language Understanding
- MVP-BERT: Redesigning Vocabularies for Chinese BERT and Multi-Vocab Pretraining