Pre-Training with Whole Word Masking for Chinese BERT

Abstract

Bidirectional Encoder Representations from Transformers (BERT) has shown marvelous improvements across various NLP tasks, and its consecutive variants have been proposed to further improve the performance of the pre-trained language models. In this paper, we aim to first introduce the whole word masking (wwm) strategy for Chinese BERT, along with a series of Chinese pre-trained language models. Then we also propose a simple but effective model called MacBERT, which improves upon RoBERTa in several ways. Especially, we propose a new masking strategy called MLM as correction (Mac). To demonstrate the effectiveness of these models, we create a series of Chinese pre-trained language models as our baselines, including BERT, RoBERTa, ELECTRA, RBT, etc. We carried out extensive experiments on ten Chinese NLP tasks to evaluate the created Chinese pre-trained language models as well as the proposed MacBERT. Experimental results show that MacBERT could achieve state-of-the-art performances on many NLP tasks, and we also ablate details with several findings that may help future research. We open-source our pre-trained language models for further facilitating our research community. Resources are available: https://github.com/ymcui/Chinese-BERT-wwm

11 pages. Journal extension to arXiv:2004.13922

References in corpus (2)

• A Span-Extraction Dataset for Chinese Machine Reading Comprehension
• CJRC: A Reliable Human-Annotated Benchmark DataSet for Chinese Judicial Reading Comprehension

Cited by in corpus (17)

• Survey of different Large Language Model Architectures: Trends, Benchmarks, and Challenges
• BrainNPT: Pre-training of Transformer networks for brain network classification
• USTC-NELSLIP at SemEval-2022 Task 11: Gazetteer-Adapted Integration Network for Multilingual Complex Named Entity Recognition
• Automatic Assessment of Divergent Thinking in Chinese Language with TransDis: A Transformer-Based Language Model Approach
• ShortcutLens: A Visual Analytics Approach for Exploring Shortcuts in Natural Language Understanding Dataset
• Multilingual Multi-Aspect Explainability Analyses on Machine Reading Comprehension Models
• Dismantling Gender Blindness in Online Discussion of a Crime/Gender Dichotomy
• Conversational Speech Recognition by Learning Audio-textual Cross-modal Contextual Representation
• Persuasion or Insulting? Unpacking Discursive Strategies of Gender Debate in Everyday Feminism in China
• ORCHID: A Chinese Debate Corpus for Target-Independent Stance Detection and Argumentative Dialogue Summarization
• SNP-S3: Shared Network Pre-training and Significant Semantic Strengthening for Various Video-Text Tasks
• Boosting LLM-based Relevance Modeling with Distribution-Aware Robust Learning
• A Knowledge-Injected Curriculum Pretraining Framework for Question Answering
• AF Adapter: Continual Pretraining for Building Chinese Biomedical Language Model
• Understanding Emotional Disclosure via Diary-keeping in Quarantine on Social Media
• More Is Better: A MoE-Based Emotion Recognition Framework with Human Preference Alignment
• A Survey on Fake Review Detection: From Pre-trained Language Models to Large Language Models