ChineseBERT: Chinese Pretraining Enhanced by Glyph and Pinyin Information

Abstract

Recent pretraining models in Chinese neglect two important aspects specific to the Chinese language: glyph and pinyin, which carry significant syntax and semantic information for language understanding. In this work, we propose ChineseBERT, which incorporates both the {\it glyph} and {\it pinyin} information of Chinese characters into language model pretraining. The glyph embedding is obtained based on different fonts of a Chinese character, being able to capture character semantics from the visual features, and the pinyin embedding characterizes the pronunciation of Chinese characters, which handles the highly prevalent heteronym phenomenon in Chinese (the same character has different pronunciations with different meanings). Pretrained on large-scale unlabeled Chinese corpus, the proposed ChineseBERT model yields significant performance boost over baseline models with fewer training steps. The porpsoed model achieves new SOTA performances on a wide range of Chinese NLP tasks, including machine reading comprehension, natural language inference, text classification, sentence pair matching, and competitive performances in named entity recognition. Code and pretrained models are publicly available at https://github.com/ShannonAI/ChineseBert.

arXiv commentTo appear at ACL2021

References in corpus (18)

• Efficient Estimation of Word Representations in Vector Space
• Distributed Representations of Words and Phrases and their Compositionality
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• ERNIE: Enhanced Representation through Knowledge Integration
• MASS: Masked Sequence to Sequence Pre-training for Language Generation
• UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training
• Incorporating BERT into Neural Machine Translation
• XNLI: Evaluating Cross-lingual Sentence Representations
• Rethinking Attention with Performers
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
• Character-based Joint Segmentation and POS Tagging for Chinese using Bidirectional RNN-CRF
• A Unified MRC Framework for Named Entity Recognition
• Description Based Text Classification with Reinforcement Learning
• CPM: A Large-scale Generative Chinese Pre-trained Language Model
• FGN: Fusion Glyph Network for Chinese Named Entity Recognition
• Chinese Embedding via Stroke and Glyph Information: A Dual-channel View
• Using Chinese Glyphs for Named Entity Recognition

Cited by in corpus (5)

• CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generation
• Interpreting Deep Learning Models in Natural Language Processing: A Review
• GNN-LM: Language Modeling based on Global Contexts via GNN
• MFE-NER: Multi-feature Fusion Embedding for Chinese Named Entity Recognition
• Paraphrase Generation as Unsupervised Machine Translation