NEZHA: Neural Contextualized Representation for Chinese Language Understanding

Abstract

The pre-trained language models have achieved great successes in various natural language understanding (NLU) tasks due to its capacity to capture the deep contextualized information in text by pre-training on large-scale corpora. In this technical report, we present our practice of pre-training language models named NEZHA (NEural contextualiZed representation for CHinese lAnguage understanding) on Chinese corpora and finetuning for the Chinese NLU tasks. The current version of NEZHA is based on BERT with a collection of proven improvements, which include Functional Relative Positional Encoding as an effective positional encoding scheme, Whole Word Masking strategy, Mixed Precision Training and the LAMB Optimizer in training the models. The experimental results show that NEZHA achieves the state-of-the-art performances when finetuned on several representative Chinese tasks, including named entity recognition (People's Daily NER), sentence matching (LCQMC), Chinese sentiment classification (ChnSenti) and natural language inference (XNLI).

References in corpus (7)

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• Mixed Precision Training
• ERNIE: Enhanced Representation through Knowledge Integration
• Horovod: fast and easy distributed deep learning in TensorFlow
• Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
• XNLI: Evaluating Cross-lingual Sentence Representations

Cited by in corpus (19)

• Pre-trained Models for Natural Language Processing: A Survey
• RoFormer: Enhanced Transformer with Rotary Position Embedding
• CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generation
• PanGu-$\alpha$: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation
• NSP-BERT: A Prompt-based Few-Shot Learner Through an Original Pre-training Task--Next Sentence Prediction
• Mengzi: Towards Lightweight yet Ingenious Pre-trained Models for Chinese
• FewCLUE: A Chinese Few-shot Learning Evaluation Benchmark
• ZEN 2.0: Continue Training and Adaption for N-gram Enhanced Text Encoders
• CPM: A Large-scale Generative Chinese Pre-trained Language Model
• Pre-Trained Models: Past, Present and Future
• ZEN: Pre-training Chinese Text Encoder Enhanced by N-gram Representations
• Censorship of Online Encyclopedias: Implications for NLP Models
• Training Multilingual Pre-trained Language Model with Byte-level Subwords
• ERNIE-Gram: Pre-Training with Explicitly N-Gram Masked Language Modeling for Natural Language Understanding
• Lattice-BERT: Leveraging Multi-Granularity Representations in Chinese Pre-trained Language Models
• You Can Do Better! If You Elaborate the Reason When Making Prediction
• Neural Multi-Task Learning for Teacher Question Detection in Online Classrooms
• An Effective System for Multi-format Information Extraction
• Multi-Task Learning based Online Dialogic Instruction Detection with Pre-trained Language Models