CPM: A Large-scale Generative Chinese Pre-trained Language Model

Abstract

Pre-trained Language Models (PLMs) have proven to be beneficial for various downstream NLP tasks. Recently, GPT-3, with 175 billion parameters and 570GB training data, drew a lot of attention due to the capacity of few-shot (even zero-shot) learning. However, applying GPT-3 to address Chinese NLP tasks is still challenging, as the training corpus of GPT-3 is primarily English, and the parameters are not publicly available. In this technical report, we release the Chinese Pre-trained Language Model (CPM) with generative pre-training on large-scale Chinese training data. To the best of our knowledge, CPM, with 2.6 billion parameters and 100GB Chinese training data, is the largest Chinese pre-trained language model, which could facilitate several downstream Chinese NLP tasks, such as conversation, essay generation, cloze test, and language understanding. Extensive experiments demonstrate that CPM achieves strong performance on many NLP tasks in the settings of few-shot (even zero-shot) learning. The code and parameters are available at https://github.com/TsinghuaAI/CPM-Generate.

References in corpus (9)

• Language Models are Few-Shot Learners
• ERNIE: Enhanced Representation through Knowledge Integration
• How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation
• Topic Aware Neural Response Generation
• Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping
• On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines
• NEZHA: Neural Contextualized Representation for Chinese Language Understanding
• CLUE: A Chinese Language Understanding Evaluation Benchmark
• Know What You Don't Need: Single-Shot Meta-Pruning for Attention Heads

Cited by in corpus (7)

• ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation
• PanGu-$\alpha$: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation
• Yuan 1.0: Large-Scale Pre-trained Language Model in Zero-Shot and Few-Shot Learning
• PLATO-XL: Exploring the Large-scale Pre-training of Dialogue Generation
• Pre-Trained Models: Past, Present and Future
• ChineseBERT: Chinese Pretraining Enhanced by Glyph and Pinyin Information
• Transfer training from smaller language model