CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generation

Abstract

In this paper, we take the advantage of previous pre-trained models (PTMs) and propose a novel Chinese Pre-trained Unbalanced Transformer (CPT). Different from previous Chinese PTMs, CPT is designed to utilize the shared knowledge between natural language understanding (NLU) and natural language generation (NLG) to boost the performance. CPT consists of three parts: a shared encoder, an understanding decoder, and a generation decoder. Two specific decoders with a shared encoder are pre-trained with masked language modeling (MLM) and denoising auto-encoding (DAE) tasks, respectively. With the partially shared architecture and multi-task pre-training, CPT can (1) learn specific knowledge of both NLU or NLG tasks with two decoders and (2) be fine-tuned flexibly that fully exploits the potential of the model. Moreover, the unbalanced Transformer saves the computational and storage cost, which makes CPT competitive and greatly accelerates the inference of text generation. Experimental results on a wide range of Chinese NLU and NLG tasks show the effectiveness of CPT.

Code is available at https://github.com/fastnlp/CPT

References in corpus (8)

• ERNIE: Enhanced Representation through Knowledge Integration
• Multilingual Denoising Pre-training for Neural Machine Translation
• Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing
• UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training
• ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation
• PanGu-$\alpha$: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation
• CPM-2: Large-scale Cost-effective Pre-trained Language Models
• ChineseBERT: Chinese Pretraining Enhanced by Glyph and Pinyin Information

Cited by in corpus (4)

• FCGEC: Fine-Grained Corpus for Chinese Grammatical Error Correction
• EVA2.0: Investigating Open-Domain Chinese Dialogue Systems with Large-Scale Pre-Training
• Enhancing User Behavior Sequence Modeling by Generative Tasks for Session Search
• Towards Improving Interpretability of Language Model Generation through a Structured Knowledge Discovery Approach