PanGu-$\alpha$: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation

Abstract

Large-scale Pretrained Language Models (PLMs) have become the new paradigm for Natural Language Processing (NLP). PLMs with hundreds of billions parameters such as GPT-3 have demonstrated strong performances on natural language understanding and generation with \textit{few-shot in-context} learning. In this work, we present our practice on training large-scale autoregressive language models named PanGu-$\alpha$, with up to 200 billion parameters. PanGu-$\alpha$ is developed under the MindSpore and trained on a cluster of 2048 Ascend 910 AI processors. The training parallelism strategy is implemented based on MindSpore Auto-parallel, which composes five parallelism dimensions to scale the training task to 2048 processors efficiently, including data parallelism, op-level model parallelism, pipeline model parallelism, optimizer model parallelism and rematerialization. To enhance the generalization ability of PanGu-$\alpha$, we collect 1.1TB high-quality Chinese data from a wide range of domains to pretrain the model. We empirically test the generation ability of PanGu-$\alpha$ in various scenarios including text summarization, question answering, dialogue generation, etc. Moreover, we investigate the effect of model scales on the few-shot performances across a broad range of Chinese NLP tasks. The experimental results demonstrate the superior capabilities of PanGu-$\alpha$ in performing various tasks under few-shot or zero-shot settings.

arXiv commentThe technique report for PanGu-$\alpha$

References in corpus (15)

• Learning Transferable Visual Models From Natural Language Supervision
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• ERNIE: Enhanced Representation through Knowledge Integration
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• Supporting Very Large Models using Automatic Dataflow Graph Partitioning
• Beyond Data and Model Parallelism for Deep Neural Networks
• On Layer Normalization in the Transformer Architecture
• NEZHA: Neural Contextualized Representation for Chinese Language Understanding
• DRCD: a Chinese Machine Reading Comprehension Dataset
• Dataset and Neural Recurrent Sequence Labeling Model for Open-Domain Factoid Question Answering
• CLUE: A Chinese Language Understanding Evaluation Benchmark
• DuReader: a Chinese Machine Reading Comprehension Dataset from Real-world Applications
• HetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU Clusters through Integration of Pipelined Model Parallelism and Data Parallelism
• CLUECorpus2020: A Large-scale Chinese Corpus for Pre-training Language Model
• CPM: A Large-scale Generative Chinese Pre-trained Language Model

Cited by in corpus (17)

• Compute Trends Across Three Eras of Machine Learning
• ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation
• CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generation
• Merak: An Efficient Distributed DNN Training Framework with Automated 3D Parallelism for Giant Foundation Models
• On Transferability of Prompt Tuning for Natural Language Processing
• AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing
• Mengzi: Towards Lightweight yet Ingenious Pre-trained Models for Chinese
• CoderEval: A Benchmark of Pragmatic Code Generation with Generative Pre-trained Models
• Yuan 1.0: Large-Scale Pre-trained Language Model in Zero-Shot and Few-Shot Learning
• Towards Efficient Post-training Quantization of Pre-trained Language Models
• EVA2.0: Investigating Open-Domain Chinese Dialogue Systems with Large-Scale Pre-Training
• M6-10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretraining
• PLATO-XL: Exploring the Large-scale Pre-training of Dialogue Generation
• Pre-Trained Models: Past, Present and Future
• What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers
• Pro-KD: Progressive Distillation by Following the Footsteps of the Teacher
• Is the Number of Trainable Parameters All That Actually Matters?