PanGu-Σ: Towards Trillion Parameter Language Model with Sparse Heterogeneous Computing

Xiaozhe Ren, Pingyi Zhou, Xinfan Meng +14

The scaling of large language models has greatly improved natural language understanding, generation, and reasoning. In this work, we develop a system that trained a trillion-param…