Automatically Planning Optimal Parallel Strategy for Large Language Models

Zongbiao Li, Xiezhao Li, Yinghao Cui +11

The number of parameters in large-scale language models based on transformers is gradually increasing, and the scale of computing clusters is also growing. The technology of quickl…