Efficient Parallelization Layouts for Large-Scale Distributed Model Training

Johannes Hagemann, Samuel Weinbach, Konstantin Dobler +2

Efficiently training large language models requires parallelizing across hundreds of hardware accelerators and invoking various compute and memory optimizations. When combined, man…