Galvatron: Automatic Distributed Training for Large Transformer Models

Esmail Gumaan

Training multi-billion to trillion-parameter language models efficiently on GPU clusters requires leveraging multiple parallelism strategies. We present Galvatron, a novel open-sou…