veScale: Consistent and Efficient Tensor Programming with Eager-Mode SPMD

Youjie Li, Cheng Wan, Zhiqi Lin +10

Large Language Models (LLMs) have scaled rapidly in size and complexity, requiring increasingly intricate parallelism for distributed training, such as 3D parallelism. This sophist…