TTrace: Lightweight Error Checking and Diagnosis for Distributed Training

Haitian Jiang, Shaowei Zhu, Zhen Zhang +5

Distributed training is essential for scaling the training of large neural network models, such as large language models (LLMs), across thousands of GPUs. However, the complexity o…