PHOENIX: Resilient LLM Training with Hot-Swapping via Zero-Overhead Checkpoint

Haotian Xie, Junlin Chen, Mingkai Zheng +2

State-of-the-art large language model (LLM) training takes tens of thousands of graphics processing units (GPUs) for months and encounters failures across the software and hardware…