Understanding LLM Checkpoint/Restore I/O Strategies and Patterns

Mikaila J. Gossman, Avinash Maurya, Bogdan Nicolae +1

As LLMs and foundation models scale, checkpoint/restore has become a critical pattern for training and inference. With 3D parallelism (tensor, pipeline, data), checkpointing involv…