ByteCheckpoint: A Unified Checkpointing System for Large Foundation Model Development

Borui Wan, Mingji Han, Yiyao Sheng +9

Checkpointing to preserve training states is crucial during the development of Large Foundation Models (LFMs), for training resumption upon various failures or changes in GPU resou…