ZenFlow: Enabling Stall-Free Offloading for LLM Training

Tingfeng Lan, Yusen Wu, Bin Ma +7

Fine-tuning large language models (LLMs) often exceeds GPU memory limits, prompting systems to offload model states to CPU memory. However, existing offloaded training frameworks l…