SmartSwap: Swap-Based Memory Optimization for LLM Training under Varying Operator Sequences

Zibo Wang, Yuhang Zhou, Zhibin Wang +13

The increasing size of large language models (LLMs) has led to a surge in memory requirements during training, often exceeding the capacity of high-bandwidth memory (HBM). Swap-bas…