Showing cs.LGShow all
2 papers · 1 filter
cs.LG2025
PipeOffload: Improving Scalability of Pipeline Parallelism with Memory Optimization
Xinyi Wan, Penghui Qi, Guangxing Huang +2
Pipeline parallelism (PP) is widely used for training large language models (LLMs), yet its scalability is often constrained by high activation memory consumption as the number of…
cs.LG2024
Pipeline Parallelism with Controllable Memory
Penghui Qi, Xinyi Wan, Nyamdavaa Amar +1
Pipeline parallelism has been widely explored, but most existing schedules lack a systematic methodology. In this paper, we propose a framework to decompose pipeline schedules as r…