KV-Pipe: On the Relation Between KV Sharing and Pipeline Parallel Efficiency in LLMs

Maryam Dialameh, Hossein Rajabzadeh, Harish Krishnamoorthy Murali +3

Pipeline parallelism (PP) is widely used to scale large language model (LLM) training, but its efficiency is often limited by stage imbalance and pipeline bubbles. Meanwhile, cross…