STAR: Decode-Phase Rescheduling for LLM Inference

Zhibin Wang, Zetao Hong, Xue Li +8

Large Language Model (LLM) inference has emerged as a fundamental paradigm, however, variations in output length cause severe workload imbalance in the decode phase, particularly f…