FairBatching: Fairness-Aware Batch Formation for LLM Inference

Hongtao Lyu, Boyue Liu, Mingyu Wu +1

Large language model (LLM) inference systems face a fundamental tension between minimizing Time-to-First-Token (TTFT) latency for new requests and maintaining a high, steady token…