Online Linear Programming for Multi-Objective Routing in LLM Serving

Zixi Chen, Yinyu Ye, Zijie Zhou

We study the online routing problem in large language model serving, where requests arrive sequentially and must be dispatched to parallel decode workers under tight batch-size and…