6 papers
Online Linear Programming for Multi-Objective Routing in LLM Serving
Zixi Chen, Yinyu Ye, Zijie Zhou
We study the online routing problem in large language model serving, where requests arrive sequentially and must be dispatched to parallel decode workers under tight batch-size and…
Tackling the Data-Parallel Load Balancing Bottleneck in LLM Serving: Practical Online Routing at Scale
Tianci Bu, Yuan Lyu, Zixi Chen +6
Data-parallel (DP) load balancing has emerged as a first-order bottleneck in large-scale LLM serving. When a model is sharded across devices via tensor parallelism (TP) or expert p…
MoE-SpAc: Efficient MoE Inference Based on Speculative Activation Utility in Heterogeneous Edge Scenarios
Shuhuai Li, Jianghao Lin, Dongdong Ge +1
Mixture-of-Experts (MoE) models enable scalable performance but face severe memory constraints on edge devices. Existing offloading strategies struggle with I/O bottlenecks due to…
Load Balancing with Evolving Workloads in Large-Scale Language Model Serving
Zixi Chen, Tianci Bu, Chendong Song +3
Serving large language models requires assigning requests whose workloads grow during processing and whose completion times are unknown. With a synchronization barrier, lightly loa…
Adaptively Robust LLM Inference Optimization under Prediction Uncertainty
Zixi Chen, Yinyu Ye, Zijie Zhou
We study the problem of optimizing Large Language Model (LLM) inference scheduling to minimize total latency. LLM inference is an online and multi-task service process and also hea…
LLM Serving Optimization with Variable Prefill and Decode Lengths
Meixuan Wang, Yinyu Ye, Zijie Zhou
We study offline scheduling for large language model (LLM) serving under a fixed KV-cache memory budget, where requests have heterogeneous prompt (prefill) and response (decode) le…