4 papers
Service-Induced Congestion in Memory-Constrained LLM Serving
Ruicheng Ao, Jing Dong, Gan Luo +1
In large language model (LLM) serving, each request accumulates persistent graphics processing unit (GPU) memory during service as its key-value cache grows with every generated to…
Achieving Linear Speedup and Near-Optimal Complexity for Decentralized Optimization over Row-stochastic Networks
Liyuan Liang, Xinyi Chen, Gan Luo +1
A key challenge in decentralized optimization is determining the optimal convergence rate and designing algorithms to achieve it. While this problem has been extensively addressed…
On the Linear Speedup of the Push-Pull Method for Decentralized Optimization over Digraphs
Liyuan Liang, Gan Luo, Kun Yuan
The linear speedup property is essential for demonstrating the advantage of distributed algorithms over their single-node counterparts. In this paper, we study the stochastic Push-…
Optimizing LLM Inference: Fluid-Guided Online Scheduling with Memory Constraints
Ruicheng Ao, Gan Luo, David Simchi-Levi +1
Large language models now serve millions of users daily, with providers incurring costs exceeding $700,000 per day. Each request requires token-by-token inference, making GPU sched…