2 papers
cs.DC2026
Lodestar: An Online-Learning LLM Inference Router
Gangmuk Lim, Wanyu Zhao, Brighten Godfrey +3
Efficiently serving large language model (LLM) inference tasks is crucial both for user-perceived latency such as time-to-first-token (TTFT) and for GPU utilization. However, LLM r…
cs.DC2023
Kivi: Verification for Cluster Management
Bingzhe Liu, Gangmuk Lim, Ryan Beckett +1
Modern cloud infrastructure is powered by cluster management systems such as Kubernetes and Docker Swarm. While these systems seek to minimize users' operational burden, the comple…