Showing cs.DCShow all
2 papers · 1 filter
cs.DC2025
CascadeInfer: Length-Aware Scheduling of LLM Serving with Low Latency and Load Balancing
Yitao Yuan, Chenqi Zhao, Bohan Zhao +3
Efficiently harnessing GPU compute is critical to improving user experience and reducing operational costs in large language model (LLM) services. However, current inference engine…
cs.DC2025
SIMPLE: Disaggregating Sampling from GPU Inference into a Decision Plane for Faster Distributed LLM Serving
Bohan Zhao, Zane Cao, Yongchao He
As large language models (LLMs) scale out with tensor parallelism (TP) and pipeline parallelism (PP) and production stacks have aggressively optimized the data plane (attention/GEM…