4 papers
CascadeInfer: Length-Aware Scheduling of LLM Serving with Low Latency and Load Balancing
Yitao Yuan, Chenqi Zhao, Bohan Zhao +3
Efficiently harnessing GPU compute is critical to improving user experience and reducing operational costs in large language model (LLM) services. However, current inference engine…
EconAI: Dynamic Persona Evolution and Memory-Aware Agents in Evolving Economic Environments
Annie Liu, Zane Cao, Lang Chen +2
The integration of large language models (LLMs) in economic simulations has significantly enhanced agent-based modeling, yet existing frameworks struggle to capture the interplay b…
A Unified Sparse Attention via Multi-Granularity Compression
Siran Liu, Zane Cao, Yongchao He
Efficient long-context understanding and reasoning are increasingly vital for large language model (LLM) applications such as multi-turn dialogue and program analysis. However, the…
SIMPLE: Disaggregating Sampling from GPU Inference into a Decision Plane for Faster Distributed LLM Serving
Bohan Zhao, Zane Cao, Yongchao He
As large language models (LLMs) scale out with tensor parallelism (TP) and pipeline parallelism (PP) and production stacks have aggressively optimized the data plane (attention/GEM…