9 papers
FlashRT: Agent Harness for Guiding Agents to Deploy Real-Time Multimodal Applications
Krish Agarwal, Zhuoming Chen, Yanyuan Qin +3
Real-time multimodal applications, including voice agents and interactive video generation, compose heterogeneous models into pipelines whose efficient deployment requires applicat…
FSA: An Alternative Efficient Implementation of Native Sparse Attention Kernel
Ran Yan, Youhe Jiang, Zhuoming Chen +3
Recent advances in sparse attention mechanisms have demonstrated strong potential for reducing the computational cost of long-context training and inference in large language model…
Vortex: Efficient and Programmable Sparse Attention Serving for AI Agents
Zhuoming Chen, Xinrui Zhong, Qilong Feng +5
Sparse attention is becoming increasingly important for serving large language models (LLMs) as generation lengths continue to grow. However, deploying and evaluating new sparse at…
WWW.Serve: Interconnecting Global LLM Services through Decentralization
Huanyu Wang, Ziyu Xia, Zhuoming Chen +1
Large language model (LLM) services are mostly centralized, leading to scalability bottlenecks and underutilization of substantial scattered GPU resources. While decentralization o…
Sequoia: Scalable, Robust, and Hardware-aware Speculative Decoding
Zhuoming Chen, Avner May, Ruslan Svirschevski +4
As the usage of large language models (LLMs) grows, performing efficient inference with these models becomes increasingly important. While speculative decoding has recently emerged…
AdaServe: Accelerating Multi-SLO LLM Serving with SLO-Customized Speculative Decoding
Zikun Li, Zhuofu Chen, Remi Delacourt +11
Modern large language model (LLM) applications exhibit diverse service-level objectives (SLOs), from low-latency requirements in interactive coding assistants to more relaxed const…