29 papers
Prism: Cost-Efficient Multi-LLM Serving via GPU Memory Ballooning
Shan Yu, Yifan Qiao, Mingyuan Ma +18
Inference providers must maintain availability for many LLMs, including low-volume but essential models, making resource efficiency increasingly important as token prices fall. Ana…
vAttention: Verified Sparse Attention
Aditya Desai, Kumar Krishna Agrawal, Shuo Yang +5
State-of-the-art sparse attention methods for reducing decoding latency fall into two main categories: approximate top- (and its extension, top-) and recently introduced samp…
Inference Time Context Sparsity: Illusion or Opportunity?
Sahil Joshi, Prithvi Dixit, Agniva Chowdhury +5
Sparsity has long been a central theme in LLM efficiency, but its role in context processing remains unresolved. As LLM workloads shift toward longer contexts and agentic interacti…
The Time is Here for Just-in-Time Systems: Challenges and Opportunities
Shu Liu, Alexander Krentsel, Shubham Agarwal +8
Core systems like key-value stores have historically taken years to build, and are designed to be general so as to amortize cost across deployments, paying a significant performanc…
Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutation
Shuo Yang, Haocheng Xi, Yilong Zhao +10
Diffusion Transformers (DiTs) are essential for video generation but suffer from significant latency due to the quadratic complexity of attention. By computing only critical tokens…
Quant VideoGen: Auto-Regressive Long Video Generation via 2-Bit KV-Cache Quantization
Haocheng Xi, Shuo Yang, Yilong Zhao +13
Despite rapid progress in autoregressive video diffusion, an emerging system algorithm bottleneck limits both deployability and generation capability: KV cache memory. In autoregre…