10 papers
Vidu S1: A Real-Time Interactive Video Generation Model
Jintao Zhang, Kai Jiang, Jintao Chen +24
We introduce Vidu S1, a real-time interactive video generation model supporting voice control of digital characters. Users can control video generation content at any moment throug…
Demystifying the Design Space and Best Practices for Heterogeneous LLM Inference and Serving
Zhixin Wang, Zhengbo Wang, Fangcheng Fu +15
Heterogeneous prefill-decode (PD) inference is now in production: prefill on cost-efficient or supply-available accelerators, decode on bandwidth-strong ones, and KV state crossing…
TurboServe: Serving Streaming Video Generation Efficiently and Economically
Youhe Jiang, Haoxu Wang, Haotong Bao +5
Streaming video generation is emerging as a new serving workload in which users interact with long-lived sessions that generate video progressively, chunk by chunk. Unlike offline…
LMDeploy Accelerates Mixed-Precision LLM Inference with TurboMind
Li Zhang, Youhe Jiang, Guoliang He +6
Mixed-precision inference techniques reduce the memory and computational demands of Large Language Models (LLMs) by applying hybrid precision formats to model weights, activations,…
Efficient Serving for Dynamic Agent Workflows with Prediction-based KV-Cache Management
Haoyu Zheng, Fangcheng Fu, Jia Wu +6
LLM-based workflows compose specialized agents to execute complex tasks, and these agents usually share substantial context, allowing KV-Cache reuse to save computation. Existing a…
OServe: Accelerating LLM Serving via Spatial-Temporal Workload Orchestration
Youhe Jiang, Fangcheng Fu, Taiyi Wang +2
Serving Large Language Models (LLMs) can benefit immensely from parallelizing both the model and input requests across multiple devices, but incoming workloads exhibit substantial…