1 citations · 1 across the 8 of their papers we have counts for
10 papers
From Overloaded to Guaranteed: High-Throughput Multi-SLO Enforcement for LoRA-Assisted On-Premise LLM Deployment
Zeshen Zhang, Han Zhao, Weihao Cui +9
As Large Language Models (LLMs) become essential in privacy-sensitive sectors like hospitals and government agencies, the on-premise LLM servers offer a cost-effective and secure a…
Capture the lifecycle: KV Cache management in ReAct Agents with KVTether
Kaihua Fu, Yukun Zhou, Chaokun Chang +6
Efficient serving of long-context reasoning-and-acting (ReAct) agents relies on KV cache reuse to reduce large language model (LLM) prefill latency and monetary cost. However, a se…
DSTAR: Accelerating Diffusion Transformers via Spatial and Temporal Redundancy Reduction
Chi Zhang, Jieru Zhao, Yu Feng +3
Diffusion Transformers (DiTs) have been widely used in many tasks, including image synthesis, video generation, and content editing. However, their multi-iteration inference proces…
MegaScale-Omni: A Hyper-Scale, Workload-Resilient System for MultiModal LLM Training in Production
Chunyu Xue, Yangrui Chen, Jianyu Jiang +14
As the foundational component of versatile AI applications, training an multimodal large language model (MLLM) relies on multimodal datasets with dynamic modality mixture proportio…
S-HPLB: Efficient LLM Attention Serving via Sparsity-Aware Head Parallelism Load Balance
Di Liu, Yifei Liu, Chen Chen +4
With the increasing volumes of Large Language Models (LLMs) and the expanding context lengths, attention computation has become a key performance bottleneck in LLM serving. For fas…
MuxTune: Efficient Multi-Task LLM Fine-Tuning in Multi-Tenant Datacenters via Spatial-Temporal Backbone Multiplexing
Chunyu Xue, Yi Pan, Weihao Cui +4
Parameter-Efficient Fine-Tuning (PEFT) is widely applied as the backend of fine-tuning APIs for large language model (LLM) customization in datacenters. Service providers deploy se…
Kairos: Low-latency Multi-Agent Serving with Shared LLMs and Excessive Loads in the Public Cloud
Jinyuan Chen, Jiuchen Shi, Quan Chen +1
Multi-agent applications utilize the advanced capabilities of large language models (LLMs) for intricate task completion through agent collaboration in a workflow. Under this situa…
Efficient Unified Caching for Accelerating Heterogeneous AI Workloads
Tianze Wang, Yifei Liu, Chen Chen +8
Modern AI clusters, which host diverse workloads like data pre-processing, training and inference, often store the large-volume data in cloud storage and employ caching frameworks…
Comet: Fine-grained Computation-communication Overlapping for Mixture-of-Experts
Shulai Zhang, Ningxin Zheng, Haibin Lin +9
Mixture-of-experts (MoE) has been extensively employed to scale large language models to trillion-plus parameters while maintaining a fixed computational cost. The development of l…
SparseTem: Boosting the Efficiency of CNN-Based Video Encoders by Exploiting Temporal Continuity
Kunyun Wang, Shuo Yang, Jieru Zhao +4
Deep learning models have become pivotal in the field of video processing and is increasingly critical in practical applications such as autonomous driving and object detection. Al…