1 citations · 1 across the 3 of their papers we have counts for
5 papers
From Overloaded to Guaranteed: High-Throughput Multi-SLO Enforcement for LoRA-Assisted On-Premise LLM Deployment
Zeshen Zhang, Han Zhao, Weihao Cui +9
As Large Language Models (LLMs) become essential in privacy-sensitive sectors like hospitals and government agencies, the on-premise LLM servers offer a cost-effective and secure a…
Capture the lifecycle: KV Cache management in ReAct Agents with KVTether
Kaihua Fu, Yukun Zhou, Chaokun Chang +6
Efficient serving of long-context reasoning-and-acting (ReAct) agents relies on KV cache reuse to reduce large language model (LLM) prefill latency and monetary cost. However, a se…
Kairos: Low-latency Multi-Agent Serving with Shared LLMs and Excessive Loads in the Public Cloud
Jinyuan Chen, Jiuchen Shi, Quan Chen +1
Multi-agent applications utilize the advanced capabilities of large language models (LLMs) for intricate task completion through agent collaboration in a workflow. Under this situa…
Improving the Serving Performance of Multi-LoRA Large Language Models via Efficient LoRA and KV Cache Management
Hang Zhang, Jiuchen Shi, Yixiao Wang +3
Multiple Low-Rank Adapters (Multi-LoRAs) are gaining popularity for task-specific Large Language Model (LLM) applications. For multi-LoRA serving, caching hot KV caches and LoRA ad…
A Microservice Graph Generator with Production Characteristics
Fanrong Du, Jiuchen Shi, Quan Chen +2
A production microservice application may provide multiple services, queries of a service may have different call graphs, and a microservice may be shared across call graphs. It is…