Improving the Serving Performance of Multi-LoRA Large Language Models via Efficient LoRA and KV Cache Management

Hang Zhang, Jiuchen Shi, Yixiao Wang +3

Multiple Low-Rank Adapters (Multi-LoRAs) are gaining popularity for task-specific Large Language Model (LLM) applications. For multi-LoRA serving, caching hot KV caches and LoRA ad…