4 papers
Arena: Efficiently Training Large Models via Dynamic Scheduling and Adaptive Parallelism Co-Design
Chunyu Xue, Weihao Cui, Quan Chen +10
Efficiently training large-scale models (LMs) in GPU clusters involves two separate avenues: inter-job dynamic scheduling and intra-job adaptive parallelism (AP). However, existing…
SageSched: Efficient LLM Scheduling Confronting Demand Uncertainty and Hybridity
Zhenghao Gan, Yichen Bao, Yifei Liu +3
Efficient LLM inference scheduling is crucial for user experience. However, LLM inferences exhibit remarkable demand uncertainty (with unknown output length beforehand) and hybridi…
Improving the Serving Performance of Multi-LoRA Large Language Models via Efficient LoRA and KV Cache Management
Hang Zhang, Jiuchen Shi, Yixiao Wang +3
Multiple Low-Rank Adapters (Multi-LoRAs) are gaining popularity for task-specific Large Language Model (LLM) applications. For multi-LoRA serving, caching hot KV caches and LoRA ad…
A Microservice Graph Generator with Production Characteristics
Fanrong Du, Jiuchen Shi, Quan Chen +2
A production microservice application may provide multiple services, queries of a service may have different call graphs, and a microservice may be shared across call graphs. It is…