Justitia: Fair and Efficient Scheduling of Task-parallel LLM Agents with Selective Pampering

Mingyan Yang, Guanjie Wang, Manqi Luo +6

LLM agents, which often comprise parallel inference tasks, are commonly adopted to solve real-world problems. When serving such task-parallel LLM agents in shared GPU servers, the…