collaborators

1 paper

cs.DC2024

Intelligent Router for LLM Workloads: Improving Performance Through Workload-Aware Load Balancing

Kunal Jain, Anjaly Parayil, Ankur Mallick +10

Large Language Model (LLM) workloads have distinct prefill and decode phases with different compute and memory requirements which should ideally be accounted for when scheduling in…