HW-Router: Hardware-Aware Routing for Scalable Multi-LLM Serving

Ahasan Kabir, Jiaqi Xue, Mengxin Zheng +1

Modern large language model (LLM) serving platforms deploy multiple models across different GPUs, requiring routers to direct incoming queries to appropriate LLMs. However, existin…