RouterWise: Joint Resource Allocation and Routing for Latency-Aware Multi-Model LLM Serving

Hossein Hosseini Kasnavieh, Gholamreza Haffari, Christopher Leckie +1

Multi-model LLM routing has emerged as an effective approach for reducing serving cost and latency while maintaining output quality by assigning each prompt to an appropriate model…