Mind the Delay: Latency-aware Query Routing for Heterogeneous LLM pools

Shivam Patel, Akaash R. Parthasarathy, Ankur Mallick +2

Large language model (LLM) query routers typically assign each query to a suitable model that balances response accuracy and monetary cost. Subsequently, separate system-level load…