CARROT: A Cost Aware Rate Optimal Router

Seamus Somerstep, Felipe Maia Polo, Allysson Flavio Melo de Oliveira +5

With the rapid growth in the number of Large Language Models (LLMs), there has been a recent interest in LLM routing, or directing queries to the cheapest LLM that can deliver a su…