Cluster, Route, Escalate: Cascaded Framework for Cost-Aware LLM Serving

Yasmin Moslem, Magdalena Kacmajor, Vasudevan Nedumpozhimana +11

Efficient deployment of large language models (LLMs) in production forces a trade-off between accuracy and cost. Operators often default to a single model that is either expensive…