Learning to Route and Schedule LLMs from User Retrials via Contextual Queueing Bandits

Seoungbin Bae, Junyoung Son, Dabeen Lee

Explosive demands for LLMs often cause user queries to accumulate in server queues, requiring efficient routing (query-LLM matching) and scheduling (query prioritization) mechanism…