Towards Cost-effective LLMs Routing with Batch Prompting

Haotian Xu, Kangfei Zhao, Jiadong Xie

Large Language Model (LLM) serving systems must balance task performance against monetary cost. Two prominent optimization techniques have emerged independently: LLM routing, which…