Efficient Routing of Inference Requests across LLM Instances in Cloud-Edge Computing

Shibo Yu, Mohammad Goudarzi, Adel Nadjaran Toosi

The rising demand for Large Language Model (LLM) inference services has intensified pressure on computational resources, resulting in latency and cost challenges. This paper introd…