SkyWalker: A Locality-Aware Cross-Region Load Balancer for LLM Inference

Tian Xia, Ziming Mao, Jamison Kerney +5

Serving Large Language Models (LLMs) efficiently in multi-region setups remains a challenge. Due to cost and GPU availability concerns, providers typically deploy LLMs in multiple…