Learned Best-Effort LLM Serving

Siddharth Jha, Coleman Hooper, Xiaoxuan Liu +2

Many applications must provide low-latency LLM service to users or risk unacceptable user experience. However, over-provisioning resources to serve fluctuating request patterns is…