RequestRouter: Request-Boundary Routing for Efficient Single-GPU LLM Inference

Aman Sunesh, Ali Alshehhi, Hivansh Dhakne

RequestRouter is a lightweight request-boundary controller for reducing the latency and energy cost of single-GPU large language model inference. Rather than serving all requests w…