CascadeInfer: Length-Aware Scheduling of LLM Serving with Low Latency and Load Balancing

Yitao Yuan, Chenqi Zhao, Bohan Zhao +3

Efficiently harnessing GPU compute is critical to improving user experience and reducing operational costs in large language model (LLM) services. However, current inference engine…