LLM Query Scheduling with Prefix Reuse and Latency Constraints

Gregory Dexter, Shao Tang, Ata Fatahi Baarzi +3

The efficient deployment of large language models (LLMs) in online settings requires optimizing inference performance under stringent latency constraints, particularly the time-to-…