AdaSpec: Adaptive Speculative Decoding for Fast, SLO-Aware Large Language Model Serving

Kaiyu Huang, Hao Wu, Zhubo Shi +3

Cloud-based Large Language Model (LLM) services often face challenges in achieving low inference latency and meeting Service Level Objectives (SLOs) under dynamic request patterns.…