OUTLETS: Output-Length Prediction from Speculative Decoding Backbones

Weihuang Wen, Yingying Liu, Yichuan Liu +5

The heavy-tailed distribution of output lengths in Large Language Model (LLM) serving poses major challenges for resource provisioning and cluster scheduling. Although output-lengt…