Robust Length Prediction: A Perspective from Heavy-Tailed Prompt-Conditioned Distributions

Jing Wang, Yu-Yang Qian, Ke Xue +3

Output-length prediction is important for efficient LLM serving, as it directly affects batching, memory reservation, and scheduling. For prompt-only length prediction, most existi…