CASTILLO: Characterizing Response Length Distributions of Large Language Models

Daniel F. Perez-Ramirez, Dejan Kostic, Magnus Boman

Efficiently managing compute resources for Large Language Model (LLM) inference remains challenging due to the inherently stochastic and variable lengths of autoregressive text gen…