BeLLMan: Controlling LLM Congestion

Tella Rajashekhar Reddy, Atharva Deshmukh, Karan Tandon +3

Large language model (LLM) applications are blindfolded to the infrastructure underneath and generate tokens autoregressively, indifferent to the system load, thus risking inferenc…