Jenga: Effective Memory Management for Serving LLM with Heterogeneity

Chen Zhang, Kuntai Du, Shu Liu +10

Large language models (LLMs) are widely used but expensive to run, especially as inference workloads grow. To lower costs, maximizing the request batch size by managing GPU memory…