Memory Offloading for Large Language Model Inference with Latency SLO Guarantees

Chenxiang Ma, Zhisheng Ye, Hanyu Zhao +9

Offloading large language models (LLMs) state to host memory during inference promises to reduce operational costs by supporting larger models, longer inputs, and larger batch size…