SparPR: Runtime KV-Cache Recovery for Resource-Constrained LLM Inference on Consumer Hardware

Jieyun Deng, Keming Hu, Yanghe Liu +1

Running large language models (LLMs) on consumer hardware is bottlenecked by the key-value (KV) cache, which grows with every decoded token and quickly exhausts the memory and band…