Harvest: Opportunistic Peer-to-Peer GPU Caching for LLM Inference

Nikhil Gopal, Kostis Kaffes

Large Language Model (LLM) inference is increasingly constrained by GPU memory capacity rather than compute throughput, driven by growing model sizes and the linear growth of the k…