When KV Meets Embeddings: Dynamic GPU Memory Allocation for Accelerating Generative Recommender Serving

Wenjun Yu, Shuguang Han, Amelie Chi Zhou

Generative Recommender (GR) inference places embedding hot caches (EMB) and KV caches in direct competition for limited GPU HBM: allocating more memory to one improves its efficien…