On the Efficacy of Eviction Policy for Key-Value Constrained Generative Language Model Inference

Siyu Ren, Kenny Q. Zhu

Despite the recent success associated with Large Language Models (LLMs), they are notably cost-prohibitive to deploy in resource-constrained environments due to their excessive mem…