A Method for Building Large Language Models with Predefined KV Cache Capacity

Zhonghua Yi, Ge Niu, Lei Wang +2

This paper introduces a novel approach, the Bounded-Cache Transformer (BCT), for building large language models with a predefined Key-Value (KV) cache capacity. The BCT addresses t…