A Systematic Study of Cross-Layer KV Sharing for Efficient LLM Inference

You Wu, Haoyi Wu, Kewei Tu

Recently, sharing key-value (KV) cache across layers has been found effective in efficient inference of large language models (LLMs). To systematically investigate different techni…