Showing cs.CLShow all
2 papers · 1 filter
cs.CL2025
In-context KV-Cache Eviction for LLMs via Attention-Gate
Zihao Zeng, Bokai Lin, Tianqi Hou +2
The KV-Cache technique has become the standard for the inference of large language models (LLMs). Yet, it is widely criticized that KV-Cache can become a bottleneck of the LLM infe…
cs.CL2024
CLLMs: Consistency Large Language Models
Siqi Kou, Lanxiang Hu, Zhezhi He +2
Parallel decoding methods such as Jacobi decoding show promise for more efficient LLM inference as it breaks the sequential nature of the LLM decoding process and transforms it int…