2 papers
cs.LG2026
LKV: End-to-End Learning of Head-wise Budgets and Token Selection for LLM KV Cache Eviction
Enshuai Zhou, Yifan Hao, Chao Wang +7
Long-context inference in Large Language Models (LLMs) is bottlenecked by the linear growth of Key-Value (KV) cache memory. Existing KV cache compression paradigms are fundamentall…
cs.AR2024
Cambricon-LLM: A Chiplet-Based Hybrid Architecture for On-Device Inference of 70B LLM
Zhongkai Yu, Shengwen Liang, Tianyun Ma +12
Deploying advanced large language models on edge devices, such as smartphones and robotics, is a growing trend that enhances user data privacy and network connectivity resilience w…