4 papers
WeightedKV: Attention Scores Weighted Key-Value Cache Merging for Large Language Models
Jian Yuan, Ziwei He, Haoli Bai +2
Large Language Models (LLMs) use key-value (KV) cache to reduce redundant computation in autoregressive generation. However, the KV cache size increases linearly during generation,…
TreeKV: Smooth Key-Value Cache Compression with Tree Structures
Ziwei He, Jian Yuan, Haoli Bai +2
Efficient key-value (KV) cache compression is critical for scaling transformer-based Large Language Models (LLMs) in long sequences and resource-limited settings. Existing methods…
A New Adaptive Balanced Augmented Lagrangian Method with Application to ISAC Beamforming Design
Jiageng Wu, Bo Jiang, Xinxin Li +2
In this paper, we consider a class of convex programming problems with linear equality constraints, which finds broad applications in machine learning and signal processing. We pro…
Fovea Transformer: Efficient Long-Context Modeling with Structured Fine-to-Coarse Attention
Ziwei He, Jian Yuan, Le Zhou +2
The quadratic complexity of self-attention in Transformers has hindered the processing of long text. To alleviate this problem, previous works have proposed to sparsify the attenti…