5 papers · 1 filter
BATQuant: Outlier-resilient MXFP4 Quantization via Learnable Block-wise Optimization
Ji-Fu Li, Manyi Zhang, Xiaobo Xia +4
Microscaling floating-point (MXFP) formats have emerged as a promising standard for deploying Multi-modal Large Language Models (MLLMs) and Large Language Models (LLMs) on modern a…
Beyond Masks: Efficient, Flexible Diffusion Language Models via Deletion-Insertion Processes
Fangyu Ding, Ding Ding, Sijin Chen +8
While Masked Diffusion Language Models (MDLMs) relying on token masking and unmasking have shown promise in language modeling, their computational efficiency and generation flexibi…
FreqKV: Key-Value Compression in Frequency Domain for Context Window Extension
Jushi Kai, Yixuan Wang, Boyi Zeng +4
Existing key-value (KV) cache compression methods for large language models (LLMs) often rely on token eviction, which risks losing critical local information in both long prefilli…
WeightedKV: Attention Scores Weighted Key-Value Cache Merging for Large Language Models
Jian Yuan, Ziwei He, Haoli Bai +2
Large Language Models (LLMs) use key-value (KV) cache to reduce redundant computation in autoregressive generation. However, the KV cache size increases linearly during generation,…
TreeKV: Smooth Key-Value Cache Compression with Tree Structures
Ziwei He, Jian Yuan, Haoli Bai +2
Efficient key-value (KV) cache compression is critical for scaling transformer-based Large Language Models (LLMs) in long sequences and resource-limited settings. Existing methods…