8 papers
RoLaQ: Joint Rotation and Range Calibration for 2-Bit KV Caches in Multi-Head Latent Attention
Shiyu Ji, Peng Ling, Chen Lin +4
Multi-head latent attention (MLA) compresses the content of a key-value cache into a shared latent per token, making low-bit storage attractive for long-context inference. At two b…
EchoKV: Efficient KV Cache Compression via Similarity-Based Reconstruction
Shiyu Ji, Yixuan Wang, Yijun Liu +2
The increasing memory demand of the Key-Value (KV) cache poses a significant bottleneck for Large Language Models (LLMs) in long-context applications. Existing low-rank KV compress…
Seer Self-Consistency: Advance Budget Estimation for Adaptive Test-Time Scaling
Shiyu Ji, Yixuan Wang, Yijun Liu +2
Test-time scaling improves the inference performance of Large Language Models (LLMs) but also incurs substantial computational costs. Although recent studies have reduced token con…
Judge Q: Trainable Queries for Optimized Information Retention in KV Cache Eviction
Yijun Liu, Yixuan Wang, Yuzhuang Xu +4
Large language models (LLMs) utilize key-value (KV) cache to store historical information during sequence processing. The size of KV cache grows linearly as the length of the seque…
CAMERA: Multi-Matrix Joint Compression for MoE Models via Micro-Expert Redundancy Analysis
Yuzhuang Xu, Xu Han, Yuanchi Zhang +5
Large Language Models (LLMs) with Mixture-of-Experts (MoE) architectures are distinguished by their strong performance scaling with increasing parameters across a wide range of tas…
Think Before You Accept: Semantic Reflective Verification for Faster Speculative Decoding
Yixuan Wang, Yijun Liu, Shiyu ji +4
Large language models (LLMs) suffer from high inference latency due to the auto-regressive decoding process. Speculative decoding accelerates inference by generating multiple draft…
Lookahead Q-Cache: Achieving More Consistent KV Cache Eviction via Pseudo Query
Yixuan Wang, Shiyu Ji, Yijun Liu +4
Large language models (LLMs) rely on key-value cache (KV cache) to accelerate decoding by reducing redundant computations. However, the KV cache memory usage grows substantially wi…
CRVQ: Channel-Relaxed Vector Quantization for Extreme Compression of LLMs
Yuzhuang Xu, Shiyu Ji, Qingfu Zhu +1
Powerful large language models (LLMs) are increasingly expected to be deployed with lower computational costs, enabling their capabilities on resource-constrained devices. Post-tra…