11 papers
SR: Selective Sampling, Subspaces, and Sparse Reconstruction for Compressed Long-Context KV Caching
Jialong Han, You Wu, Kewei Tu
The growth of context window lengths in Large Language Models (LLMs) significantly enhances their long-context capabilities but incurs prohibitive memory costs due to the Key-Value…
GRKV: Global Regression for Training-Free KV Cache Compression in Long-Context LLMs
Junjie Peng, You Wu, Haoyi Wu +4
Large language models (LLMs) with extended context lengths rely on the key-value (KV) cache to support attention over prior tokens. However, maintaining the KV cache incurs substan…
Target-Agnostic Calibration under Distribution Shift with Frequency-Aware Gradient Rectification
Yilin Zhang, Cai Xu, You Wu +2
Real-world model deployments inevitably encounter distribution shifts, rendering the confidence estimates of deep neural networks highly unreliable, posing severe risks in safety-c…
Exploring the Potential of Probabilistic Transformer for Time Series Modeling: A Report on the ST-PT Framework
Zhangzhi Xiong, Haoyi Wu, You Wu +3
The Probabilistic Transformer (PT) establishes that the Transformer's self-attention plus its feed-forward block is mathematically equivalent to Mean-Field Variational Inference (M…
SpiralFormer: Looped Transformers Can Learn Hierarchical Dependencies via Multi-Resolution Recursion
Chengting Yu, Xiaobo Shu, Yadao Wang +8
Recursive (looped) Transformers decouple computational depth from parameter depth by repeatedly applying shared layers, providing an explicit architectural primitive for iterative…
YOCO++: Enhancing YOCO with KV Residual Connections for Efficient LLM Inference
You Wu, Ziheng Chen, Yizhen Zhang +6
Cross-layer key-value (KV) compression has been found to be effective in efficient inference of large language models (LLMs). Although they reduce the memory consumption of the KV…