11 papers
SR: Selective Sampling, Subspaces, and Sparse Reconstruction for Compressed Long-Context KV Caching
Jialong Han, You Wu, Kewei Tu
The growth of context window lengths in Large Language Models (LLMs) significantly enhances their long-context capabilities but incurs prohibitive memory costs due to the Key-Value…
GRKV: Global Regression for Training-Free KV Cache Compression in Long-Context LLMs
Junjie Peng, You Wu, Haoyi Wu +4
Large language models (LLMs) with extended context lengths rely on the key-value (KV) cache to support attention over prior tokens. However, maintaining the KV cache incurs substan…
Exploring the Potential of Probabilistic Transformer for Time Series Modeling: A Report on the ST-PT Framework
Zhangzhi Xiong, Haoyi Wu, You Wu +3
The Probabilistic Transformer (PT) establishes that the Transformer's self-attention plus its feed-forward block is mathematically equivalent to Mean-Field Variational Inference (M…
YOCO++: Enhancing YOCO with KV Residual Connections for Efficient LLM Inference
You Wu, Ziheng Chen, Yizhen Zhang +6
Cross-layer key-value (KV) compression has been found to be effective in efficient inference of large language models (LLMs). Although they reduce the memory consumption of the KV…
SpiralFormer: Looped Transformers Can Learn Hierarchical Dependencies via Multi-Resolution Recursion
Chengting Yu, Xiaobo Shu, Yadao Wang +8
Recursive (looped) Transformers decouple computational depth from parameter depth by repeatedly applying shared layers, providing an explicit architectural primitive for iterative…
JoyVoice: Long-Context Conditioning for Anthropomorphic Multi-Speaker Conversational Synthesis
Fan Yu, Tao Wang, You Wu +22
Large speech generation models are evolving from single-speaker, short sentence synthesis to multi-speaker, long conversation geneartion. Current long-form speech generation models…
Spectral Alignment as Predictor of Loss Explosion in Neural Network Training
Haiquan Qiu, You Wu, Yingjie Tan +2
Loss explosions in training deep neural networks can nullify multi-million dollar training runs. Conventional monitoring metrics like weight and gradient norms are often lagging an…
Target-Agnostic Calibration under Distribution Shift with Frequency-Aware Gradient Rectification
Yilin Zhang, Cai Xu, You Wu +2
Real-world model deployments inevitably encounter distribution shifts, rendering the confidence estimates of deep neural networks highly unreliable, posing severe risks in safety-c…
Affordable AI Assistants with Knowledge Graph of Thoughts
Maciej Besta, Lorenzo Paleari, Jia Hao Andrea Jiang +15
Large Language Models (LLMs) are revolutionizing the development of AI assistants capable of performing diverse tasks across domains. However, current state-of-the-art LLM-driven a…
Superpose Task-specific Features for Model Merging
Haiquan Qiu, You Wu, Dong Li +2
Model merging enables powerful capabilities in neural networks without requiring additional training. In this paper, we introduce a novel perspective on model merging by leveraging…
A Systematic Study of Cross-Layer KV Sharing for Efficient LLM Inference
You Wu, Haoyi Wu, Kewei Tu
Recently, sharing key-value (KV) cache across layers has been found effective in efficient inference of large language models (LLMs). To systematically investigate different techni…