11 papers
DA-DLM: Explicitly Modeling Token Dependencies in Diffusion Language Models
Pengyu Ji, Zichen Zhang, Xiang Hu +1
Diffusion Language Models (DLMs) generate text by iteratively denoising a masked sequence, independently predicting multiple tokens at each step. This conditional independence disc…
SR: Selective Sampling, Subspaces, and Sparse Reconstruction for Compressed Long-Context KV Caching
Jialong Han, You Wu, Kewei Tu
The growth of context window lengths in Large Language Models (LLMs) significantly enhances their long-context capabilities but incurs prohibitive memory costs due to the Key-Value…
Evolution-Aware MSA Reasoning for Subsampling via Factor Graphs
Zhangzhi Xiong, Minzhang Li, Haotian Yu +6
Multiple Sequence Alignments (MSAs) provide protein language models with explicit evolutionary context, but their large depth makes subsampling unavoidable under limited token budg…
Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling
Xiang Hu, Xinyu Wei, Hao Gu +10
Scaling modern large language models (LLMs) to long contexts is limited by the quadratic computation cost, and poor length extrapolation of dense attention. Chunk-wise sparse atten…
GRKV: Global Regression for Training-Free KV Cache Compression in Long-Context LLMs
Junjie Peng, You Wu, Haoyi Wu +4
Large language models (LLMs) with extended context lengths rely on the key-value (KV) cache to support attention over prior tokens. However, maintaining the KV cache incurs substan…
Exploring the Potential of Probabilistic Transformer for Time Series Modeling: A Report on the ST-PT Framework
Zhangzhi Xiong, Haoyi Wu, You Wu +3
The Probabilistic Transformer (PT) establishes that the Transformer's self-attention plus its feed-forward block is mathematically equivalent to Mean-Field Variational Inference (M…
YOCO++: Enhancing YOCO with KV Residual Connections for Efficient LLM Inference
You Wu, Ziheng Chen, Yizhen Zhang +6
Cross-layer key-value (KV) compression has been found to be effective in efficient inference of large language models (LLMs). Although they reduce the memory consumption of the KV…
Flash Multi-Head Feed-Forward Network
Minshen Zhang, Xiang Hu, Jianguo Li +2
We explore Multi-Head FFN (MH-FFN) as a replacement of FFN in the Transformer architecture, motivated by the structural similarity between single-head attention and FFN. While mult…
A Systematic Study of Compositional Syntactic Transformer Language Models
Yida Zhao, Hao Xve, Xiang Hu +1
Syntactic language models (SLMs) enhance Transformers by incorporating syntactic biases through the modeling of linearized syntactic parse trees alongside surface sentences. This p…
Hardware-aligned Hierarchical Sparse Attention for Efficient Long-term Memory Access
Xiang Hu, Jiaqi Leng, Jun Zhao +2
A key advantage of Recurrent Neural Networks (RNNs) over Transformers is their linear computational and space complexity enables faster training and inference for long sequences. H…
Efficient Length-Generalizable Attention via Causal Retrieval for Long-Context Language Modeling
Xiang Hu, Zhihao Teng, Jun Zhao +2
Despite the success of Transformers, handling long contexts remains challenging due to the limited length generalization and quadratic complexity of self-attention. Thus Transforme…