4 papers · 1 filter
YOCO++: Enhancing YOCO with KV Residual Connections for Efficient LLM Inference
You Wu, Ziheng Chen, Yizhen Zhang +6
Cross-layer key-value (KV) compression has been found to be effective in efficient inference of large language models (LLMs). Although they reduce the memory consumption of the KV…
A Systematic Study of Compositional Syntactic Transformer Language Models
Yida Zhao, Hao Xve, Xiang Hu +1
Syntactic language models (SLMs) enhance Transformers by incorporating syntactic biases through the modeling of linearized syntactic parse trees alongside surface sentences. This p…
Hardware-aligned Hierarchical Sparse Attention for Efficient Long-term Memory Access
Xiang Hu, Jiaqi Leng, Jun Zhao +2
A key advantage of Recurrent Neural Networks (RNNs) over Transformers is their linear computational and space complexity enables faster training and inference for long sequences. H…
Efficient Length-Generalizable Attention via Causal Retrieval for Long-Context Language Modeling
Xiang Hu, Zhihao Teng, Jun Zhao +2
Despite the success of Transformers, handling long contexts remains challenging due to the limited length generalization and quadratic complexity of self-attention. Thus Transforme…