4 papers
Dual Dimensionality for Local and Global Attention
Zhiyuan Wang, Xuan Luo, Sirui Zeng +1
Decoder-only Transformers compute attention over the KV cache of preceding tokens. Keys (and Values) are typically represented with the same dimensionality, regardless of its dista…
Learning When Not to Attend Globally
Xuan Luo, Kailai Zhang, Xifeng Yan
When reading books, humans focus primarily on the current page, flipping back to recap prior context only when necessary. Similarly, we demonstrate that Large Language Models (LLMs…
Direct Multi-Token Decoding
Xuan Luo, Weizhi Wang, Xifeng Yan
Decoder-only transformers have become the standard architecture for large language models (LLMs) due to their strong performance. Recent studies suggest that, in pre-trained LLMs,…
Learning 1D Causal Visual Representation with De-focus Attention Networks
Chenxin Tao, Xizhou Zhu, Shiqian Su +8
Modality differences have led to the development of heterogeneous architectures for vision and language models. While images typically require 2D non-causal modeling, texts utilize…