5 papers
RoPE Attention Can Be Trained in Almost Linear Time
Yang Cao, Jiayan Huo, Yingyu Liang +2
The Rotary Position Embedding (RoPE) mechanism has become a powerful enhancement to the Transformer architecture, which enables models to capture token relationships when encoding…
Time and Memory Trade-off of KV-Cache Compression in Tensor Transformer Decoding
Yifang Chen, Xiaoyu Li, Yingyu Liang +3
The key-value (KV) cache in the tensor version of transformers presents a significant bottleneck during inference. While previous work analyzes the fundamental space complexity bar…
Scaling Law Phenomena Across Regression Paradigms: Multiple and Kernel Approaches
Yifang Chen, Xuyang Guo, Xiaoyu Li +3
Recently, Large Language Models (LLMs) have achieved remarkable success. A key factor behind this success is the scaling law observed by OpenAI. Specifically, for models with Trans…
The Computational Limits of State-Space Models and Mamba via the Lens of Circuit Complexity
Yifang Chen, Xiaoyu Li, Yingyu Liang +2
In this paper, we analyze the computational limitations of Mamba and State-space Models (SSMs) by using the circuit complexity framework. Despite Mamba's stateful design and recent…
Universal Approximation of Visual Autoregressive Transformers
Yifang Chen, Xiaoyu Li, Yingyu Liang +2
We investigate the fundamental limits of transformer-based foundation models, extending our analysis to include Visual Autoregressive (VAR) transformers. VAR represents a big step…