5 citations · 18 across the 12 of their papers we have counts for
8 papers · 1 filter
RoPE Attention Can Be Trained in Almost Linear Time
Yang Cao, Jiayan Huo, Yingyu Liang +2
The Rotary Position Embedding (RoPE) mechanism has become a powerful enhancement to the Transformer architecture, which enables models to capture token relationships when encoding…
Grams: Gradient Descent with Adaptive Momentum Scaling
Yang Cao, Xiaoyu Li, Zhao Song
We introduce radient Descent with daptive omentum caling (), a novel optimization algorithm that decouples the direc…
The Computational Limits of State-Space Models and Mamba via the Lens of Circuit Complexity
Yifang Chen, Xiaoyu Li, Yingyu Liang +2
In this paper, we analyze the computational limitations of Mamba and State-space Models (SSMs) by using the circuit complexity framework. Despite Mamba's stateful design and recent…
Bypassing the Exponential Dependency: Looped Transformers Efficiently Learn In-context by Multi-step Gradient Descent
Bo Chen, Xiaoyu Li, Yingyu Liang +2
In-context learning has been recognized as a key factor in the success of Large Language Models (LLMs). It refers to the model's ability to learn patterns on the fly from provided…
On Fine-Grained I/O Complexity of Attention Backward Passes
Xiaoyu Li, Yingyu Liang, Zhenmei Shi +3
Large Language Models (LLMs) exhibit exceptional proficiency in handling extensive context windows in natural language. Nevertheless, the quadratic scaling of attention computation…
Quantum Speedups for Approximating the John Ellipsoid
Xiaoyu Li, Zhao Song, Junwei Yu
In 1948, Fritz John proposed a theorem stating that every convex body has a unique maximal volume inscribed ellipsoid, known as the John ellipsoid. The John ellipsoid has become fu…
A Tighter Complexity Analysis of SparseGPT
Xiaoyu Li, Yingyu Liang, Zhenmei Shi +1
In this work, we improved the analysis of the running time of SparseGPT [Frantar, Alistarh ICML 2023] from to for any $a \in […
Fast John Ellipsoid Computation with Differential Privacy Optimization
Xiaoyu Li, Yingyu Liang, Zhenmei Shi +2
Determining the John ellipsoid - the largest volume ellipsoid contained within a convex polytope - is a fundamental problem with applications in machine learning, optimization, and…