11 papers
Beyond Sparse Weights: When Is Attention Compressible?
Chiwun Yang, Xiaoyu Li
KV-cache compression is often justified by attention maps with a few large weights. This is incomplete: large weights may not contain most of the mass, omitted values can cancel, a…
RoPE Attention Can Be Trained in Almost Linear Time
Yang Cao, Jiayan Huo, Yingyu Liang +2
The Rotary Position Embedding (RoPE) mechanism has become a powerful enhancement to the Transformer architecture, which enables models to capture token relationships when encoding…
On Fine-Grained I/O Complexity of Attention Backward Passes
Xiaoyu Li, Yingyu Liang, Zhenmei Shi +3
Large Language Models (LLMs) exhibit exceptional proficiency in handling extensive context windows in natural language. Nevertheless, the quadratic scaling of attention computation…
Faster Algorithm for Structured John Ellipsoid Computation
Yang Cao, Xiaoyu Li, Zhao Song +2
The famous theorem of Fritz John states that any convex body has a unique maximal volume inscribed ellipsoid, known as the John Ellipsoid. Computing the John Ellipsoid is a fundame…
Accept More, Reject Less: Reducing up to 19% Unnecessary Desk-Rejections over 11 Years of ICLR Data
Xiaoyu Li, Zhao Song, Jiahao Zhang
The explosive growth of AI research has driven paper submissions at flagship AI conferences to unprecedented levels, necessitating many venues in 2025 (e.g., CVPR, ICCV, KDD, AAAI,…
Grams: Gradient Descent with Adaptive Momentum Scaling
Yang Cao, Xiaoyu Li, Zhao Song
We introduce radient Descent with daptive omentum caling (), a novel optimization algorithm that decouples the direc…