5 citations · 18 across the 12 of their papers we have counts for
8 papers · 1 filter
Recalibrating Many Questions with Few Labels: Minimax Rates and Identification Limits
Xiaoyu Li, Chiwun Yang
Many related prediction problems, which we call questions, come with few labels each. What information can be shared across them, and what is actually necessary for good decisions?…
The Birkhoff Geometry of Manifold-Constrained Hyper-Connections: Two Channels, Vertex Viscosity, and Sinkhorn as a Retraction
Xiaoyu Li, Zhizhou Sha, Chiwun Yang
Hyper-connections widen the residual stream of a Transformer to parallel streams. Their manifold-constrained version (mHC) mixes the streams at each layer with a doubly stochas…
Beyond Sparse Weights: When Is Attention Compressible?
Chiwun Yang, Xiaoyu Li
KV-cache compression is often justified by attention maps with a few large weights. This is incomplete: large weights may not contain most of the mass, omitted values can cancel, a…
Dissecting Submission Limit in Desk-Rejections: A Mathematical Analysis of Fairness in AI Conference Policies
Yuefan Cao, Xiaoyu Li, Yingyu Liang +4
As AI research surges in both impact and volume, conferences have imposed submission limits to maintain paper quality and alleviate organizational pressure. In this work, we examin…
RoPE Attention Can Be Trained in Almost Linear Time
Yang Cao, Jiayan Huo, Yingyu Liang +2
The Rotary Position Embedding (RoPE) mechanism has become a powerful enhancement to the Transformer architecture, which enables models to capture token relationships when encoding…
Grams: Gradient Descent with Adaptive Momentum Scaling
Yang Cao, Xiaoyu Li, Zhao Song
We introduce radient Descent with daptive omentum caling (), a novel optimization algorithm that decouples the direc…
Bypassing the Exponential Dependency: Looped Transformers Efficiently Learn In-context by Multi-step Gradient Descent
Bo Chen, Xiaoyu Li, Yingyu Liang +2
In-context learning has been recognized as a key factor in the success of Large Language Models (LLMs). It refers to the model's ability to learn patterns on the fly from provided…
On Fine-Grained I/O Complexity of Attention Backward Passes
Xiaoyu Li, Yingyu Liang, Zhenmei Shi +3
Large Language Models (LLMs) exhibit exceptional proficiency in handling extensive context windows in natural language. Nevertheless, the quadratic scaling of attention computation…