2 citations · 9 across the 25 of their papers we have counts for
14 papers · 1 filter
A Categorical Atlas of Parameter-Efficient Fine-Tuning: Cones, Orbits, and Gauges
Xiaoyu Li, Zhizhou Sha, Chiwun Yang +1
Parameter-efficient fine-tuning has hundreds of methods, usually compared by parameter count and benchmark score. We compare them as maps. A pretrained checkpoint is a base point i…
The Birkhoff Geometry of Manifold-Constrained Hyper-Connections: Two Channels, Vertex Viscosity, and Sinkhorn as a Retraction
Xiaoyu Li, Zhizhou Sha, Chiwun Yang
Hyper-connections widen the residual stream of a Transformer to parallel streams. Their manifold-constrained version (mHC) mixes the streams at each layer with a doubly stochas…
QATFactory: A Versatile, Deployment-Aligned Framework for Quantization-aware Training and Distillation of LLMs
Weili Xu, Jisen Li, Yuqing Jian +8
Large language model (LLM) inference is increasingly moving toward lower precision to realize the throughput of hardware accelerators, but aggressive post-training quantization (PT…
Nearly Tight Rademacher Bounds for Sparsely Activated Neural Networks
Xiaoyu Li, Zhizhou Sha, Jiaojiao Jiang +2
An input may activate few hidden units even when different inputs collectively use an entire network. We study the statistical complexity of this input-dependent sparsity in the on…
ISO: An RLVR-Native Optimization Stack
Hanqing Zhu, Wenyan Cong, Zhizhou Sha +8
Reinforcement learning with verifiable rewards (RLVR) advances the reasoning capabilities of language models, yet the optimization layer that converts reward feedback into weight-s…
The Path Not Taken: RLVR Provably Learns Off the Principals
Hanqing Zhu, Zhenyu Zhang, Hanxian Huang +11
Reinforcement Learning with Verifiable Rewards (RLVR) reliably improves the reasoning performance of large language models, yet it appears to modify only a small fraction of parame…
Force Matching with Relativistic Constraints: A Physics-Inspired Approach to Stable and Efficient Generative Modeling
Yang Cao, Bo Chen, Xiaoyu Li +5
This paper introduces Force Matching (ForM), a novel framework for generative modeling that represents an initial exploration into leveraging special relativistic mechanics to enha…
Dissecting Submission Limit in Desk-Rejections: A Mathematical Analysis of Fairness in AI Conference Policies
Yuefan Cao, Xiaoyu Li, Yingyu Liang +4
As AI research surges in both impact and volume, conferences have imposed submission limits to maintain paper quality and alleviate organizational pressure. In this work, we examin…
On Computational Limits and Provably Efficient Criteria of Visual Autoregressive Models: A Fine-Grained Complexity Analysis
Yekun Ke, Xiaoyu Li, Yingyu Liang +3
Recently, Visual Autoregressive () Models introduced a groundbreaking advancement in the field of image generation, offering a scalable approach through a coarse-to-f…
On Computational Limits of FlowAR Models: Expressivity and Efficiency
Yang Cao, Chengyue Gong, Yekun Ke +5
The expressive power and computational complexity of deep visual generative models, such as flow-based and autoregressive (AR) models, have gained considerable interest for their w…
HSR-Enhanced Sparse Attention Acceleration
Bo Chen, Yingyu Liang, Zhizhou Sha +2
Large Language Models (LLMs) have demonstrated remarkable capabilities across various applications, but their performance on long-context tasks is often limited by the computationa…
Looped ReLU MLPs May Be All You Need as Practical Programmable Computers
Yingyu Liang, Zhizhou Sha, Zhenmei Shi +2
Previous work has demonstrated that attention mechanisms are Turing complete. More recently, it has been shown that a looped 9-layer Transformer can function as a universal program…
Multi-Layer Transformers Gradient Can be Approximated in Almost Linear Time
Yingyu Liang, Zhizhou Sha, Zhenmei Shi +2
The computational complexity of the self-attention mechanism in popular transformer architectures poses significant challenges for training and inference, and becomes the bottlenec…
Differential Privacy Mechanisms in Neural Tangent Kernel Regression
Jiuxiang Gu, Yingyu Liang, Zhizhou Sha +2
Training data privacy is a fundamental problem in modern Artificial Intelligence (AI) applications, such as face recognition, recommendation systems, language generation, and many…