From the 1 of 7 linked papers with an AI index.
7 papers
DeepLoop: Depth Scaling for Looped Transformers
Shuzhen Li, Yifan Zhang, Jiacheng Guo +2
DeepLoop reuses a compact stack of transformer blocks across multiple passes to increase model depth without adding parameters, and introduces new residual scaling rules to keep tr…
Unlocking Feature Learning in Gated Delta Networks at Scale
Yifeng Liu, Quanquan Gu
Training and scaling Large Language Models demand enormous computational resources, motivating both efficient sub-quadratic architectures and principled hyperparameter tuning metho…
Self-Distilled Policy Gradient
Yifeng Liu, Shiyuan Zhang, Yifan Zhang +1
On-policy self-distillation, where a language model conditions on privileged context to supervise its own generations, is a promising source of dense supervision for sparse-reward…
Deep Delta Learning
Yifan Zhang, Yifeng Liu, Mengdi Wang +1
Transformer residual streams evolve through additive updates. Although a sufficiently expressive residual block can represent content replacement, standard architectures do not par…
Transformers Trained via Gradient Descent Can Provably Learn a Class of Teacher Models
Chenyang Zhang, Qingyue Zhao, Quanquan Gu +1
Transformers have achieved great success across a wide range of applications, yet the theoretical foundations underlying their success remain largely unexplored. To demystify the s…
MARS-M: When Variance Reduction Meets Matrices
Yifeng Liu, Angela Yuan, Quanquan Gu
Matrix-based preconditioned optimizers, such as Muon, have recently been shown to be more efficient than scalar-based optimizers for training large-scale neural networks, including…