7 papers · 1 filter
DeepLoop: Depth Scaling for Looped Transformers
Shuzhen Li, Yifan Zhang, Jiacheng Guo +2
Looped Transformers scale sequential computation by applying a compact stack of physical blocks for multiple rounds, increasing unrolled depth without increasing stored parameters.…
Unlocking Feature Learning in Gated Delta Networks at Scale
Yifeng Liu, Quanquan Gu
Training and scaling Large Language Models demand enormous computational resources, motivating both efficient sub-quadratic architectures and principled hyperparameter tuning metho…
Self-Distilled Policy Gradient
Yifeng Liu, Shiyuan Zhang, Yifan Zhang +1
On-policy self-distillation, where a language model conditions on privileged context to supervise its own generations, is a promising source of dense supervision for sparse-reward…
Transformers Trained via Gradient Descent Can Provably Learn a Class of Teacher Models
Chenyang Zhang, Qingyue Zhao, Quanquan Gu +1
Transformers have achieved great success across a wide range of applications, yet the theoretical foundations underlying their success remain largely unexplored. To demystify the s…
Deep Delta Learning
Yifan Zhang, Yifeng Liu, Mengdi Wang +1
Transformer residual streams evolve through additive updates. Although a sufficiently expressive residual block can represent content replacement, standard architectures do not par…
Robust Layerwise Scaling Rules by Proper Weight Decay Tuning
Zhiyuan Fan, Yifeng Liu, Qingyue Zhao +2
Empirical scaling laws prescribe how to allocate parameters, data, and compute, while maximal-update parameterization (P) enables learning-rate transfer across widths by equaliz…