Showing cs.LGShow all
3 papers · 1 filter
cs.LG2026
Latent Recurrent Transformer: Architecture Exploration, Training Strategies, and Scaling Behavior
Zeyi Huang, Xuehai He, LiLiang Ren +8
We study Latent Recurrent Transformer (LRT), a lightweight augmentation of autoregressive transformers that reuses a high-level source-layer hidden state from the previous token as…
cs.LG2026
Rethinking Language Model Scaling under Transferable Hypersphere Optimization
Liliang Ren, Yang Liu, Yelong Shen +1
Scaling laws for large language models depend critically on the optimizer and parameterization. Existing hyperparameter transfer laws are mainly developed for first-order optimizer…
cs.LG2026
GeoNorm: Unify Pre-Norm and Post-Norm with Geodesic Optimization
Chuanyang Zheng, Jiankai Sun, Yihang Gao +11
The placement of normalization layers, specifically Pre-Norm and Post-Norm, remains an open question in Transformer architecture design. In this work, we rethink these approaches t…