5 papers
Allspark: Weak to Strong Transfer via Alternating Chain of Thought
Kaizhao Liang, Junxiong Wang, Chen Liang +2
Recent progress in frontier models has renewed interest in large-scale reinforcement learning (RL), but the cost of generating large-model rollouts makes even testing RL recipes ex…
Training-Free Looped Transformers
Lizhang Chen, Jonathan Li, Chen Liang +2
We introduce training-free looped transformers, in which a lightweight inference-time wrapper loops a contiguous mid-stack block of layers of a frozen checkpoint without additional…
-Balancing for Mixture-of-Experts Training
Lizhang Chen, Jonathan Li, Qi Wang +5
Mixture-of-Experts (MoE) models rely on balanced expert utilization to fully realize their scalability. However, existing load-balancing methods are largely heuristic and operate o…
Cautious Weight Decay
Lizhang Chen, Jonathan Li, Kaizhao Liang +6
We introduce Cautious Weight Decay (CWD), a one-line, optimizer-agnostic modification that applies weight decay only to parameter coordinates whose signs align with the optimizer u…
Communication Efficient Distributed Training with Distributed Lion
Bo Liu, Lemeng Wu, Lizhang Chen +5
The Lion optimizer has been a promising competitor with the AdamW for training large AI models, with advantages on memory, computation, and sample efficiency. In this paper, we int…