9 papers
Signed Rectified Flow: Negativity-Controlled Generation
Runlong Liao, Baiyu Su, Lizhang Chen +1
We introduce Signed Rectified Flow (Signed RF), a generalization of Rectified Flow that targets the signed measure , where , is the di…
Momentum Guidance: Plug-and-Play Guidance for Flow Models
Runlong Liao, Jian Yu, Baiyu Su +3
Flow-based generative methods offer a simple and effective framework for high-fidelity generation, yet pretrained flow models are rarely used in their vanilla conditional form: in…
Training-Free Looped Transformers
Lizhang Chen, Jonathan Li, Chen Liang +2
We introduce training-free looped transformers, in which a lightweight inference-time wrapper loops a contiguous mid-stack block of layers of a frozen checkpoint without additional…
-Balancing for Mixture-of-Experts Training
Lizhang Chen, Jonathan Li, Qi Wang +5
Mixture-of-Experts (MoE) models rely on balanced expert utilization to fully realize their scalability. However, existing load-balancing methods are largely heuristic and operate o…
Cautious Weight Decay
Lizhang Chen, Jonathan Li, Kaizhao Liang +6
We introduce Cautious Weight Decay (CWD), a one-line, optimizer-agnostic modification that applies weight decay only to parameter coordinates whose signs align with the optimizer u…
Cautious Optimizers: Improving Training with One Line of Code
Kaizhao Liang, Lizhang Chen, Bo Liu +1
AdamW has been the default optimizer for transformer pretraining. For many years, our community searched for faster and more stable optimizers with only constrained positive outcom…