13 papers
Maglev: Sliding Recurrent Memory
Bo Liu, Qiang Liu
We introduce \ours{}, a recurrent Transformer architecture with fixed-size memory that generalizes sliding-window attention while remaining parallelizable during training. \ours{}…
RoboSSM: Scalable In-context Imitation Learning via State-Space Models
Youngju Yoo, Jiaheng Hu, Yifeng Zhu +4
In-context imitation learning (ICIL) enables robots to learn tasks from prompts consisting of just a handful of demonstrations. By eliminating the need for parameter updates at dep…
Gumbel Distillation for Parallel Text Generation
Chi Zhang, Xixi Hu, Bo Liu +1
The slow, sequential nature of autoregressive (AR) language models has driven the adoption of parallel decoding methods. However, these non-AR models often sacrifice generation qua…
Cautious Optimizers: Improving Training with One Line of Code
Kaizhao Liang, Lizhang Chen, Bo Liu +1
AdamW has been the default optimizer for transformer pretraining. For many years, our community searched for faster and more stable optimizers with only constrained positive outcom…
Improving Rectified Flow with Boundary Conditions
Xixi Hu, Runlong Liao, Keyang Xu +5
Rectified Flow offers a simple and effective approach to high-quality generative modeling by learning a velocity field. However, we identify a limitation in directly modeling the v…
Lion Secretly Solves Constrained Optimization: As Lyapunov Predicts
Lizhang Chen, Bo Liu, Kaizhao Liang +1
Lion (Evolved Sign Momentum), a new optimizer discovered through program search, has shown promising results in training large AI models. It performs comparably or favorably to Ada…