3 citations · 4 across the 11 of their papers we have counts for
9 papers · 1 filter
Maglev: Sliding Recurrent Memory
Bo Liu, Qiang Liu
We introduce \ours{}, a recurrent Transformer architecture with fixed-size memory that generalizes sliding-window attention while remaining parallelizable during training. \ours{}…
Improving Rectified Flow with Boundary Conditions
Xixi Hu, Runlong Liao, Keyang Xu +5
Rectified Flow offers a simple and effective approach to high-quality generative modeling by learning a velocity field. However, we identify a limitation in directly modeling the v…
Muon Optimizes Under Spectral Norm Constraints
Lizhang Chen, Jonathan Li, Qiang Liu
The pursuit of faster optimization algorithms remains an active and important research direction in deep learning. Recently, the Muon optimizer [JJB+24] has demonstrated promising…
Improving Adaptive Moment Optimization via Preconditioner Diagonalization
Son Nguyen, Bo Liu, Lizhang Chen +1
Modern adaptive optimization methods, such as Adam and its variants, have emerged as the most widely used tools in deep learning over recent years. These algorithms offer automatic…
SKIM: Any-bit Quantization Pushing The Limits of Post-Training Quantization
Runsheng Bai, Bo Liu, Qiang Liu
Large Language Models (LLMs) exhibit impressive performance across various tasks, but deploying them for inference poses challenges. Their high resource demands often necessitate c…
Cautious Optimizers: Improving Training with One Line of Code
Kaizhao Liang, Lizhang Chen, Bo Liu +1
AdamW has been the default optimizer for transformer pretraining. For many years, our community searched for faster and more stable optimizers with only constrained positive outcom…
Memory-Efficient LLM Training with Online Subspace Descent
Kaizhao Liang, Bo Liu, Lizhang Chen +1
Recently, a wide range of memory-efficient LLM training algorithms have gained substantial popularity. These methods leverage the low-rank structure of gradients to project optimiz…
Memory-Efficient Optimization with Factorized Hamiltonian Descent
Son Nguyen, Lizhang Chen, Bo Liu +1
Modern deep learning heavily depends on adaptive optimizers such as Adam and its variants, which are renowned for their capacity to handle model scaling and streamline hyperparamet…
AdaFlow: Imitation Learning with Variance-Adaptive Flow-Based Policies
Xixi Hu, Bo Liu, Xingchao Liu +1
Diffusion-based imitation learning improves Behavioral Cloning (BC) on multi-modal decision-making, but comes at the cost of significantly slower inference due to the recursion in…