3 citations · 4 across the 15 of their papers we have counts for
16 papers
GradLev: Token-Parallel Test-Time Training Via Costate Prediction
Bo Liu, Qiang Liu
Test-time training (TTT) allows a model to improve its predictions at inference time by updating weights after every observed token. However, sequential gra- dient writes make para…
Maglev: Sliding Recurrent Memory
Bo Liu, Qiang Liu
We introduce \ours{}, a recurrent Transformer architecture with fixed-size memory that generalizes sliding-window attention while remaining parallelizable during training. \ours{}…
Gumbel Distillation for Parallel Text Generation
Chi Zhang, Xixi Hu, Bo Liu +1
The slow, sequential nature of autoregressive (AR) language models has driven the adoption of parallel decoding methods. However, these non-AR models often sacrifice generation qua…
RoboSSM: Scalable In-context Imitation Learning via State-Space Models
Youngju Yoo, Jiaheng Hu, Yifeng Zhu +4
In-context imitation learning (ICIL) enables robots to learn tasks from prompts consisting of just a handful of demonstrations. By eliminating the need for parameter updates at dep…
Improving Rectified Flow with Boundary Conditions
Xixi Hu, Runlong Liao, Keyang Xu +5
Rectified Flow offers a simple and effective approach to high-quality generative modeling by learning a velocity field. However, we identify a limitation in directly modeling the v…
Improving Adaptive Moment Optimization via Preconditioner Diagonalization
Son Nguyen, Bo Liu, Lizhang Chen +1
Modern adaptive optimization methods, such as Adam and its variants, have emerged as the most widely used tools in deep learning over recent years. These algorithms offer automatic…
SKIM: Any-bit Quantization Pushing The Limits of Post-Training Quantization
Runsheng Bai, Bo Liu, Qiang Liu
Large Language Models (LLMs) exhibit impressive performance across various tasks, but deploying them for inference poses challenges. Their high resource demands often necessitate c…
AMO Sampler: Enhancing Text Rendering with Overshooting
Xixi Hu, Keyang Xu, Bo Liu +2
Achieving precise alignment between textual instructions and generated images in text-to-image generation is a significant challenge, particularly in rendering written text within…
Cautious Optimizers: Improving Training with One Line of Code
Kaizhao Liang, Lizhang Chen, Bo Liu +1
AdamW has been the default optimizer for transformer pretraining. For many years, our community searched for faster and more stable optimizers with only constrained positive outcom…
Fine-Grained Gradient Restriction: A Simple Approach for Mitigating Catastrophic Forgetting
Bo Liu, Mao Ye, Peter Stone +1
A fundamental challenge in continual learning is to balance the trade-off between learning new tasks and remembering the previously acquired knowledge. Gradient Episodic Memory (GE…
Memory-Efficient LLM Training with Online Subspace Descent
Kaizhao Liang, Bo Liu, Lizhang Chen +1
Recently, a wide range of memory-efficient LLM training algorithms have gained substantial popularity. These methods leverage the low-rank structure of gradients to project optimiz…
Longhorn: State Space Models are Amortized Online Learners
Bo Liu, Rui Wang, Lemeng Wu +3
Modern large language models are built on sequence modeling via next-token prediction. While the Transformer remains the dominant architecture for sequence modeling, its quadratic…
Memory-Efficient Optimization with Factorized Hamiltonian Descent
Son Nguyen, Lizhang Chen, Bo Liu +1
Modern deep learning heavily depends on adaptive optimizers such as Adam and its variants, which are renowned for their capacity to handle model scaling and streamline hyperparamet…
Communication Efficient Distributed Training with Distributed Lion
Bo Liu, Lemeng Wu, Lizhang Chen +5
The Lion optimizer has been a promising competitor with the AdamW for training large AI models, with advantages on memory, computation, and sample efficiency. In this paper, we int…
AdaFlow: Imitation Learning with Variance-Adaptive Flow-Based Policies
Xixi Hu, Bo Liu, Xingchao Liu +1
Diffusion-based imitation learning improves Behavioral Cloning (BC) on multi-modal decision-making, but comes at the cost of significantly slower inference due to the recursion in…
Lion Secretly Solves Constrained Optimization: As Lyapunov Predicts
Lizhang Chen, Bo Liu, Kaizhao Liang +1
Lion (Evolved Sign Momentum), a new optimizer discovered through program search, has shown promising results in training large AI models. It performs comparably or favorably to Ada…