8 papers
Demystifying Transition Matching: When and Why It Can Beat Flow Matching
Jaihoon Kim, Rajarshi Saha, Minhyuk Sung +1
Flow Matching (FM) underpins many state-of-the-art generative models, yet recent results indicate that Transition Matching (TM) can achieve higher quality with fewer sampling steps…
Scaling Laws Meet Model Architecture: Toward Inference-Efficient LLMs
Song Bian, Tao Yu, Shivaram Venkataraman +1
Scaling the number of parameters and the size of training data has proven to be an effective strategy for improving large language model (LLM) performance. Yet, as these models gro…
Online Posterior Sampling with a Diffusion Prior
Branislav Kveton, Boris Oreshkin, Youngsuk Park +2
Posterior sampling in contextual bandits with a Gaussian prior can be implemented exactly or approximately using the Laplace approximation. The Gaussian prior is computationally ef…
MuonBP: Faster Muon via Block-Periodic Orthogonalization
Ahmed Khaled, Kaan Ozkara, Tao Yu +2
Gradient orthogonalization is a simple strategy that shows great utility in speeding up gradient descent. The Muon optimizer (Jordan, Jin, et al., 2024) combines gradient orthogona…
Training LLMs with MXFP4
Albert Tseng, Tao Yu, Youngsuk Park
Low precision (LP) datatypes such as MXFP4 can accelerate matrix multiplications (GEMMs) and reduce training costs. However, directly using MXFP4 instead of BF16 during training si…
RoSTE: An Efficient Quantization-Aware Supervised Fine-Tuning Approach for Large Language Models
Quan Wei, Chung-Yiu Yau, Hoi-To Wai +4
Supervised fine-tuning is a standard method for adapting pre-trained large language models (LLMs) to downstream tasks. Quantization has been recently studied as a post-training tec…