5 papers · 1 filter
Query-Efficient Rejection Sampling for Parallel Masked Diffusion
Shiyuan Zhang, Xuheng Li, Qiwei Di +1
Masked diffusion models (MDMs) accelerate generation by unmasking multiple tokens in parallel, but sampling these tokens independently ignores their correlations and biases the out…
One Block Ahead: Promoting Sampler State in Diffusion Language Models
Yangyi Shen, Pin-Jui Ku, Nicolai Oswald +6
Block-diffusion language models denoise a whole block of token positions in parallel so each forward pass can emit many tokens. However, blocks are still denoised in strict sequenc…
Muon Learns Facts Better: Understanding the Role of Spectral Orthogonalization
Xuheng Li, Qiwei Di, Yuan Cao +1
The Muon optimizer applies spectral orthogonalization to matrix-valued updates and has shown strong performance in large-scale neural network training, yet the mechanisms of this t…
Understanding Off- vs On-Policy Distillation: A Tale of Distinct Training Objectives
Qiwei Di, Xuheng Li, Kaixuan Ji +3
On-policy distillation (OPD) learns from teacher feedback on student-generated responses and has shown promise in reducing forgetting relative to supervised fine-tuning (SFT). Howe…
Dimension-Independent Convergence of Underdamped Langevin Monte Carlo in KL Divergence
Shiyuan Zhang, Qiwei Di, Xuheng Li +1
Underdamped Langevin dynamics (ULD) is a widely-used sampler for Gibbs distributions , and is often empirically effective in high dimensions. However, existing non…