11 papers
Query-Efficient Rejection Sampling for Parallel Masked Diffusion
Shiyuan Zhang, Xuheng Li, Qiwei Di +1
Masked diffusion models (MDMs) accelerate generation by unmasking multiple tokens in parallel, but sampling these tokens independently ignores their correlations and biases the out…
One Block Ahead: Promoting Sampler State in Diffusion Language Models
Yangyi Shen, Pin-Jui Ku, Nicolai Oswald +6
Block-diffusion language models denoise a whole block of token positions in parallel so each forward pass can emit many tokens. However, blocks are still denoised in strict sequenc…
Muon Learns Facts Better: Understanding the Role of Spectral Orthogonalization
Xuheng Li, Qiwei Di, Yuan Cao +1
The Muon optimizer applies spectral orthogonalization to matrix-valued updates and has shown strong performance in large-scale neural network training, yet the mechanisms of this t…
Understanding Off- vs On-Policy Distillation: A Tale of Distinct Training Objectives
Qiwei Di, Xuheng Li, Kaixuan Ji +3
On-policy distillation (OPD) learns from teacher feedback on student-generated responses and has shown promise in reducing forgetting relative to supervised fine-tuning (SFT). Howe…
Saturation-Insensitive Dueling Bandits with General Function Approximation
Chenggong Zhang, Xuheng Li, Qiwei Di +2
We study contextual dueling bandits with general function approximation under the Bradley-Terry-Luce (BTL) preference model. A key challenge in this setting is the saturation of th…
Online KL-Regularized Reinforcement Learning with Function Approximation under Misspecification
Haoyang Hong, Zichen Wang, Quanquan Gu +1
We study KL-regularized contextual bandits and episodic reinforcement learning (RL) under general function approximation with model misspecification. Existing guarantees rely on re…
Fast Rates for Offline Contextual Bandits with Forward-KL Regularization under Single-Policy Concentrability
Qingyue Zhao, Kaixuan Ji, Heyang Zhao +1
\emph{Kullback-Leibler} (KL) regularization is ubiquitous in reinforcement learning algorithms in the form of \emph{reverse} or \emph{forward} KL. Recent studies have demonstrated…
Dimension-Independent Convergence of Underdamped Langevin Monte Carlo in KL Divergence
Shiyuan Zhang, Qiwei Di, Xuheng Li +1
Underdamped Langevin dynamics (ULD) is a widely-used sampler for Gibbs distributions , and is often empirically effective in high dimensions. However, existing non…
Variance-Aware Feel-Good Thompson Sampling for Contextual Bandits
Xuheng Li, Quanquan Gu
Variance-dependent regret bounds have received increasing attention in recent studies on contextual bandits. However, most of these studies are focused on upper confidence bound (U…
Best-of-Majority: Minimax-Optimal Strategy for Pass@ Inference Scaling
Qiwei Di, Kaixuan Ji, Xuheng Li +2
LLM inference often generates a batch of candidates for a prompt and selects one via strategies like majority voting or Best-of- N (BoN). For difficult tasks, this single-shot sele…
Understanding SGD with Exponential Moving Average: A Case Study in Linear Regression
Xuheng Li, Quanquan Gu
Exponential moving average (EMA) has recently gained significant popularity in training modern deep learning models, especially diffusion-based generative models. However, there ha…