6 papers
Online KL-Regularized Reinforcement Learning with Function Approximation under Misspecification
Haoyang Hong, Zichen Wang, Quanquan Gu +1
We study KL-regularized contextual bandits and episodic reinforcement learning (RL) under general function approximation with model misspecification. Existing guarantees rely on re…
Fast Rates for Offline Contextual Bandits with Forward-KL Regularization under Single-Policy Concentrability
Qingyue Zhao, Kaixuan Ji, Heyang Zhao +1
\emph{Kullback-Leibler} (KL) regularization is ubiquitous in reinforcement learning algorithms in the form of \emph{reverse} or \emph{forward} KL. Recent studies have demonstrated…
Dimension-Independent Convergence of Underdamped Langevin Monte Carlo in KL Divergence
Shiyuan Zhang, Qiwei Di, Xuheng Li +1
Underdamped Langevin dynamics (ULD) is a widely-used sampler for Gibbs distributions , and is often empirically effective in high dimensions. However, existing no…
Variance-Aware Feel-Good Thompson Sampling for Contextual Bandits
Xuheng Li, Quanquan Gu
Variance-dependent regret bounds have received increasing attention in recent studies on contextual bandits. However, most of these studies are focused on upper confidence bound (U…
Best-of-Majority: Minimax-Optimal Strategy for Pass@ Inference Scaling
Qiwei Di, Kaixuan Ji, Xuheng Li +2
LLM inference often generates a batch of candidates for a prompt and selects one via strategies like majority voting or Best-of- N (BoN). For difficult tasks, this single-shot sele…
Understanding SGD with Exponential Moving Average: A Case Study in Linear Regression
Xuheng Li, Quanquan Gu
Exponential moving average (EMA) has recently gained significant popularity in training modern deep learning models, especially diffusion-based generative models. However, there ha…