266 citations · 298 across the 14 of their papers we have counts for
12 papers · 1 filter
Understanding Off- vs On-Policy Distillation: A Tale of Distinct Training Objectives
Qiwei Di, Xuheng Li, Kaixuan Ji +3
On-policy distillation (OPD) learns from teacher feedback on student-generated responses and has shown promise in reducing forgetting relative to supervised fine-tuning (SFT). Howe…
Fast Rates for Offline Contextual Bandits with Forward-KL Regularization under Single-Policy Concentrability
Qingyue Zhao, Kaixuan Ji, Heyang Zhao +1
\emph{Kullback-Leibler} (KL) regularization is ubiquitous in reinforcement learning algorithms in the form of \emph{reverse} or \emph{forward} KL. Recent studies have demonstrated…
On the Optimal Sample Complexity of Offline Multi-Armed Bandits with KL Regularization
Kaixuan Ji, Qiwei Di, Heyang Zhao +2
Kullback-Leibler (KL) regularization is widely used in offline decision-making and offers several benefits, motivating recent work on the sample complexity of offline learning with…
Near-Optimal Regret for KL-Regularized Multi-Armed Bandits
Kaixuan Ji, Qingyue Zhao, Heyang Zhao +2
Recent studies have shown that reinforcement learning with KL-regularized objectives can enjoy faster rates of convergence or logarithmic regret, in contrast to the classical $\sqr…
Best-of-Majority: Minimax-Optimal Strategy for Pass@ Inference Scaling
Qiwei Di, Kaixuan Ji, Xuheng Li +2
LLM inference often generates a batch of candidates for a prompt and selects one via strategies like majority voting or Best-of- N (BoN). For difficult tasks, this single-shot sele…
Towards a Sharp Analysis of Offline Policy Learning for -Divergence-Regularized Contextual Bandits
Qingyue Zhao, Kaixuan Ji, Heyang Zhao +2
Many offline reinforcement learning algorithms are underpinned by -divergence regularization, but their sample complexity *defined with respect to regularized objectives* still…
Enhancing Multi-Step Reasoning Abilities of Language Models through Direct Q-Function Optimization
Kaixuan Ji, Guanlin Liu, Ning Dai +6
Reinforcement Learning (RL) plays a crucial role in aligning large language models (LLMs) with human preferences and improving their ability to perform complex tasks. However, curr…
Self-Play Preference Optimization for Language Model Alignment
Yue Wu, Zhiqing Sun, Huizhuo Yuan +3
Standard reinforcement learning from human feedback (RLHF) approaches relying on parametric models like the Bradley-Terry model fall short in capturing the intransitivity and irrat…
Self-Play Fine-Tuning of Diffusion Models for Text-to-Image Generation
Huizhuo Yuan, Zixiang Chen, Kaixuan Ji +1
Fine-tuning Diffusion Models remains an underexplored frontier in generative artificial intelligence (GenAI), especially when compared with the remarkable progress made in fine-tun…
Reinforcement Learning from Human Feedback with Active Queries
Kaixuan Ji, Jiafan He, Quanquan Gu
Aligning large language models (LLM) with human preference plays a key role in building modern generative models and can be achieved by reinforcement learning from human feedback (…
Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models
Zixiang Chen, Yihe Deng, Huizhuo Yuan +2
Harnessing the power of human-annotated data through Supervised Fine-Tuning (SFT) is pivotal for advancing Large Language Models (LLMs). In this paper, we delve into the prospect o…
Horizon-free Reinforcement Learning in Adversarial Linear Mixture MDPs
Kaixuan Ji, Qingyue Zhao, Jiafan He +2
Recent studies have shown that episodic reinforcement learning (RL) is no harder than bandits when the total reward is bounded by , and proved regret bounds that have a polyloga…