4 papers · 1 filter
: Provably Optimal Distributional RL for LLM Post-Training
Jin Peng Zhou, Kaiwen Wang, Jonathan Chang +5
Reinforcement learning (RL) post-training is crucial for LLM alignment and reasoning, but existing policy-based methods, such as PPO and DPO, can fall short of fixing shortcuts inh…
Value-Guided Search for Efficient Chain-of-Thought Reasoning
Kaiwen Wang, Jin Peng Zhou, Jonathan Chang +4
In this paper, we propose a simple and efficient method for value model training on long-context reasoning traces. Compared to existing process reward models (PRMs), our method doe…
A Reductions Approach to Risk-Sensitive Reinforcement Learning with Optimized Certainty Equivalents
Kaiwen Wang, Dawen Liang, Nathan Kallus +1
We study risk-sensitive RL where the goal is learn a history-dependent policy that optimizes some risk measure of cumulative rewards. We consider a family of risks called the optim…
More Benefits of Being Distributional: Second-Order Bounds for Reinforcement Learning
Kaiwen Wang, Owen Oertell, Alekh Agarwal +2
In this paper, we prove that Distributional Reinforcement Learning (DistRL), which learns the return distribution, can obtain second-order bounds in both online and offline RL in g…