5 papers · 1 filter
Revisiting Zeroth-Order Hessian Approximation: A Single-Step Policy Optimization Lens
Junbin Qiu, Zhaowei Hong, Renzhe Xu +1
Accurate Zeroth-Order (ZO) Hessian estimation is a cornerstone of derivative-free methods, essential for tasks such as bilevel optimization, Bayesian inference, and uncertainty qua…
Reference-Sampled Boltzmann Projection for KL-Regularized RLVR: Target-Matched Weighted SFT, Finite One-Shot Gaps, and Policy Mirror Descent
Yao Shu, Chenxing Wei, Hongbin Lin +2
Online reinforcement learning with verifiable rewards (RLVR) turns checkable outcomes into a scalable training signal, but it keeps rollout generation, verifier scoring, and refere…
Model-based Offline RL via Robust Value-Aware Model Learning with Implicitly Differentiable Adaptive Weighting
Zhongjian Qiao, Jiafei Lyu, Boxiang Lyu +3
Model-based offline reinforcement learning (RL) aims to enhance offline RL with a dynamics model that facilitates policy exploration. However, \textit{model exploitation} could occ…
Efficient Token-Level Policy Optimization for Masked Diffusion Models via Velocity Alignment
Chenxing Wei, Jiazhen Kang, Hong Wang +8
Reinforcement Learning with Verifiable Rewards (RLVR) successfully aligns autoregressive models in rigorous domains like mathematical reasoning and code generation. However, applyi…
ReDit: Reward Dithering for Improved LLM Policy Optimization
Chenxing Wei, Jiarui Yu, Ying Tiffany He +3
DeepSeek-R1 has successfully enhanced Large Language Model (LLM) reasoning capabilities through its rule-based reward system. While it's a ''perfect'' reward system that effectivel…