4 papers · 1 filter
RRPO: Reference-Relative Policy Optimization with Stratified Conditional Rollouts
Yuxin Xiong, Xunyi Jiang, Rohan Surana +8
Group Relative Policy Optimization (GRPO) has shown strong effectiveness in reinforcement learning from verifiable feedback, where sampled rollouts can be compared within a group u…
MASS-DPO: Multi-negative Active Sample Selection for Direct Policy Optimization
Rohan Surana, Xintong Li, Sheldon Yu +7
Multi-negative preference optimization under the Plackett--Luce (PL) model extends Direct Preference Optimization (DPO) by leveraging comparative signals across one preferred and m…
Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learning
Rohan Surana, Gagan Mundada, Xunyi Jiang +19
Reinforcement learning (RL) has become a central post-training tool for improving the reasoning abilities of large language models (LLMs). In these systems, the rollout, the trajec…
WS-GRPO: Weakly-Supervised Group-Relative Policy Optimization for Rollout-Efficient Reasoning
Gagan Mundada, Zihan Huang, Rohan Surana +8
Group Relative Policy Optimization (GRPO) is effective for training language models on complex reasoning. However, since the objective is defined relative to a group of sampled tra…