5 papers · 1 filter
Efficient Exploration for Iterative Nash Preference Optimization
Tianlong Nan, Xiaopeng Li, Christian Kroer +1
Preference alignment is central to improving large language models (LLMs), but reward-based formulations can be restrictive when human preferences are non-transitive. Nash learning…
DRIFT: Decoupled Rollouts and Importance-Weighted Fine-Tuning for Efficient Multi-Turn Optimization
Jian Mu, Tianyi Lin, Chengwei Qin +2
Large language models are increasingly deployed in multi-turn interactive settings where users or environments can iteratively provide lightweight feedback. Unfortunately, optimizi…
R1-SyntheticVL: Is Synthetic Data from Generative Models Ready for Multimodal Large Language Model?
Jingyi Zhang, Tianyi Lin, Huanjin Yao +3
In this work, we aim to develop effective data synthesis techniques that autonomously synthesize multimodal training data for enhancing MLLMs in solving complex real-world tasks. T…
Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward
Peter Chen, Xiaopeng Li, Ziniu Li +3
This paper examines the exploration-exploitation trade-off in reinforcement learning with verifiable rewards (RLVR), a framework for improving the reasoning of Large Language Model…
Stepwise Guided Policy Optimization: Coloring your Incorrect Reasoning in GRPO
Peter Chen, Xiaopeng Li, Ziniu Li +2
Reinforcement learning (RL) has proven effective in strengthening the reasoning capabilities of large language models (LLMs). A widely adopted method, Group Relative Policy Optimiz…