4 papers
Open Rubric System: Scaling Reinforcement Learning with Pairwise Adaptive Rubric
Ruipeng Jia, Yunyi Yang, Yuxin Wu +6
Scalar reward models compress multi-dimensional human preferences into a single opaque score, creating an information bottleneck that often leads to brittleness and reward hacking…
Weights-Rotated Preference Optimization for Large Language Models
Chenxu Yang, Ruipeng Jia, Mingyu Zheng +6
Despite the efficacy of Direct Preference Optimization (DPO) in aligning Large Language Models (LLMs), reward hacking remains a pivotal challenge. This issue emerges when LLMs exce…
Orthogonal Finetuning for Direct Preference Optimization
Chenxu Yang, Ruipeng Jia, Naibin Gu +7
DPO is an effective preference optimization algorithm. However, the DPO-tuned models tend to overfit on the dispreferred samples, manifested as overly long generations lacking dive…
Writing-Zero: Bridge the Gap Between Non-verifiable Tasks and Verifiable Rewards
Ruipeng Jia, Yunyi Yang, Yongbo Gai +5
Reinforcement learning with verifiable rewards (RLVR) has enabled large language models (LLMs) to achieve remarkable breakthroughs in reasoning tasks with objective ground-truth an…