9 papers
Co-Evolving LLM Evaluators and Policies via DynamicRubric
Beining Wang, Weihang Su, Hongtao Tian +8
Post-training with evaluator feedback on policy-induced samples serves as a major mechanism for improving large language models. As policies improve, these sampled responses become…
Process Advantage Signal Shaping: Advantage Construction for Process-Supervised Reinforcement Learning in LLM Reasoners
Chao Wang, Hongtao Tian, Tao Yang +3
Dense process supervision supplies local feedback but leaves open how that feedback should become a policy advantage. We introduce PASS (Process Advantage Signal Shaping), an advan…
From <Answer> to <Think>: Multidimensional Supervision of Reasoning Process for LLM Optimization
Beining Wang, Weihang Su, Hongtao Tian +5
Improving the multi-step reasoning ability of Large Language Models (LLMs) is a critical yet challenging task. The dominant paradigm, outcome-supervised reinforcement learning (RLV…
From Faithfulness to Correctness: Generative Reward Models that Think Critically
Qiyao Ma, Yunsheng Shi, Hongtao Tian +3
Through reinforcement learning with verifiable rewards (RLVR), large language models have achieved substantial progress in domains with easily verifiable outcomes, such as mathemat…
Learning More with Less: A Dynamic Dual-Level Down-Sampling Framework for Efficient Policy Optimization
Chao Wang, Tao Yang, Hongtao Tian +5
Critic-free methods like GRPO reduce memory demands by estimating advantages from multiple rollouts but tend to converge slowly, as critical learning signals are diluted by an abun…
WeChat-YATT: A Scalable, Simple, Efficient, and Production Ready Training Library
Junyu Wu, Weiming Chang, Xiaotao Liu +10
Reinforcement Learning from Human Feedback (RLHF) has emerged as a prominent paradigm for training large language models and multimodal systems. Despite the notable advances enable…
CAPO: Towards Enhancing LLM Reasoning through Generative Credit Assignment
Guofu Xie, Yunsheng Shi, Hongtao Tian +2
Reinforcement Learning with Verifiable Rewards (RLVR) has improved the reasoning abilities of Large Language Models (LLMs) by using rule-based binary feedback. However, current RLV…
G-Core: A Simple, Scalable and Balanced RLHF Trainer
Junyu Wu, Weiming Chang, Xiaotao Liu +8
Reinforcement Learning from Human Feedback (RLHF) has become an increasingly popular paradigm for training large language models (LLMs) and diffusion models. While existing RLHF tr…
Discriminative Policy Optimization for Token-Level Reward Models
Hongzhan Chen, Tao Yang, Shiping Gao +4
Process reward models (PRMs) provide more nuanced supervision compared to outcome reward models (ORMs) for optimizing policy models, positioning them as a promising approach to enh…