14 papers
Experience Augmented Policy Optimization for LLM Reasoning
Jinda Lu, Kexin Huang, Junkang Wu +7
Reinforcement Learning with Verifiable Rewards (RLVR) is a powerful paradigm for improving the reasoning capabilities of large language models (LLMs). However, existing RLVR method…
ARMOR: Stabilizing On-Policy LLM RL with Off-Policy Anchor Samples
Kexin Huang, Junkang Wu, Jinda Lu +7
Reinforcement learning (RL) has significantly enhanced the reasoning capabilities of large language models (LLMs), yet the training process remains notoriously fragile. In this wor…
Clipping Bottleneck: Stabilizing RLVR via Stochastic Recovery of Near-Boundary Signals
Shuo Yang, Jinda Lu, Chiyu Ma +8
Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a central paradigm for scaling LLM reasoning, yet its optimization often suffers from training instability and…
One-Way Policy Optimization for Self-Evolving LLMs
Shuo Yang, Jinda Lu, Kexin Huang +6
Reinforcement Learning with Verifiable Rewards (RLVR) has become a promising paradigm for scaling reasoning capabilities of Large Language Models (LLMs). However, the sparsity of b…
FIPO: Eliciting Deep Reasoning with Future-KL Influenced Policy Optimization
Chiyu Ma, Shuo Yang, Kexin Huang +7
We present Future-KL Influenced Policy Optimization (FIPO), a reinforcement learning algorithm designed to overcome reasoning bottlenecks in large language models. While GRPO style…
Beyond Where to Look: Trajectory-Guided Reinforcement Learning for Multimodal RLVR
Jinda Lu, Junkang Wu, Jinghan Li +6
Recent advances in Reinforcement Learning with Verifiable Rewards (RLVR) for multimodal large language models (MLLMs) have mainly focused on improving final answer correctness and…
Bridging Perception and Reasoning: Token Reweighting for RLVR in Multimodal LLMs
Jinda Lu, Junkang Wu, Jinghan Li +6
Extending Reinforcement Learning with Verifiable Rewards (RLVR) to multimodal large language models (MLLMs) faces a fundamental challenge: their responses inherently interleave per…
Sparse but Critical: A Token-Level Analysis of Distributional Shifts in RLVR Fine-Tuning of LLMs
Haoming Meng, Kexin Huang, Shaohang Wei +6
Reinforcement learning with verifiable rewards (RLVR) has significantly improved reasoning in large language models (LLMs), yet the token-level mechanisms underlying these improvem…
On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitation
Kexin Huang, Haoming Meng, Junkang Wu +10
Reinforcement learning with verifiable rewards (RLVR) has substantially improved the reasoning capabilities of large language models. While existing analyses identify that RLVR-ind…
MOSS-TTSD: Text to Spoken Dialogue Generation
Yuqian Zhang, Donghua Yu, Zhengyuan Lin +15
Spoken dialogue generation is crucial for applications like podcasts, dynamic commentary, and entertainment content, but poses significant challenges compared to single-utterance t…
Quantile Advantage Estimation: Stabilizing RLVR for LLM Reasoning
Junkang Wu, Kexin Huang, Jiancan Wu +3
Reinforcement Learning with Verifiable Rewards (RLVR) strengthens LLM reasoning, but training often oscillates between {entropy collapse} and {entropy explosion}. We trace both haz…
RePO: Understanding Preference Learning Through ReLU-Based Optimization
Junkang Wu, Kexin Huang, Xue Wang +5
Aligning large language models (LLMs) with human preferences is critical for real-world deployment, yet existing methods like RLHF face computational and stability challenges. Whil…
Larger or Smaller Reward Margins to Select Preferences for Alignment?
Kexin Huang, Junkang Wu, Ziqian Chen +6
Preference learning is critical for aligning large language models (LLMs) with human values, with the quality of preference datasets playing a crucial role in this process. While e…
LaMP-Val: Large Language Models Empower Personalized Valuation in Auction
Jie Sun, Tianyu Zhang, Houcheng Jiang +11
Auctions are a vital economic mechanism used to determine the market value of goods or services through competitive bidding within a specific framework. However, much of the curren…