4 papers
Gibbs-Gap Policy Optimization for Off-Policy RL with Verifiable Rewards
Xuanchang Zhang, Tong Zhang
Reinforcement learning with verifiable rewards often trains language models on responses generated by a different, possibly stale behavior policy. Correcting this mismatch with lea…
Adaptive Layerwise Perturbation: Unifying Off-Policy Corrections for LLM RL
Chenlu Ye, Xuanchang Zhang, Yifan Hao +6
Off-policy problems such as policy staleness and training--inference mismatch have become a major bottleneck for training stability and further exploration in LLM RL. The distribut…
From Lists to Emojis: How Format Bias Affects Model Alignment
Xuanchang Zhang, Wei Xiong, Lichang Chen +3
In this paper, we study format biases in reinforcement learning from human feedback (RLHF). We observe that many widely-used preference models, including human evaluators, GPT-4, a…
GLaPE: Gold Label-agnostic Prompt Evaluation and Optimization for Large Language Model
Xuanchang Zhang, Zhuosheng Zhang, Hai Zhao
Despite the rapid progress of large language models (LLMs), their task performance remains sensitive to prompt design. Recent studies have explored leveraging the LLM itself as an…