Beyond Entropy: Correctness-Aware Advantage Shaping via Contrastive Policy Optimization
arXiv:2607.14614
The paper proposes Contrastive Policy Optimization, which leverages token‑level contrastive disagreement between reference‑guided and standard generation distributions to provide a correctness‑aware advantage signal for reinforcement learning, outperforming traditional entropy‑based methods.
Abstract
Reinforcement learning with verifiable rewards (RLVR) commonly uses entropy for advantage shaping. However, entropy cannot distinguish useful uncertainty from detrimental confusion, limiting its effectiveness as a correctness signal. We propose Contrastive Policy Optimization (CPO), which uses token-level contrastive disagreement between reference-guided and vanilla generation distributions for correctness-aware advantage shaping. Both theoretical and empirical results show that this disagreement reliably indicates token-level correctness. We further show that On-policy Distillation is a special case of CPO, where the posterior distribution is instantiated by an external teacher model. CPO also resolves the zero-advantage problem. Experiments on in-domain and out-of-domain benchmarks demonstrate that CPO substantially outperforms entropy-based RLVR methods while maintaining strong generalization. Further analysis shows that correct and incorrect responses naturally support exploration and exploitation respectively, and balancing both leads to the best performance.