Clip Your Sequences Fairly: Enforcing Length Fairness for Sequence-Level RL

Hanyi Mao, Quanjia Xiao, Lei Pang +1

We propose FSPO (Fair Sequence Policy Optimization), a sequence-level reinforcement learning method for LLMs that enforces length-fair clipping on the importance-sampling (IS) weig…