Rethinking GSPO: The Perplexity-Entropy Equivalence

Abstract

We provide a new perspective on GSPO's length-normalized importance ratios by establishing their connection to information-theoretic quantities. We show that GSPO's sequence-level weight $s(\hat{I}¸)=(\ddot{I}{€}_{\hat{I}}¸/\ddot{I}{€}_{\hat{I}{¸}_{\text{old}}}{)}^{1/|y|}$ can be equivalently expressed as the inverse perplexity ratio ${\text{PPL}}_{\hat{I}{¸}_{\text{old}}}/{\text{PPL}}_{\hat{I}}¸$ and as the exponential cross-entropy change $\exp (\hat{I}”H)$. While the perplexity-entropy relationship follows from standard definitions, this observation provides a useful lens for understanding GSPO: the algorithm weights policy gradient updates by perplexity ratios, offering an information-theoretic interpretation of the importance weights. This perspective helps explain GSPO's empirical properties, including log-domain variance reduction through geometric averaging and stability in training mixture-of-experts models. We validate the mathematical equivalences and variance predictions through controlled experiments on mathematical reasoning tasks.

10 pages, 2 figures