Non-vacuous Generalization Bounds for Reinforcement Learning with Verifiable Rewards
arXiv:2607.14506
The paper derives the first non‑vacuous PAC‑Bayes generalization bounds for parameter‑efficient reinforcement learning with verifiable rewards applied to billion‑parameter language models, and introduces a Progressive RLVR framework that combines on‑policy distillation, TinyLoRA, and quantization to achieve high compression while retaining most performance.
Abstract
While reinforcement learning with verifiable rewards (RLVR) is widely used to improve the reasoning capabilities of large language models (LLMs), the generalizability of the resulting models remains poorly understood. In this work, we establish the first non-vacuous generalization bounds for parameter-efficient RLVR fine-tuning at the billion-parameter scale. Our approach adapts PAC-Bayes compression bounds to this setting, and addresses the inherent stochasticity of token generation by applying the Gumbel-max reparameterization trick. To operationalize these bounds, we propose the Progressive RLVR framework, which integrates RLVR with on-policy distillation, TinyLoRA, and model quantization. Progressive RLVR empirically retains 84-97% performance of standard LoRA fine-tuning while producing models that are 14,796x more compressible. We show that this framework yields non-vacuous generalization bounds in four domains: mathematical problem-solving, programming, general-knowledge reasoning, and Text-to-SQL. Our bounds exceed the accuracy of the base model by 9-51% and lie within 6-11% of the accuracy of the fine-tuned models.
22 pages, 7 figures