Zero Collapse: A Failure Mode of Policy Gradient Methods in Discontinuous Reward Environments

Abstract

Policy-gradient learning can deteriorate even after substantial initial improvement. We study this behavior in a repeated first-price auction setting with a sharp threshold between winning and losing bids. We identify a failure mode, zero collapse, in which a policy shifts into persistently near-zero reward as informative outcomes become rarely sampled. Experiments with REINFORCE and a TD state-value actor-critic illustrate this failure and evaluate practical mitigations. Adaptive update-magnitude control and smooth-output parameterization support sustained REINFORCE learning, while a learned baseline substantially stabilizes the fixed-rate comparison. Some actor-critic runs still collapse under these interventions. A diagnostic near one observed collapse reveals severe continuation-value errors that reverse the relative desirability of actions across the winning threshold. These results illustrate how thresholded outcomes, policy updates, and inaccurate value estimates can contribute to zero collapse, and demonstrate practical mitigations in the studied setting.

27 pages, 8 figures; includes Appendix