Reward-Preserving Attacks For Robust Reinforcement Learning

Abstract

Adversarial training in reinforcement learning (RL) is challenging because perturbations cascade through trajectories and compound over time, making fixed-strength attacks either overly destructive or too conservative. We propose reward-preserving attacks, which adapt adversarial strength so that an $\hat{I}\pm$ fraction of the nominal-to-worst-case return gap remains achievable at each state. In deep RL, perturbation magnitudes $\hat{I}\cdot$ are selected dynamically, using a learned critic $Q((s,a),\hat{I}\cdot )$ that estimates the expected return of $\hat{I}\pm$-reward-preserving rollouts. For intermediate values of $\hat{I}\pm$, this adaptive training yields policies that are robust across a wide range of perturbation magnitudes while preserving nominal performance, outperforming fixed-radius and uniformly sampled-radius adversarial training.

27 pages, 28 figures, 4 algorithms, 3 tables, preprint