Robust Policy Gradient against Strong Data Corruption

Abstract

We study the problem of robust reinforcement learning under adversarial corruption on both rewards and transitions. Our attack model assumes an \textit{adaptive} adversary who can arbitrarily corrupt the reward and transition at every step within an episode, for at most $\epsilon$-fraction of the learning episodes. Our attack model is strictly stronger than those considered in prior works. Our first result shows that no algorithm can find a better than $O(\epsilon )$-optimal policy under our attack model. Next, we show that surprisingly the natural policy gradient (NPG) method retains a natural robustness property if the reward corruption is bounded, and can find an $O(\sqrt{\epsilon })$-optimal policy. Consequently, we develop a Filtered Policy Gradient (FPG) algorithm that can tolerate even unbounded reward corruption and can find an $O({\epsilon }^{1/4})$-optimal policy. We emphasize that FPG is the first that can achieve a meaningful learning guarantee when a constant fraction of episodes are corrupted. Complimentary to the theoretical results, we show that a neural implementation of FPG achieves strong robust learning performance on the MuJoCo continuous control benchmarks.

References in corpus (34)

• Trust Region Policy Optimization
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• Dota 2 with Large Scale Deep Reinforcement Learning
• Solving Rubik's Cube with a Robot Hand
• Robust Adversarial Reinforcement Learning
• Provably Efficient Reinforcement Learning with Linear Function Approximation
• Contextual Decision Processes with Low Bellman Rank are PAC-Learnable
• Sever: A Robust Meta-Algorithm for Stochastic Optimization
• Unifying PAC and Regret: Uniform PAC Bounds for Episodic Reinforcement Learning
• On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shift
• Recent Advances in Algorithmic High-Dimensional Robust Statistics
• Sample Complexity of Episodic Fixed-Horizon Reinforcement Learning
• Model-Based Reinforcement Learning with Value-Targeted Regression
• Being Robust (in High Dimensions) Can Be Practical
• Model-based RL in Contextual Decision Processes: PAC bounds and Exponential Improvements over Model-free Approaches
• A Short Note on Concentration Inequalities for Random Vectors with SubGaussian Norm
• On Lower Bounds for Regret in Reinforcement Learning
• Model-based Reinforcement Learning and the Eluder Dimension
• Policy Poisoning in Batch Reinforcement Learning and Control
• Reinforcement Learning in Feature Space: Matrix Bandit, Kernels, and Regret Bound
• Provably Efficient $Q$-learning with Function Approximation via Distribution Shift Error Checking Oracle
• FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPs
• Adaptive Reward-Poisoning Attacks against Reinforcement Learning
• PC-PG: Policy Cover Directed Exploration for Provable Policy Gradient Learning
• Provably Efficient Reinforcement Learning for Discounted MDPs with Feature Mapping
• Variational Regret Bounds for Reinforcement Learning
• Information Theoretic Regret Bounds for Online Nonlinear Control
• Bias no more: high-probability data-dependent regret bounds for adversarial bandits and MDPs
• Better Algorithms for Stochastic Bandits with Adversarial Corruptions
• A Kernel-Based Approach to Non-Stationary Reinforcement Learning in Metric Spaces
• Online learning in MDPs with linear function approximation and bandit feedback
• Derivative-Free Policy Optimization for Linear Risk-Sensitive and Robust Control Design: Implicit Regularization and Sample Complexity
• Simultaneously Learning Stochastic and Adversarial Episodic MDPs with Known Transition
• Learning and Planning for Time-Varying MDPs Using Maximum Likelihood Estimation

Cited by in corpus (4)

• Derivative-Free Policy Optimization for Linear Risk-Sensitive and Robust Control Design: Implicit Regularization and Sample Complexity
• Corruption-Robust Offline Reinforcement Learning
• Softmax Policy Gradient Methods Can Take Exponential Time to Converge
• A Model Selection Approach for Corruption Robust Reinforcement Learning