Robust Reinforcement Learning using Adversarial Populations

Abstract

Reinforcement Learning (RL) is an effective tool for controller design but can struggle with issues of robustness, failing catastrophically when the underlying system dynamics are perturbed. The Robust RL formulation tackles this by adding worst-case adversarial noise to the dynamics and constructing the noise distribution as the solution to a zero-sum minimax game. However, existing work on learning solutions to the Robust RL formulation has primarily focused on training a single RL agent against a single adversary. In this work, we demonstrate that using a single adversary does not consistently yield robustness to dynamics variations under standard parametrizations of the adversary; the resulting policy is highly exploitable by new adversaries. We propose a population-based augmentation to the Robust RL formulation in which we randomly initialize a population of adversaries and sample from the population uniformly during training. We empirically validate across robotics benchmarks that the use of an adversarial population results in a more robust policy that also improves out-of-distribution generalization. Finally, we demonstrate that this approach provides comparable robustness and generalization as domain randomization on these benchmarks while avoiding a ubiquitous domain randomization failure mode.

References in corpus (9)

• Robust Adversarial Reinforcement Learning
• EPOpt: Learning Robust Neural Network Policies Using Model Ensembles
• Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approach
• Adversarial Policies: Attacking Deep Reinforcement Learning
• Action Robust Reinforcement Learning and Applications in Continuous Control
• Reinforcement Learning for Pivoting Task
• Active Domain Randomization
• Robust Reinforcement Learning via Adversarial training with Langevin Dynamics
• Real World Games Look Like Spinning Tops

Cited by in corpus (9)

• Robust Multi-agent Communication via Multi-view Message Certification
• Robust Control for Dynamical Systems With Non-Gaussian Noise via Formal Abstractions
• Robust Reinforcement Learning using Least Squares Policy Iteration with Provable Performance Guarantees
• Online Robust Reinforcement Learning with Model Uncertainty
• Model-Based Domain Generalization
• Combining Pessimism with Optimism for Robust and Efficient Model-Based Deep Reinforcement Learning
• Policy Smoothing for Provably Robust Reinforcement Learning
• Generating Adversarial Disturbances for Controller Verification
• Collaboration Promotes Group Resilience in Multi-Agent RL