Wasserstein Robust Reinforcement Learning

Abstract

Reinforcement learning algorithms, though successful, tend to over-fit to training environments hampering their application to the real-world. This paper proposes $\text{W}{\text{R}}^2\text{L}$ -- a robust reinforcement learning algorithm with significant robust performance on low and high-dimensional control tasks. Our method formalises robust reinforcement learning as a novel min-max game with a Wasserstein constraint for a correct and convergent solver. Apart from the formulation, we also propose an efficient and scalable solver following a novel zero-order optimisation method that we believe can be useful to numerical optimisation in general. We empirically demonstrate significant gains compared to standard and robust state-of-the-art algorithms on high-dimensional MuJuCo environments.

References in corpus (9)

• Playing Atari with Deep Reinforcement Learning
• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• Robust Adversarial Reinforcement Learning
• Assessing Generalization in Deep Reinforcement Learning
• Deep Lagrangian Networks: Using Physics as Model Prior for Deep Learning
• Action Robust Reinforcement Learning and Applications in Continuous Control
• Learning-based Model Predictive Control for Safe Exploration and Reinforcement Learning
• Investigating Generalisation in Continuous Deep Reinforcement Learning
• Non-Stationary Markov Decision Processes, a Worst-Case Approach using Model-Based Reinforcement Learning, Extended version

Cited by in corpus (8)

• A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
• Game-Theoretic Multiagent Reinforcement Learning
• Are we Forgetting about Compositional Optimisers in Bayesian Optimisation?
• Reliable Off-policy Evaluation for Reinforcement Learning
• Compositional ADAM: An Adaptive Compositional Solver
• Sinkhorn Distributionally Robust Optimization
• Online Robust Reinforcement Learning with Model Uncertainty
• L2E: Learning to Exploit Your Opponent