Distributional Robustness and Regularization in Reinforcement Learning

Abstract

Distributionally Robust Optimization (DRO) has enabled to prove the equivalence between robustness and regularization in classification and regression, thus providing an analytical reason why regularization generalizes well in statistical learning. Although DRO's extension to sequential decision-making overcomes $\text{external uncertainty}$ through the robust Markov Decision Process (MDP) setting, the resulting formulation is hard to solve, especially on large domains. On the other hand, existing regularization methods in reinforcement learning only address $\text{internal uncertainty}$ due to stochasticity. Our study aims to facilitate robust reinforcement learning by establishing a dual relation between robust MDPs and regularization. We introduce Wasserstein distributionally robust MDPs and prove that they hold out-of-sample performance guarantees. Then, we introduce a new regularizer for empirical value functions and show that it lower bounds the Wasserstein distributionally robust value function. We extend the result to linear value function approximation for large state spaces. Our approach provides an alternative formulation of robustness with guaranteed finite-sample performance. Moreover, it suggests using regularization as a practical tool for dealing with $\text{external uncertainty}$ in reinforcement learning methods.

arXiv commentAccepted at the "Theoretical Foundations of Reinforcement Learning" Workshop - ICML 2020

References in corpus (9)

• Robustness and Regularization of Support Vector Machines
• Robust Wasserstein Profile Inference and Applications to Machine Learning
• A Study on Overfitting in Deep Reinforcement Learning
• Distributionally Robust Logistic Regression
• Wasserstein Distributionally Robust Stochastic Control: A Data-Driven Approach
• Policy Gradient for Coherent Risk Measures
• Shallow Updates for Deep Reinforcement Learning
• Soft-Robust Actor-Critic Policy-Gradient
• Beyond Confidence Regions: Tight Bayesian Ambiguity Sets for Robust MDPs

Cited by in corpus (8)

• Distributionally Robust Optimization: A Review
• Algorithmic Bias and Data Bias: Understanding the Relation between Distributionally Robust Optimization and Data Curation
• Sinkhorn Distributionally Robust Optimization
• Twice regularized MDPs and the equivalence between robustness and regularization
• Towards Theoretical Understandings of Robust Markov Decision Processes: Sample Complexity and Asymptotics
• Robust Batch Policy Learning in Markov Decision Processes
• On the Optimality of Batch Policy Optimization Algorithms
• Learning Provably Robust Motion Planners Using Funnel Libraries