Maximum Entropy RL (Provably) Solves Some Robust RL Problems

Abstract

Many potential applications of reinforcement learning (RL) require guarantees that the agent will perform well in the face of disturbances to the dynamics or reward function. In this paper, we prove theoretically that maximum entropy (MaxEnt) RL maximizes a lower bound on a robust RL objective, and thus can be used to learn policies that are robust to some disturbances in the dynamics and the reward function. While this capability of MaxEnt RL has been observed empirically in prior work, to the best of our knowledge our work provides the first rigorous proof and theoretical characterization of the MaxEnt RL robust set. While a number of prior robust RL algorithms have been designed to handle similar disturbances to the reward function or dynamics, these methods typically require additional moving parts and hyperparameters on top of a base RL algorithm. In contrast, our results suggest that MaxEnt RL by itself is robust to certain disturbances, without requiring any additional modifications. While this does not imply that MaxEnt RL is the best available robust RL method, MaxEnt RL is a simple robust RL method with appealing formal guarantees.

arXiv commentPublished at ICLR 2022. Blog post and videos: https://bair.berkeley.edu/blog/2021/03/10/maxent-robust-rl/. arXiv admin note: text overlap with arXiv:1910.01913

References in corpus (28)

• Continuous control with deep reinforcement learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Game theory, maximum entropy, minimum discrepancy and robust Bayesian decision theory
• Adversarial Examples Are Not Bugs, They Are Features
• Robust Adversarial Reinforcement Learning
• Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
• Learning Continuous Control Policies by Stochastic Value Gradients
• Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
• Continuous Deep Q-Learning with Model-based Acceleration
• A Study on Overfitting in Deep Reinforcement Learning
• Maximum a Posteriori Policy Optimisation
• Bridging the Gap Between Value and Policy Based Reinforcement Learning
• Lyapunov-based Safe Policy Optimization for Continuous Control
• EPOpt: Learning Robust Neural Network Policies Using Model Ensembles
• Illuminating Generalization in Deep Reinforcement Learning through Procedural Level Generation
• Constrained Policy Optimization
• A unified view of entropy-regularized Markov decision processes
• A Theory of Regularized Markov Decision Processes
• The Many Faces of Exponential Weights in Online Learning
• Taming the Noise in Reinforcement Learning via Soft Updates
• Inverse Reward Design
• Understanding the impact of entropy on policy optimization
• Worst Cases Policy Gradients
• Learning to be Safe: Deep RL with a Safety Critic
• Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learning
• Recovery RL: Safe Reinforcement Learning with Learned Recovery Zones
• Tsallis Reinforcement Learning: A Unified Framework for Maximum Entropy Reinforcement Learning
• Understanding Learned Reward Functions

Cited by in corpus (11)

• A review on deep reinforcement learning for fluid mechanics: an update
• A review on Deep Reinforcement Learning for Fluid Mechanics
• Active Learning in Robotics: A Review of Control Principles
• Efficient (Soft) Q-Learning for Text Generation with Limited Good Data
• Exploratory State Representation Learning
• Twice regularized MDPs and the equivalence between robustness and regularization
• Reinforcement Learning by Guided Safe Exploration
• A Max-Min Entropy Framework for Reinforcement Learning
• CROP: Certifying Robust Policies for Reinforcement Learning through Functional Smoothing
• Optimizing Quantum Variational Circuits with Deep Reinforcement Learning
• Count-Based Temperature Scheduling for Maximum Entropy Reinforcement Learning