If MaxEnt RL is the Answer, What is the Question?

Abstract

Experimentally, it has been observed that humans and animals often make decisions that do not maximize their expected utility, but rather choose outcomes randomly, with probability proportional to expected utility. Probability matching, as this strategy is called, is equivalent to maximum entropy reinforcement learning (MaxEnt RL). However, MaxEnt RL does not optimize expected utility. In this paper, we formally show that MaxEnt RL does optimally solve certain classes of control problems with variability in the reward function. In particular, we show (1) that MaxEnt RL can be used to solve a certain class of POMDPs, and (2) that MaxEnt RL is equivalent to a two-player game where an adversary chooses the reward function. These results suggest a deeper connection between MaxEnt RL, robust control, and POMDPs, and provide insight for the types of problems for which we might expect MaxEnt RL to produce effective solutions. Specifically, our results suggest that domains with uncertainty in the task goal may be especially well-suited for MaxEnt RL methods.

References in corpus (16)

• Continuous control with deep reinforcement learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Theoretical Models of Learning to Learn
• Game theory, maximum entropy, minimum discrepancy and robust Bayesian decision theory
• Adversarial Examples Are Not Bugs, They Are Features
• Robust Adversarial Reinforcement Learning
• Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
• Path integrals and symmetry breaking for optimal control theory
• Maximum a Posteriori Policy Optimisation
• Bridging the Gap Between Value and Policy Based Reinforcement Learning
• Efficient Exploration via State Marginal Matching
• Taming the Noise in Reinforcement Learning via Soft Updates
• Skew-Fit: State-Covering Self-Supervised Reinforcement Learning
• Understanding the impact of entropy on policy optimization
• Tsallis Reinforcement Learning: A Unified Framework for Maximum Entropy Reinforcement Learning
• Variational Bayesian Reinforcement Learning with Regret Bounds

Cited by in corpus (15)

• Learning Implicit Credit Assignment for Cooperative Multi-Agent Reinforcement Learning
• Deep Reinforcement Learning for Closed-Loop Blood Glucose Control
• Interpretable End-to-end Urban Autonomous Driving with Latent Deep Reinforcement Learning
• Soft Hindsight Experience Replay
• Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observability
• Ready Policy One: World Building Through Active Learning
• Inverse Decision Modeling: Learning Interpretable Representations of Behavior
• Towards Automatic Actor-Critic Solutions to Continuous Control
• Control as Hybrid Inference
• Imitative Planning using Conditional Normalizing Flow
• Divergence-Regularized Multi-Agent Actor-Critic
• Planning and Learning Using Adaptive Entropy Tree Search
• Ranking Policy Gradient
• Model-Free Risk-Sensitive Reinforcement Learning
• Diversity in Action: General-Sum Multi-Agent Continuous Inverse Optimal Control