A unified view of entropy-regularized Markov decision processes

Abstract

We propose a general framework for entropy-regularized average-reward reinforcement learning in Markov decision processes (MDPs). Our approach is based on extending the linear-programming formulation of policy optimization in MDPs to accommodate convex regularization functions. Our key result is showing that using the conditional entropy of the joint state-action distributions as regularization yields a dual optimization problem closely resembling the Bellman optimality equations. This result enables us to formalize a number of state-of-the-art entropy-regularized reinforcement learning algorithms as approximate variants of Mirror Descent or Dual Averaging, and thus to argue about the convergence properties of these methods. In particular, we show that the exact version of the TRPO algorithm of Schulman et al. (2015) actually converges to the optimal policy, while the entropy-regularized policy gradient methods of Mnih et al. (2016) may fail to converge to a fixed point. Finally, we illustrate empirically the effects of using various regularization techniques on learning performance in a simple reinforcement learning setup.

References in corpus (1)

• Combining policy gradient and Q-learning

Cited by in corpus (62)

• Game-Theoretic Multiagent Reinforcement Learning
• On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shift
• AlgaeDICE: Policy Gradient from Arbitrary Experience
• A Lyapunov-based Approach to Safe Reinforcement Learning
• A reinforcement learning approach to rare trajectory sampling
• Understanding the impact of entropy on policy optimization
• Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularization
• Reinforcement learning of rare diffusive dynamics
• Lipschitz Continuity in Model-based Reinforcement Learning
• Maximum Entropy RL (Provably) Solves Some Robust RL Problems
• PC-PG: Policy Cover Directed Exploration for Provable Policy Gradient Learning
• Reinforcement Learning via Fenchel-Rockafellar Duality
• Model-Based Multi-Agent RL in Zero-Sum Markov Games with Near-Optimal Sample Complexity
• Mirror Descent Policy Optimization
• Cold-Start Reinforcement Learning with Softmax Policy Gradient
• Regularization Matters in Policy Optimization
• Average-reward model-free reinforcement learning: a systematic review and literature mapping
• A Unified Bellman Optimality Principle Combining Reward Maximization and Empowerment
• Revisiting Design Choices in Proximal Policy Optimization
• CAQL: Continuous Action Q-Learning
• Neural Temporal-Difference and Q-Learning Provably Converge to Global Optima
• Entropy Regularization for Mean Field Games with Learning
• Optimistic Policy Optimization with Bandit Feedback
• MADE: Exploration via Maximizing Deviation from Explored Regions
• A Tutorial on Sparse Gaussian Processes and Variational Inference
• Fast Policy Extragradient Methods for Competitive Games with Entropy Regularization
• Greedification Operators for Policy Optimization: Investigating Forward and Reverse KL Divergences
• Policy Mirror Descent for Regularized Reinforcement Learning: A Generalized Framework with Linear Convergence
• On Entropy Regularized Path Integral Control for Trajectory Optimization
• Adaptive Approximate Policy Iteration
• Equivalence Between Wasserstein and Value-Aware Loss for Model-based Reinforcement Learning
• Path Consistency Learning in Tsallis Entropy Regularized MDPs
• On Connections between Constrained Optimization and Reinforcement Learning
• Stable Policy Optimization via Off-Policy Divergence Regularization
• VFunc: a Deep Generative Model for Functions
• Cautiously Optimistic Policy Optimization and Exploration with Linear Function Approximation
• Deep RL With Information Constrained Policies: Generalization in Continuous Control
• A Constrained Randomized Shortest-Paths Framework for Optimal Exploration
• A Survey of Exploration Methods in Reinforcement Learning
• Learning Infinite-horizon Average-reward MDPs with Linear Function Approximation
• Faster saddle-point optimization for solving large-scale Markov decision processes
• Mutual-Information Regularization in Markov Decision Processes and Actor-Critic Learning
• A short variational proof of equivalence between policy gradients and soft Q learning
• Divergence-Regularized Multi-Agent Actor-Critic
• Approximate Newton policy gradient algorithms
• Finding the Near Optimal Policy via Adaptive Reduced Regularization in MDPs
• Optimization Issues in KL-Constrained Approximate Policy Iteration
• Large Scale Markov Decision Processes with Changing Rewards
• Sample-Efficient Reinforcement Learning with Maximum Entropy Mellowmax Episodic Control
• Reparameterized Variational Divergence Minimization for Stable Imitation
• On the Convergence of Approximate and Regularized Policy Iteration Schemes
• Exploiting Language Instructions for Interpretable and Compositional Reinforcement Learning
• Logistic Q-Learning
• Bregman Gradient Policy Optimization
• Robust Generalization despite Distribution Shift via Minimum Discriminating Information
• Near Optimal Policy Optimization via REPS
• Distributionally-Constrained Policy Optimization via Unbalanced Optimal Transport
• Lagrangian Duality in Reinforcement Learning
• Transfer Learning by Modeling a Distribution over Policies
• Parameterized MDPs and Reinforcement Learning Problems -- A Maximum Entropy Principle Based Framework
• Low Variance Trust Region Optimization with Independent Actors and Sequential Updates in Cooperative Multi-agent Reinforcement Learning
• The Confluence of Networks, Games and Learning