A unified view of entropy-regularized Markov decision processes

Abstract

We propose a general framework for entropy-regularized average-reward reinforcement learning in Markov decision processes (MDPs). Our approach is based on extending the linear-programming formulation of policy optimization in MDPs to accommodate convex regularization functions. Our key result is showing that using the conditional entropy of the joint state-action distributions as regularization yields a dual optimization problem closely resembling the Bellman optimality equations. This result enables us to formalize a number of state-of-the-art entropy-regularized reinforcement learning algorithms as approximate variants of Mirror Descent or Dual Averaging, and thus to argue about the convergence properties of these methods. In particular, we show that the exact version of the TRPO algorithm of Schulman et al. (2015) actually converges to the optimal policy, while the entropy-regularized policy gradient methods of Mnih et al. (2016) may fail to converge to a fixed point. Finally, we illustrate empirically the effects of using various regularization techniques on learning performance in a simple reinforcement learning setup.

Cited by in corpus (21)

• AlgaeDICE: Policy Gradient from Arbitrary Experience
• PC-PG: Policy Cover Directed Exploration for Provable Policy Gradient Learning
• Reinforcement Learning via Fenchel-Rockafellar Duality
• Cold-Start Reinforcement Learning with Softmax Policy Gradient
• Revisiting Design Choices in Proximal Policy Optimization
• MADE: Exploration via Maximizing Deviation from Explored Regions
• On Connections between Constrained Optimization and Reinforcement Learning
• A Survey of Exploration Methods in Reinforcement Learning
• A short variational proof of equivalence between policy gradients and soft Q learning
• Sample-Efficient Reinforcement Learning with Maximum Entropy Mellowmax Episodic Control
• Large Scale Markov Decision Processes with Changing Rewards
• Finding the Near Optimal Policy via Adaptive Reduced Regularization in MDPs
• Optimization Issues in KL-Constrained Approximate Policy Iteration
• On the Convergence of Approximate and Regularized Policy Iteration Schemes
• Exploiting Language Instructions for Interpretable and Compositional Reinforcement Learning
• Reparameterized Variational Divergence Minimization for Stable Imitation
• Near Optimal Policy Optimization via REPS
• Robust Generalization despite Distribution Shift via Minimum Discriminating Information
• Lagrangian Duality in Reinforcement Learning
• Transfer Learning by Modeling a Distribution over Policies
• Distributionally-Constrained Policy Optimization via Unbalanced Optimal Transport