A short variational proof of equivalence between policy gradients and soft Q learning

Abstract

Two main families of reinforcement learning algorithms, Q-learning and policy gradients, have recently been proven to be equivalent when using a softmax relaxation on one part, and an entropic regularization on the other. We relate this result to the well-known convex duality of Shannon entropy and the softmax function. Such a result is also known as the Donsker-Varadhan formula. This provides a short proof of the equivalence. We then interpret this duality further, and use ideas of convex analysis to prove a new policy inequality relative to soft Q-learning.

References in corpus (6)

• Asynchronous Methods for Deep Reinforcement Learning
• Equivalence Between Policy Gradients and Soft Q-Learning
• A unified view of entropy-regularized Markov decision processes
• Combining policy gradient and Q-learning
• Taming the Noise in Reinforcement Learning via Soft Updates
• Trust-PCL: An Off-Policy Trust Region Method for Continuous Control

Cited by in corpus (2)

• Variational Bayesian Reinforcement Learning with Regret Bounds
• Sample-Efficient Reinforcement Learning with Maximum Entropy Mellowmax Episodic Control