Maximum Entropy Reinforcement Learning with Mixture Policies

Abstract

Mixture models are an expressive hypothesis class that can approximate a rich set of policies. However, using mixture policies in the Maximum Entropy (MaxEnt) framework is not straightforward. The entropy of a mixture model is not equal to the sum of its components, nor does it have a closed-form expression in most cases. Using such policies in MaxEnt algorithms, therefore, requires constructing a tractable approximation of the mixture entropy. In this paper, we derive a simple, low-variance mixture-entropy estimator. We show that it is closely related to the sum of marginal entropies. Equipped with our entropy estimator, we derive an algorithmic variant of Soft Actor-Critic (SAC) to the mixture policy case and evaluate it on a series of continuous control tasks.

References in corpus (12)

• Adam: A Method for Stochastic Optimization
• Continuous control with deep reinforcement learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Addressing Function Approximation Error in Actor-Critic Methods
• Soft Actor-Critic Algorithms and Applications
• Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learning
• SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learning
• Mix&Match - Agent Curricula for Reinforcement Learning
• On the Entropy Computation of Large Complex Gaussian Mixture Distributions
• Distributional Policy Optimization: An Alternative Approach for Continuous Control
• Optimistic Distributionally Robust Policy Optimization