Improving Exploration in Soft-Actor-Critic with Normalizing Flows Policies

Abstract

Deep Reinforcement Learning (DRL) algorithms for continuous action spaces are known to be brittle toward hyperparameters as well as \cut{being}sample inefficient. Soft Actor Critic (SAC) proposes an off-policy deep actor critic algorithm within the maximum entropy RL framework which offers greater stability and empirical gains. The choice of policy distribution, a factored Gaussian, is motivated by \cut{chosen due}its easy re-parametrization rather than its modeling power. We introduce Normalizing Flow policies within the SAC framework that learn more expressive classes of policies than simple factored Gaussians. \cut{We also present a series of stabilization tricks that enable effective training of these policies in the RL setting.}We show empirically on continuous grid world tasks that our approach increases stability and is better suited to difficult exploration in sparse reward settings.

INNF workshop, International Conference on Machine Learning 2019, Long Beach CA, USA

References in corpus (5)

• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Density estimation using Real NVP
• Gradient Estimation Using Stochastic Computation Graphs
• FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models
• Boosting Trust Region Policy Optimization by Normalizing Flows Policy

Cited by in corpus (10)

• Normalizing Flows: An Introduction and Review of Current Methods
• Normalizing Flows for Probabilistic Modeling and Inference
• Coupling-based Invertible Neural Networks Are Universal Diffeomorphism Approximators
• Discrete and Continuous Action Representation for Practical RL in Video Games
• Augmented Normalizing Flows: Bridging the Gap Between Generative Flows and Latent Variable Models
• Greedification Operators for Policy Optimization: Investigating Forward and Reverse KL Divergences
• Implicit Distributional Reinforcement Learning
• Robust normalizing flows using Bernstein-type polynomials
• Improved Soft Actor-Critic: Mixing Prioritized Off-Policy Samples with On-Policy Experience
• Implicitly Regularized RL with Implicit Q-Values