Leveraging exploration in off-policy algorithms via normalizing flows

Abstract

The ability to discover approximately optimal policies in domains with sparse rewards is crucial to applying reinforcement learning (RL) in many real-world scenarios. Approaches such as neural density models and continuous exploration (e.g., Go-Explore) have been proposed to maintain the high exploration rate necessary to find high performing and generalizable policies. Soft actor-critic(SAC) is another method for improving exploration that aims to combine efficient learning via off-policy updates while maximizing the policy entropy. In this work, we extend SAC to a richer class of probability distributions (e.g., multimodal) through normalizing flows (NF) and show that this significantly improves performance by accelerating the discovery of good policies while using much smaller policy representations. Our approach, which we call SAC-NF, is a simple, efficient,easy-to-implement modification and improvement to SAC on continuous control baselines such as MuJoCo and PyBullet Roboschool domains. Finally, SAC-NF does this while being significantly parameter efficient, using as few as 5.5% the parameters for an equivalent SAC model.

Accepted to 3rd Conference on Robot Learning (CoRL 2019); Keywords: Exploration, soft actor-critic, normalizing flow, off-policy; maximum entropy, reinforcement learning; deceptive reward; sparse reward; inverse autoregressive flow

References in corpus (13)

• Auto-Encoding Variational Bayes
• Continuous control with deep reinforcement learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Variational Inference with Normalizing Flows
• Density estimation using Real NVP
• Reinforcement Learning with Deep Energy-Based Policies
• Markov Chain Monte Carlo and Variational Inference: Bridging the Gap
• Unifying Count-Based Exploration and Intrinsic Motivation
• Go-Explore: a New Approach for Hard-Exploration Problems
• Latent Space Policies for Hierarchical Reinforcement Learning
• Boosting Trust Region Policy Optimization by Normalizing Flows Policy
• Bayesian Policy Gradients via Alpha Divergence Dropout Inference
• GAN Q-learning

Cited by in corpus (6)

• Discrete and Continuous Action Representation for Practical RL in Video Games
• Diversity Actor-Critic: Sample-Aware Entropy Regularization for Sample-Efficient Exploration
• A Max-Min Entropy Framework for Reinforcement Learning
• Band-limited Soft Actor Critic Model
• Design of Restricted Normalizing Flow towards Arbitrary Stochastic Policy with Computational Efficiency
• A unified view of likelihood ratio and reparameterization gradients and an optimal importance sampling scheme