Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model

Abstract

Deep reinforcement learning (RL) algorithms can use high-capacity deep networks to learn directly from image observations. However, these high-dimensional observation spaces present a number of challenges in practice, since the policy must now solve two problems: representation learning and task learning. In this work, we tackle these two problems separately, by explicitly learning latent representations that can accelerate reinforcement learning from images. We propose the stochastic latent actor-critic (SLAC) algorithm: a sample-efficient and high-performing RL algorithm for learning policies for complex continuous control tasks directly from high-dimensional image inputs. SLAC provides a novel and principled approach for unifying stochastic sequential models and RL into a single method, by learning a compact latent representation and then performing RL in the model's learned latent space. Our experimental evaluation demonstrates that our method outperforms both model-free and model-based alternatives in terms of final performance and sample efficiency, on a range of difficult image-based control tasks. Our code and videos of our results are available at our website.

arXiv commentProject website: https://alexlee-gk.github.io/slac/

References in corpus (27)

• Auto-Encoding Variational Bayes
• Representation Learning with Contrastive Predictive Coding
• Playing Atari with Deep Reinforcement Learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Soft Actor-Critic Algorithms and Applications
• Learning to Communicate with Deep Multi-Agent Reinforcement Learning
• DeepMind Control Suite
• Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
• Learning Latent Dynamics for Planning from Pixels
• Continuous Deep Q-Learning with Model-based Acceleration
• Distributed Distributional Deterministic Policy Gradients
• Visual Reinforcement Learning with Imagined Goals
• Reinforcement Learning with Unsupervised Auxiliary Tasks
• Maximum a Posteriori Policy Optimisation
• Deep Kalman Filters
• SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning
• Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
• A Disentangled Recognition and Nonlinear Dynamics Model for Unsupervised Learning
• BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling
• Black box variational inference for state space models
• Loss is its own Reward: Self-Supervision for Reinforcement Learning
• On Improving Deep Reinforcement Learning for POMDPs
• Learning and Querying Fast Generative Models for Reinforcement Learning
• Relative Entropy Regularized Policy Iteration
• Variational Temporal Abstraction
• The Value Function Polytope in Reinforcement Learning

Cited by in corpus (29)

• Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
• Data-Efficient Reinforcement Learning with Self-Predictive Representations
• Reinforcement Learning with Prototypical Representations
• A Comprehensive Overview and Survey of Recent Advances in Meta-Learning
• Interpretable End-to-end Urban Autonomous Driving with Latent Deep Reinforcement Learning
• Action and Perception as Divergence Minimization
• Sample-efficient Reinforcement Learning Representation Learning with Curiosity Contrastive Forward Dynamics Model
• Structure in Deep Reinforcement Learning: A Survey and Open Problems
• On the model-based stochastic value gradient for continuous reinforcement learning
• Offline Reinforcement Learning from Images with Latent Space Models
• Learning Latent Representations to Influence Multi-Agent Interaction
• ROBEL: Robotics Benchmarks for Learning with Low-Cost Robots
• PlaNet of the Bayesians: Reconsidering and Improving Deep Planning Network by Incorporating Bayesian Inference
• Dreaming: Model-based Reinforcement Learning by Latent Imagination without Reconstruction
• Which Mutual-Information Representation Learning Objectives are Sufficient for Control?
• Intervention Design for Effective Sim2Real Transfer
• Acceleration of Actor-Critic Deep Reinforcement Learning for Visual Grasping in Clutter by State Representation Learning Based on Disentanglement of a Raw Input Image
• Local Search for Policy Iteration in Continuous Control
• Exploratory State Representation Learning
• Mutual Information Maximization for Robust Plannable Representations
• Weakly-Supervised Reinforcement Learning for Controllable Behavior
• Meta-Model-Based Meta-Policy Optimization
• Probably Approximately Correct Vision-Based Planning using Motion Primitives
• MELD: Meta-Reinforcement Learning from Images via Latent State Models
• End-to-end Autonomous Driving Perception with Sequential Latent Representation Learning
• Measuring Progress in Deep Reinforcement Learning Sample Efficiency
• Hierarchical and Partially Observable Goal-driven Policy Learning with Goals Relational Graph
• Self-Consistent Models and Values
• Predictive Control Using Learned State Space Models via Rolling Horizon Evolution