Learning and Querying Fast Generative Models for Reinforcement Learning

Abstract

A key challenge in model-based reinforcement learning (RL) is to synthesize computationally efficient and accurate environment models. We show that carefully designed generative models that learn and operate on compact state representations, so-called state-space models, substantially reduce the computational costs for predicting outcomes of sequences of actions. Extensive experiments establish that state-space models accurately capture the dynamics of Atari games from the Arcade Learning Environment from raw pixels. The computational speed-up of state-space models while maintaining high accuracy makes their application in RL feasible: We demonstrate that agents which query these models for decision making outperform strong model-free baselines on the game MSPACMAN, demonstrating the potential of using learned environment models for planning.

References in corpus (13)

• Adam: A Method for Stochastic Optimization
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Asynchronous Methods for Deep Reinforcement Learning
• Action-Conditional Video Prediction using Deep Networks in Atari Games
• Sequential Neural Models with Stochastic Layers
• The Predictron: End-To-End Learning and Planning
• Deep Kalman Filters
• Learning to Poke by Poking: Experiential Learning of Intuitive Physics
• Recurrent Environment Simulators
• Black box variational inference for state space models
• Value Prediction Network
• Neural Adaptive Sequential Monte Carlo

Cited by in corpus (40)

• Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model
• Dream to Control: Learning Behaviors by Latent Imagination
• DeepMDP: Learning Continuous Latent Space Models for Representation Learning
• Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model
• Planning to Explore via Self-Supervised World Models
• Long-Horizon Visual Planning with Goal-Conditioned Hierarchical Predictors
• Mastering Atari with Discrete World Models
• Learning Dynamics Model in Reinforcement Learning by Incorporating the Long Term Future
• Shaping Belief States with Generative Environment Models for RL
• Combining Q-Learning and Search with Amortized Value Estimates
• Consistent Generative Query Networks
• Temporal Difference Variational Auto-Encoder
• Clockwork Variational Autoencoders
• Learning to Predict Without Looking Ahead: World Models Without Forward Prediction
• Neural Density Estimation and Likelihood-free Inference
• Deep Model-Based Reinforcement Learning for High-Dimensional Problems, a Survey
• Neural Recursive Belief States in Multi-Agent Reinforcement Learning
• Forward Prediction for Physical Reasoning
• Model-Based Regularization for Deep Reinforcement Learning with Transcoder Networks
• Adaptive Skip Intervals: Temporal Abstraction for Recurrent Dynamical Models
• Deep Affordance Foresight: Planning Through What Can Be Done in the Future
• Variational Inference for Data-Efficient Model Learning in POMDPs
• Data-efficient visuomotor policy training using reinforcement learning and generative models
• Joint Adversarial Training: Incorporating both Spatial and Pixel Attacks
• Document-editing Assistants and Model-based Reinforcement Learning as a Path to Conversational AI
• Learning Powerful Policies by Using Consistent Dynamics Model
• Learning State Representations via Retracing in Reinforcement Learning
• Relational State-Space Model for Stochastic Multi-Object Systems
• Towards Target-Driven Visual Navigation in Indoor Scenes via Generative Imitation Learning
• Mind the Gap when Conditioning Amortised Inference in Sequential Latent-Variable Models
• NeoNav: Improving the Generalization of Visual Navigation via Generating Next Expected Observations
• Evaluating the Apperception Engine
• An investigation of model-free planning
• Learning What To Do by Simulating the Past
• Planning in Dynamic Environments with Conditional Autoregressive Models
• Visual Perspective Taking for Opponent Behavior Modeling
• Variational Predictive Routing with Nested Subjective Timescales
• Visual Reaction: Learning to Play Catch with Your Drone
• Predictive Control Using Learned State Space Models via Rolling Horizon Evolution
• Contrastive Active Inference