Shaping Belief States with Generative Environment Models for RL

Abstract

When agents interact with a complex environment, they must form and maintain beliefs about the relevant aspects of that environment. We propose a way to efficiently train expressive generative models in complex environments. We show that a predictive algorithm with an expressive generative model can form stable belief-states in visually rich and dynamic 3D environments. More precisely, we show that the learned representation captures the layout of the environment as well as the position and orientation of the agent. Our experiments show that the model substantially improves data-efficiency on a number of reinforcement learning (RL) tasks compared with strong model-free baseline agents. We find that predicting multiple steps into the future (overshooting), in combination with an expressive generative model, is critical for stable representations to emerge. In practice, using expressive generative models in RL is computationally expensive and we propose a scheme to reduce this computational burden, allowing us to build agents that are competitive with model-free baselines.

pre-print

References in corpus (21)

• Adam: A Method for Stochastic Optimization
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Conditional Image Generation with PixelCNN Decoders
• Density estimation using Real NVP
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Value-Function Approximations for Partially Observable Markov Decision Processes
• The Predictron: End-To-End Learning and Planning
• DARLA: Improving Zero-Shot Transfer in Reinforcement Learning
• Neural SLAM: Learning to Explore with External Memory
• Recurrent Environment Simulators
• Playing FPS Games with Deep Reinforcement Learning
• Deep Successor Reinforcement Learning
• Self-Consistent Trajectory Autoencoder: Hierarchical Reinforcement Learning with Trajectory Embeddings
• Learning and Querying Fast Generative Models for Reinforcement Learning
• Recurrent Reinforcement Learning: A Hybrid Approach
• Fixing a Broken ELBO
• Preventing Posterior Collapse with delta-VAEs
• The Kanerva Machine: A Generative Distributed Memory
• Learning Awareness Models
• Learning Attractor Dynamics for Generative Memory
• Conditional Adversarial Generative Flow for Controllable Image Synthesis

Cited by in corpus (3)

• A Whole Brain Probabilistic Generative Model: Toward Realizing Cognitive Architectures for Developmental Robots
• Spatial and Temporal Hierarchy for Autonomous Navigation using Active Inference in Minigrid Environment
• Self-Consistent Models and Values