Recurrent Environment Simulators

Abstract

Models that can simulate how environments change in response to actions can be used by agents to plan and act efficiently. We improve on previous environment simulators from high-dimensional pixel observations by introducing recurrent neural networks that are able to make temporally and spatially coherent predictions for hundreds of time-steps into the future. We present an in-depth analysis of the factors affecting performance, providing the most extensive attempt to advance the understanding of the properties of these models. We address the issue of computationally inefficiency with a model that does not need to generate a high-dimensional image at each time-step. We show that our approach can be used to improve exploration and is adaptable to many diverse environments, namely 10 Atari games, a 3D car racing environment, and complex 3D mazes.

References in corpus (1)

• Recurrent Neural Network Regularization

Cited by in corpus (45)

• A Brief Survey of Deep Reinforcement Learning
• Model-Based Reinforcement Learning for Atari
• A Review on Deep Learning Techniques for Video Prediction
• Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control
• Stochastic Adversarial Video Prediction
• The Predictron: End-To-End Learning and Planning
• A Disentangled Recognition and Nonlinear Dynamics Model for Unsupervised Learning
• Self-Supervised Visual Planning with Temporal Skip Connections
• Learning and Querying Fast Generative Models for Reinforcement Learning
• Psychlab: A Psychology Laboratory for Deep Reinforcement Learning Agents
• Imitating Latent Policies from Observation
• Value Prediction Network
• Simulating Action Dynamics with Neural Process Networks
• Model-based Reinforcement Learning for Semi-Markov Decision Processes with Neural ODEs
• Recall Traces: Backtracking Models for Efficient Reinforcement Learning
• Mastering Atari with Discrete World Models
• Human-Level Reinforcement Learning through Theory-Based Modeling, Exploration, and Planning
• Learning Dynamics Model in Reinforcement Learning by Incorporating the Long Term Future
• Shaping Belief States with Generative Environment Models for RL
• Causally Correct Partial Models for Reinforcement Learning
• Mathematical Reasoning in Latent Space
• On the use of recurrent neural networks for predictions of turbulent flows
• Temporal Difference Variational Auto-Encoder
• Promptable Game Models: Text-Guided Game Simulation via Masked Diffusion Models
• Novelty Search in Representational Space for Sample Efficient Exploration
• Deep Model-Based Reinforcement Learning for High-Dimensional Problems, a Survey
• Model-Based Regularization for Deep Reinforcement Learning with Transcoder Networks
• Adaptive Skip Intervals: Temporal Abstraction for Recurrent Dynamical Models
• Model-based Behavioral Cloning with Future Image Similarity Learning
• Video Extrapolation with an Invertible Linear Embedding
• Learning Abstract Models for Strategic Exploration and Fast Reward Transfer
• Learning to Slide Unknown Objects with Differentiable Physics Simulations
• Modular Action Concept Grounding in Semantic Video Prediction
• A survey of benchmarking frameworks for reinforcement learning
• Evaluating the Apperception Engine
• Object-Oriented Dynamics Learning through Multi-Level Abstraction
• Learning Transition Models with Time-delayed Causal Relations
• Action-conditional Sequence Modeling for Recommendation
• Relevance-Guided Modeling of Object Dynamics for Reinforcement Learning
• Variational State-Space Models for Localisation and Dense 3D Mapping in 6 DoF
• Visual Perspective Taking for Opponent Behavior Modeling
• Layered Controllable Video Generation
• Learning the Reward Function for a Misspecified Model
• Generative Adversarial Imagination for Sample Efficient Deep Reinforcement Learning
• High Performance Across Two Atari Paddle Games Using the Same Perceptual Control Architecture Without Training