Recurrent World Models Facilitate Policy Evolution

Abstract

A generative recurrent neural network is quickly trained in an unsupervised manner to model popular reinforcement learning environments through compressed spatio-temporal representations. The world model's extracted features are fed into compact and simple policies trained by evolution, achieving state of the art results in various environments. We also train our agent entirely inside of an environment generated by its own internal world model, and transfer this policy back into the actual environment. Interactive version of paper at https://worldmodels.github.io

arXiv commentTo appear at NIPS 2018, selected for an oral presentation. arXiv admin note: substantial text overlap with arXiv:1803.10122

Cited by in corpus (162)

• Hamiltonian Neural Networks
• Model-Based Reinforcement Learning for Atari
• A Survey of Deep Reinforcement Learning in Video Games
• Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
• Exploring Model-based Planning with Policy Networks
• Dynamics-Aware Unsupervised Discovery of Skills
• Social Attention for Autonomous Decision-Making in Dense Traffic
• DeepMDP: Learning Continuous Latent Space Models for Representation Learning
• Self-Supervised Policy Adaptation during Deployment
• Improving Generalization in Meta Reinforcement Learning using Learned Objectives
• Reinforcement Learning through Active Inference
• SQIL: Imitation Learning via Reinforcement Learning with Sparse Rewards
• Reinforcement Learning for Generative AI: State of the Art, Opportunities and Open Research Challenges
• The Free Energy Principle for Perception and Action: A Deep Learning Perspective
• Bootstrap Latent-Predictive Representations for Multitask Reinforcement Learning
• Embodied Visual Navigation with Automatic Curriculum Learning in Real Environments
• Learning to Paint With Model-based Deep Reinforcement Learning
• Never Stop Learning: The Effectiveness of Fine-Tuning in Robotic Reinforcement Learning
• Neural Painters: A learned differentiable constraint for generating brushstroke paintings
• ALFWorld: Aligning Text and Embodied Environments for Interactive Learning
• Reinforcement-learning-based control of convectively-unstable flows
• Unsupervised State Representation Learning in Atari
• Long-Horizon Visual Planning with Goal-Conditioned Hierarchical Predictors
• Visual Grounding of Learned Physical Models
• Meta-Learning through Hebbian Plasticity in Random Networks
• Generative Slate Recommendation with Reinforcement Learning
• World Model as a Graph: Learning Latent Landmarks for Planning
• Sample-efficient Reinforcement Learning Representation Learning with Curiosity Contrastive Forward Dynamics Model
• Evolutionary Reinforcement Learning via Cooperative Coevolutionary Negatively Correlated Search
• Variational Recurrent Models for Solving Partially Observable Control Tasks
• R-SQAIR: Relational Sequential Attend, Infer, Repeat
• Continual State Representation Learning for Reinforcement Learning using Generative Replay
• Innate Motivation for Robot Swarms by Minimizing Surprise: From Simple Simulations to Real-World Experiments
• On the model-based stochastic value gradient for continuous reinforcement learning
• A Perspective on Objects and Systematic Generalization in Model-Based RL
• Symmetry-Based Disentangled Representation Learning requires Interaction with Environments
• On the role of planning in model-based deep reinforcement learning
• Parallel Exploration via Negatively Correlated Search
• Mathematical Reasoning in Latent Space
• Learning Belief Representations for Imitation Learning in POMDPs
• Beyond Fine-Tuning: Transferring Behavior in Reinforcement Learning
• Muesli: Combining Improvements in Policy Optimization
• Gradient-Aware Model-based Policy Search
• DreamerPro: Reconstruction-Free Model-Based Reinforcement Learning with Prototypical Representations
• Improving Gradient Estimation in Evolutionary Strategies With Past Descent Directions
• Engineered Self-Organization for Resilient Robot Self-Assembly with Minimal Surprise
• Training Larger Networks for Deep Reinforcement Learning
• The Sensory Neuron as a Transformer: Permutation-Invariant Neural Networks for Reinforcement Learning
• A Consciousness-Inspired Planning Agent for Model-Based Reinforcement Learning
• Rotation, Translation, and Cropping for Zero-Shot Generalization
• Augmented World Models Facilitate Zero-Shot Dynamics Generalization From a Single Offline Environment
• Learning to Predict Without Looking Ahead: World Models Without Forward Prediction
• Improving Generative Imagination in Object-Centric World Models
• Learning Human Objectives by Evaluating Hypothetical Behavior
• Dreaming: Model-based Reinforcement Learning by Latent Imagination without Reconstruction
• Ready Policy One: World Building Through Active Learning
• Evolving Inborn Knowledge For Fast Adaptation in Dynamic POMDP Problems
• Capability Iteration Network for Robot Path Planning
• Direct and indirect reinforcement learning
• Revisiting Hierarchical Approach for Persistent Long-Term Video Prediction
• Forward Prediction for Physical Reasoning
• Contrastive Variational Reinforcement Learning for Complex Observations
• Novelty Search in Representational Space for Sample Efficient Exploration
• Task Selection Policies for Multitask Learning
• Policy-Aware Model Learning for Policy Gradient Methods
• Deep Model-Based Reinforcement Learning for High-Dimensional Problems, a Survey
• Jointly-Learned State-Action Embedding for Efficient Reinforcement Learning
• XLVIN: eXecuted Latent Value Iteration Nets
• Vision-Based Autonomous Car Racing Using Deep Imitative Reinforcement Learning
• From Complexity to Simplicity: Adaptive ES-Active Subspaces for Blackbox Optimization
• Neuromorphic computing for attitude estimation onboard quadrotors
• Surprisal-Triggered Conditional Computation with Neural Networks
• Environment Predictive Coding for Embodied Agents
• A Closed-Loop Perception, Decision-Making and Reasoning Mechanism for Human-Like Navigation
• Explainable Autonomous Robots: A Survey and Perspective
• Generic Itemset Mining Based on Reinforcement Learning
• Hierarchical principles of embodied reinforcement learning: A review
• Improving Sequential Latent Variable Models with Autoregressive Flows
• The LoCA Regret: A Consistent Metric to Evaluate Model-Based Behavior in Reinforcement Learning
• Only Relevant Information Matters: Filtering Out Noisy Samples to Boost RL
• Deep Learning and the Global Workspace Theory
• Self-organization of action hierarchy and compositionality by reinforcement learning with recurrent neural networks
• Skill Transfer in Deep Reinforcement Learning under Morphological Heterogeneity
• Human-in-the-Loop Methods for Data-Driven and Reinforcement Learning Systems
• Continual Learning Using World Models for Pseudo-Rehearsal
• Agent Modelling under Partial Observability for Deep Reinforcement Learning
• Continuous 3D Multi-Channel Sign Language Production via Progressive Transformers and Mixture Density Networks
• Procedural Generalization by Planning with Self-Supervised World Models
• Machine Learning for Mechanical Ventilation Control
• Probably Approximately Correct Vision-Based Planning using Motion Primitives
• Efficient Reinforcement Learning for StarCraft by Abstract Forward Models and Transfer Learning
• Detection of False-Reading Attacks in the AMI Net-Metering System
• Goal-Directed Planning by Reinforcement Learning and Active Inference
• Predict-then-Decide: A Predictive Approach for Wait or Answer Task in Dialogue Systems
• Model-Based Episodic Memory Induces Dynamic Hybrid Controls
• Continuous Control for Searching and Planning with a Learned Model
• Planning from Pixels using Inverse Dynamics Models
• Crowd Density Forecasting by Modeling Patch-based Dynamics
• Forethought and Hindsight in Credit Assignment
• Deep Latent Competition: Learning to Race Using Visual Control Policies in Latent Space
• Regularizing Model-Based Planning with Energy-Based Models
• Implicit Behavioral Cloning
• Reinforcement Learning-based Visual Navigation with Information-Theoretic Regularization
• Domain-Adversarial and Conditional State Space Model for Imitation Learning
• Approximate Model-Based Shielding for Safe Reinforcement Learning
• Assisted Perception: Optimizing Observations to Communicate State
• A Simple Approach to Continual Learning by Transferring Skill Parameters
• Understanding the Origin of Information-Seeking Exploration in Probabilistic Objectives for Control
• Causal World Models by Unsupervised Deconfounding of Physical Dynamics
• Language (Re)modelling: Towards Embodied Language Understanding
• Beyond Tabula-Rasa: a Modular Reinforcement Learning Approach for Physically Embedded 3D Sokoban
• A unified view of likelihood ratio and reparameterization gradients and an optimal importance sampling scheme
• World Programs for Model-Based Learning and Planning in Compositional State and Action Spaces
• Accelerating Reinforcement Learning with a Directional-Gaussian-Smoothing Evolution Strategy
• Detecting and Adapting to Novelty in Games
• Learning Knowledge Graph-based World Models of Textual Environments
• Model-based micro-data reinforcement learning: what are the crucial model properties and which model to choose?
• Iterative Model-Based Reinforcement Learning Using Simulations in the Differentiable Neural Computer
• On the Sensory Commutativity of Action Sequences for Embodied Agents
• Reinforcement Learning with Convolutional Reservoir Computing
• Safe Reinforcement Learning with Mixture Density Network: A Case Study in Autonomous Highway Driving
• Deep Learning: Our Miraculous Year 1990-1991
• A Roadmap for Robust End-to-End Alignment
• Predicting Real-time Scientific Experiments Using Transformer models and Reinforcement Learning
• Reinforcement Learning for Robotics and Control with Active Uncertainty Reduction
• Evaluating the Apperception Engine
• Koopman Q-learning: Offline Reinforcement Learning via Symmetries of Dynamics
• Modeling Worlds in Text
• Detection of Lying Electrical Vehicles in Charging Coordination Application Using Deep Learning
• Self-Organized Construction by Minimal Surprise
• Convolutional Reservoir Computing for World Models
• Meta-Reinforcement Learning for Adaptive Motor Control in Changing Robot Dynamics and Environments
• Imagination-Augmented Deep Learning for Goal Recognition
• Joint Perception and Control as Inference with an Object-based Implementation
• Channel Decomposition into Painting Actions
• Few-Shot Image-to-Semantics Translation for Policy Transfer in Reinforcement Learning
• SketchTransfer: A Challenging New Task for Exploring Detail-Invariance and the Abstractions Learned by Deep Networks
• Techniques Toward Optimizing Viewability in RTB Ad Campaigns Using Reinforcement Learning
• Emergent Communication with World Models
• Privacy-Preserving and Efficient Data Collection Scheme for AMI Networks Using Deep Learning
• Sample Complexity of Estimating the Policy Gradient for Nearly Deterministic Dynamical Systems
• Distilling a Hierarchical Policy for Planning and Control via Representation and Reinforcement Learning
• Planning and Learning Using Adaptive Entropy Tree Search
• Low-Variance Policy Gradient Estimation with World Models
• Multitask Adaptation by Retrospective Exploration with Learned World Models
• Learning Memory-Dependent Continuous Control from Demonstrations
• Convergence Rate of the (1+1)-Evolution Strategy with Success-Based Step-Size Adaptation on Convex Quadratic Functions
• A Self-Supervised Auxiliary Loss for Deep RL in Partially Observable Settings
• Interpretable Option Discovery using Deep Q-Learning and Variational Autoencoders
• Progressive growing of self-organized hierarchical representations for exploration
• Data-Efficient Reinforcement Learning for Malaria Control
• Variational State-Space Models for Localisation and Dense 3D Mapping in 6 DoF
• A Framework of Explanation Generation toward Reliable Autonomous Robots
• Learning of feature points without additional supervision improves reinforcement learning from images
• Stochastic Flows and Geometric Optimization on the Orthogonal Group
• Inducing Functions through Reinforcement Learning without Task Specification
• Neural Algorithmic Reasoners are Implicit Planners
• A Temporal Variational Model for Story Generation
• Dynamic Horizon Value Estimation for Model-based Reinforcement Learning
• Learning to See via Epiretinal Implant Stimulation in silico with Model-Based Deep Reinforcement Learning
• Episodic Memory for Learning Subjective-Timescale Models
• Meta Adaptation using Importance Weighted Demonstrations