Learning to Predict Without Looking Ahead: World Models Without Forward Prediction

Abstract

Much of model-based reinforcement learning involves learning a model of an agent's world, and training an agent to leverage this model to perform a task more efficiently. While these models are demonstrably useful for agents, every naturally occurring model of the world of which we are aware---e.g., a brain---arose as the byproduct of competing evolutionary pressures for survival, not minimization of a supervised forward-predictive loss via gradient descent. That useful models can arise out of the messy and slow optimization process of evolution suggests that forward-predictive modeling can arise as a side-effect of optimization under the right circumstances. Crucially, this optimization process need not explicitly be a forward-predictive loss. In this work, we introduce a modification to traditional reinforcement learning which we call observational dropout, whereby we limit the agents ability to observe the real environment at each timestep. In doing so, we can coerce an agent into learning a world model to fill in the observation gaps during reinforcement learning. We show that the emerged world model, while not explicitly trained to predict the future, can help the agent learn key skills required to perform well in its environment. Videos of our results available at https://learningtopredict.github.io/

To appear at the Thirty-third Conference on Neural Information Processing Systems (NeurIPS 2019)

References in corpus (18)

• Trust Region Policy Optimization
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning
• Recurrent World Models Facilitate Policy Evolution
• Model-Based Reinforcement Learning for Atari
• Deep multi-scale video prediction beyond mean square error
• Towards a Definition of Disentangled Representations
• Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control
• Spatio-temporal video autoencoder with differentiable memory
• Emergence of grid-like representations by training recurrent neural networks to perform spatial localization
• Building high-level features using large scale unsupervised learning
• The Predictron: End-To-End Learning and Planning
• Early Visual Concept Learning with Unsupervised Deep Learning
• Learning and Querying Fast Generative Models for Reinforcement Learning
• Woulda, Coulda, Shoulda: Counterfactually-Guided Policy Search
• Lipschitz Continuity in Model-based Reinforcement Learning
• On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models

Cited by in corpus (3)

• EvoJAX: Hardware-Accelerated Neuroevolution
• Only Relevant Information Matters: Filtering Out Noisy Samples to Boost RL
• Predictive Control Using Learned State Space Models via Rolling Horizon Evolution