Learning to Act by Predicting the Future

Abstract

We present an approach to sensorimotor control in immersive environments. Our approach utilizes a high-dimensional sensory stream and a lower-dimensional measurement stream. The cotemporal structure of these streams provides a rich supervisory signal, which enables training a sensorimotor control model by interacting with the environment. The model is trained using supervised learning techniques, but without extraneous supervision. It learns to act based on raw sensory input from a complex three-dimensional environment. The presented formulation enables learning without a fixed goal at training time, and pursuing dynamically changing goals at test time. We conduct extensive experiments in three-dimensional simulations based on the classical first-person game Doom. The results demonstrate that the presented approach outperforms sophisticated prior formulations, particularly on challenging tasks. The results also show that trained models successfully generalize across environments and goals. A model trained using the presented approach won the Full Deathmatch track of the Visual Doom AI Competition, which was held in previously unseen environments.

Published as a conference paper at ICLR 2017

Cited by in corpus (25)

• A Review on Deep Learning Techniques for Video Prediction
• Deep Active Inference
• Spatial Action Maps for Mobile Manipulation
• Mid-Level Visual Representations Improve Generalization and Sample Efficiency for Learning Visuomotor Policies
• Benchmarking Classic and Learned Navigation in Complex 3D Environments
• Composable Planning with Attributes
• A Geometric Perspective on Optimal Representations for Reinforcement Learning
• A neural network trained to predict future video frames mimics critical properties of biological neuronal responses and perception
• Robust Policies via Mid-Level Visual Representations: An Experimental Study in Manipulation and Navigation
• Mathematical Reasoning in Latent Space
• Deep Learning for Embodied Vision Navigation: A Survey
• Learning to Navigate Using Mid-Level Visual Priors
• Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actions
• Topological Planning with Transformers for Vision-and-Language Navigation
• Microscopic Traffic Simulation by Cooperative Multi-agent Deep Reinforcement Learning
• Motion Perception in Reinforcement Learning with Dynamic Objects
• TD or not TD: Analyzing the Role of Temporal Differencing in Deep Reinforcement Learning
• Deep Affordance Foresight: Planning Through What Can Be Done in the Future
• Playing Minecraft with Behavioural Cloning
• Simple Sensor Intentions for Exploration
• Arena: a toolkit for Multi-Agent Reinforcement Learning
• Disentangling Dynamics and Returns: Value Function Decomposition with Future Prediction
• Visual Reaction: Learning to Play Catch with Your Drone
• Building Intelligent Autonomous Navigation Agents
• Neural Online Graph Exploration