Environment Predictive Coding for Embodied Agents
arXiv:2102.02337
Abstract
We introduce environment predictive coding, a self-supervised approach to learn environment-level representations for embodied agents. In contrast to prior work on self-supervised learning for images, we aim to jointly encode a series of images gathered by an agent as it moves about in 3D environments. We learn these representations via a zone prediction task, where we intelligently mask out portions of an agent's trajectory and predict them from the unmasked portions, conditioned on the agent's camera poses. By learning such representations on a collection of videos, we demonstrate successful transfer to multiple downstream navigation-oriented tasks. Our experiments on the photorealistic 3D environments of Gibson and Matterport3D show that our method outperforms the state-of-the-art on challenging tasks with only a limited budget of experience.
9 pages, 6 figures, appendix
References in corpus (13)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- A Simple Framework for Contrastive Learning of Visual Representations
- End to End Learning for Self-Driving Cars
- Recurrent World Models Facilitate Policy Evolution
- The Replica Dataset: A Digital Replica of Indoor Spaces
- Learning to Navigate in Complex Environments
- AI2-THOR: An Interactive 3D Environment for Visual AI
- Learning Video Representations using Contrastive Bidirectional Transformer
- Memory-augmented Dense Predictive Coding for Video Representation Learning
- Emergence of Exploratory Look-Around Behaviors through Active Observation Completion
- Situational Fusion of Visual Representation for Visual Navigation
- Geometry-Aware Recurrent Neural Networks for Active Visual Recognition
- Consistent Generative Query Networks