VisualEchoes: Spatial Image Representation Learning through Echolocation
arXiv:2005.01616
Abstract
Several animal species (e.g., bats, dolphins, and whales) and even visually impaired humans have the remarkable ability to perform echolocation: a biological sonar used to perceive spatial layout and locate objects in the world. We explore the spatial cues contained in echoes and how they can benefit vision tasks that require spatial reasoning. First we capture echo responses in photo-realistic 3D indoor scene environments. Then we propose a novel interaction-based representation learning framework that learns useful visual features via echolocation. We show that the learned image features are useful for multiple downstream vision tasks requiring spatial reasoning---monocular depth estimation, surface normal estimation, and visual navigation---with results comparable or even better than heavily supervised pre-training. Our work opens a new path for representation learning for embodied agents, where supervision comes from interacting with the physical world.
Appears in ECCV 2020
References in corpus (12)
- A Survey on Deep Learning in Medical Image Analysis
- Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
- DeMoN: Depth and Motion Network for Learning Monocular Stereo
- DepthTransfer: Depth Extraction from Video Using Non-parametric Sampling
- On Evaluation of Embodied Navigation Agents
- The Replica Dataset: A Digital Replica of Indoor Spaces
- SfM-Net: Learning of Structure and Motion from Video
- Matterport3D: Learning from RGB-D Data in Indoor Environments
- A 64mW DNN-based Visual Navigation Engine for Autonomous Nano-Drones
- Unsupervised Learning of Geometry with Edge-aware Depth-Normal Consistency
- Self-Supervised Generation of Spatial Audio for 360 Video
- DIODE: A Dense Indoor and Outdoor DEpth Dataset