Learning to Set Waypoints for Audio-Visual Navigation
arXiv:2008.09622
Abstract
In audio-visual navigation, an agent intelligently travels through a complex, unmapped 3D environment using both sights and sounds to find a sound source (e.g., a phone ringing in another room). Existing models learn to act at a fixed granularity of agent motion and rely on simple recurrent aggregations of the audio observations. We introduce a reinforcement learning approach to audio-visual navigation with two key novel elements: 1) waypoints that are dynamically set and learned end-to-end within the navigation policy, and 2) an acoustic memory that provides a structured, spatially grounded record of what the agent has heard as it moves. Both new ideas capitalize on the synergy of audio and visual data for revealing the geometry of an unmapped space. We demonstrate our approach on two challenging datasets of real-world 3D scenes, Replica and Matterport3D. Our model improves the state of the art by a substantial margin, and our experiments reveal that learning the links between sights, sounds, and space is essential for audio-visual navigation. Project: http://vision.cs.utexas.edu/projects/audio_visual_waypoints.
arXiv commentAccepted to ICLR 2021
References in corpus (15)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- A Recurrent Latent Variable Model for Sequential Data
- On Evaluation of Embodied Navigation Agents
- The Replica Dataset: A Digital Replica of Indoor Spaces
- Learning to Navigate in Complex Environments
- Data-Efficient Hierarchical Reinforcement Learning
- Learning to Explore using Active Neural SLAM
- Driving Policy Transfer via Modularity and Abstraction
- Spatial Action Maps for Mobile Manipulation
- Learning Multi-Level Hierarchies with Hindsight
- Unifying Map and Landmark Based Representations for Visual Navigation
- Mid-Level Visual Representations Improve Generalization and Sample Efficiency for Learning Visuomotor Policies
- Benchmarking Classic and Learned Navigation in Complex 3D Environments
- Scene Memory Transformer for Embodied Agents in Long-Horizon Tasks
- BatVision: Learning to See 3D Spatial Layout with Two Ears