SoundSpaces: Audio-Visual Navigation in 3D Environments

Abstract

Moving around in the world is naturally a multisensory experience, but today's embodied agents are deaf---restricted to solely their visual perception of the environment. We introduce audio-visual navigation for complex, acoustically and visually realistic 3D environments. By both seeing and hearing, the agent must learn to navigate to a sounding object. We propose a multi-modal deep reinforcement learning approach to train navigation policies end-to-end from a stream of egocentric audio-visual observations, allowing the agent to (1) discover elements of the geometry of the physical space indicated by the reverberating audio and (2) detect and follow sound-emitting targets. We further introduce SoundSpaces: a first-of-its-kind dataset of audio renderings based on geometrical acoustic simulations for two sets of publicly available 3D environments (Matterport3D and Replica), and we instrument Habitat to support the new sensor, making it possible to insert arbitrary sound sources in an array of real-world scanned environments. Our results show that audio greatly benefits embodied visual navigation in 3D spaces, and our work lays groundwork for new research in embodied AI with audio-visual perception.

Accepted to ECCV 2020 (Spotlight). Project page: http://vision.cs.utexas.edu/projects/audio_visual_navigation/

References in corpus (16)

• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Joint 2D-3D-Semantic Data for Indoor Scene Understanding
• A Recurrent Latent Variable Model for Sequential Data
• On Evaluation of Embodied Navigation Agents
• The Replica Dataset: A Digital Replica of Indoor Spaces
• Learning to Navigate in Complex Environments
• AI2-THOR: An Interactive 3D Environment for Visual AI
• HoME: a Household Multimodal Environment
• Self-Supervised Generation of Spatial Audio for 360 Video
• Learning Physical Intuition of Block Towers by Example
• See, Hear, and Read: Deep Aligned Representations
• Unifying Map and Landmark Based Representations for Visual Navigation
• PyRobot: An Open-source Robotics Framework for Research and Benchmarking
• MazeBase: A Sandbox for Learning from Games
• Benchmarking Classic and Learned Navigation in Complex 3D Environments
• To Learn or Not to Learn: Analyzing the Role of Learning for Navigation in Virtual Environments