Scene-Aware Audio Rendering via Deep Acoustic Analysis

Abstract

We present a new method to capture the acoustic characteristics of real-world rooms using commodity devices, and use the captured characteristics to generate similar sounding sources with virtual models. Given the captured audio and an approximate geometric model of a real-world room, we present a novel learning-based method to estimate its acoustic material properties. Our approach is based on deep neural networks that estimate the reverberation time and equalization of the room from recorded audio. These estimates are used to compute material properties related to room reverberation using a novel material optimization objective. We use the estimated acoustic material characteristics for audio rendering using interactive geometric sound propagation and highlight the performance on many real-world scenarios. We also perform a user study to evaluate the perceptual similarity between the recorded sounds and our rendered audio.

Accepted to IEEE VR 2020 Journal Track (TVCG)

References in corpus (9)

• Adam: A Method for Stochastic Optimization
• Deep Convolutional Neural Networks and Data Augmentation for Environmental Sound Classification
• Regression and Classification for Direction-of-Arrival Estimation with Convolutional Recurrent Neural Networks
• Scene-Aware Audio for 360\textdegree{} Videos
• Improving Reverberant Speech Training Using Diffuse Acoustic Simulation
• Low-frequency Compensated Synthetic Impulse Responses for Improved Far-field Speech Recognition
• Learning to Predict Indoor Illumination from a Single Image
• Receiver Placement for Speech Enhancement using Sound Propagation Optimization
• Interactive Sound Rendering on Mobile Devices using Ray-Parameterized Reverberation Filters

Cited by in corpus (10)

• Improving Reverberant Speech Training Using Diffuse Acoustic Simulation
• GWA: A Large High-Quality Acoustic Dataset for Audio Processing
• Low-frequency Compensated Synthetic Impulse Responses for Improved Far-field Speech Recognition
• Sound Synthesis, Propagation, and Rendering: A Survey
• Learning Acoustic Scattering Fields for Dynamic Interactive Sound Propagation
• A State-of-the-Art Review on Acoustic Preservation of Historical Worship Spaces through Auralization
• Blind identification of Ambisonic reduced room impulse response
• Scene-aware Far-field Automatic Speech Recognition
• Echo-Reconstruction: Audio-Augmented 3D Scene Reconstruction
• Binaural Audio Generation via Multi-task Learning