Variational Autoencoder for Speech Enhancement with a Noise-Aware Encoder

Abstract

Recently, a generative variational autoencoder (VAE) has been proposed for speech enhancement to model speech statistics. However, this approach only uses clean speech in the training phase, making the estimation particularly sensitive to noise presence, especially in low signal-to-noise ratios (SNRs). To increase the robustness of the VAE, we propose to include noise information in the training phase by using a noise-aware encoder trained on noisy-clean speech pairs. We evaluate our approach on real recordings of different noisy environments and acoustic conditions using two different noise datasets. We show that our proposed noise-aware VAE outperforms the standard VAE in terms of overall distortion without increasing the number of model parameters. At the same time, we demonstrate that our model is capable of generalizing to unseen noise conditions better than a supervised feedforward deep neural network (DNN). Furthermore, we demonstrate the robustness of the model performance to a reduction of the noisy-clean speech training data size.

arXiv commentICASSP 2021. (c) 2021 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works

References in corpus (7)

• Adam: A Method for Stochastic Optimization
• Auto-Encoding Variational Bayes
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Semi-Supervised Learning with Deep Generative Models
• Statistical Speech Enhancement Based on Probabilistic Integration of Variational Autoencoder and Non-Negative Matrix Factorization
• Semi-supervised multichannel speech enhancement with variational autoencoders and non-negative matrix factorization
• A variance modeling framework based on variational autoencoders for speech enhancement

Cited by in corpus (9)

• Speech Enhancement and Dereverberation with Diffusion-based Generative Models
• StoRM: A Diffusion-based Stochastic Regeneration Model for Speech Enhancement and Dereverberation
• A Multi-dimensional Deep Structured State Space Approach to Speech Enhancement Using Small-footprint Models
• Disentanglement Learning for Variational Autoencoders Applied to Audio-Visual Speech Enhancement
• Learning and controlling the source-filter representation of speech with a variational autoencoder
• Integrating Statistical Uncertainty into Neural Network-Based Speech Enhancement
• Integrating Uncertainty into Neural Network-based Speech Enhancement
• Speech Enhancement Using Continuous Embeddings of Neural Audio Codec
• The Whole Is Greater than the Sum of Its Parts: Improving Music Source Separation by Bridging Network