Improved Lite Audio-Visual Speech Enhancement

Abstract

Numerous studies have investigated the effectiveness of audio-visual multimodal learning for speech enhancement (AVSE) tasks, seeking a solution that uses visual data as auxiliary and complementary input to reduce the noise of noisy speech signals. Recently, we proposed a lite audio-visual speech enhancement (LAVSE) algorithm for a car-driving scenario. Compared to conventional AVSE systems, LAVSE requires less online computation and to some extent solves the user privacy problem on facial data. In this study, we extend LAVSE to improve its ability to address three practical issues often encountered in implementing AVSE systems, namely, the additional cost of processing visual data, audio-visual asynchronization, and low-quality visual data. The proposed system is termed improved LAVSE (iLAVSE), which uses a convolutional recurrent neural network architecture as the core AVSE model. We evaluate iLAVSE on the Taiwan Mandarin speech with video dataset. Experimental results confirm that compared to conventional AVSE systems, iLAVSE can effectively overcome the aforementioned three practical issues and can improve enhancement performance. The results also confirm that iLAVSE is suitable for real-world scenarios, where high-quality audio-visual sensors may not always be available.

References in corpus (11)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Supervised and Unsupervised Speech Enhancement Using Nonnegative Matrix Factorization
• Training and Inference with Integers in Deep Neural Networks
• Conditional Generative Adversarial Networks for Speech Enhancement and Noise-Robust Speaker Verification
• Improving the Performance of K-Means for Color Quantization
• Multi-modal Multi-channel Target Speech Separation
• Audio-visual Speech Enhancement Using Conditional Variational Auto-Encoders
• Characterizing Speech Adversarial Examples Using Self-Attention U-Net Enhancement
• Time-Domain Multi-modal Bone/air Conducted Speech Enhancement
• Deep-Learning-Based Audio-Visual Speech Enhancement in Presence of Lombard Effect
• Joint NN-Supported Multichannel Reduction of Acoustic Echo, Reverberation and Noise

Cited by in corpus (3)

• Decentralizing Feature Extraction with Quantum Convolutional Neural Network for Automatic Speech Recognition
• CITISEN: A Deep Learning-Based Speech Signal-Processing Mobile Application
• A Preliminary Study of a Two-Stage Paradigm for Preserving Speaker Identity in Dysarthric Voice Conversion