Time-Domain Multi-modal Bone/air Conducted Speech Enhancement

Abstract

Previous studies have proven that integrating video signals, as a complementary modality, can facilitate improved performance for speech enhancement (SE). However, video clips usually contain large amounts of data and pose a high cost in terms of computational resources and thus may complicate the SE system. As an alternative source, a bone-conducted speech signal has a moderate data size while manifesting speech-phoneme structures, and thus complements its air-conducted counterpart. In this study, we propose a novel multi-modal SE structure in the time domain that leverages bone- and air-conducted signals. In addition, we examine two ensemble-learning-based strategies, early fusion (EF) and late fusion (LF), to integrate the two types of speech signals, and adopt a deep learning-based fully convolutional network to conduct the enhancement. The experiment results on the Mandarin corpus indicate that this newly presented multi-modal (integrating bone- and air-conducted signals) SE structure significantly outperforms the single-source SE counterparts (with a bone- or air-conducted signal only) in various speech evaluation metrics. In addition, the adoption of an LF strategy other than an EF in this novel SE multi-modal structure achieves better results.

multi-modal, bone/air-conducted signals, speech enhancement, fully convolutional network

References in corpus (4)

• Conditional Generative Adversarial Networks for Speech Enhancement and Noise-Robust Speaker Verification
• Raw Waveform-based Speech Enhancement by Fully Convolutional Networks
• Multi-Modal Hybrid Deep Neural Network for Speech Enhancement
• Tensor-to-Vector Regression for Multi-channel Speech Enhancement based on Tensor-Train Network

Cited by in corpus (7)

• EPG2S: Speech Generation and Speech Enhancement based on Electropalatography and Audio Signals using Multimodal Learning
• Multi-Microphone Noise Data Augmentation for DNN-based Own Voice Reconstruction for Hearables in Noisy Environments
• Improved Lite Audio-Visual Speech Enhancement
• Dictionary-Based Fusion of Contact and Acoustic Microphones for Wind Noise Reduction
• Exploring Deep Hybrid Tensor-to-Vector Network Architectures for Regression Based Speech Enhancement
• Low-Complexity Own Voice Reconstruction for Hearables with an In-Ear Microphone
• Speech-dependent Data Augmentation for Own Voice Reconstruction with Hearable Microphones in Noisy Environments