Deep Multimodal Learning for Audio-Visual Speech Recognition

Abstract

In this paper, we present methods in deep multimodal learning for fusing speech and visual modalities for Audio-Visual Automatic Speech Recognition (AV-ASR). First, we study an approach where uni-modal deep networks are trained separately and their final hidden layers fused to obtain a joint feature space in which another deep network is built. While the audio network alone achieves a phone error rate (PER) of $41\%$ under clean condition on the IBM large vocabulary audio-visual studio dataset, this fusion model achieves a PER of $35.83\%$ demonstrating the tremendous value of the visual channel in phone classification even in audio with high signal to noise ratio. Second, we present a new deep network architecture that uses a bilinear softmax layer to account for class specific correlations between modalities. We show that combining the posteriors from the bilinear networks with those from the fused model mentioned above results in a further significant phone error rate reduction, yielding a final PER of $34.03\%$.

ICASSP 2015

Cited by in corpus (12)

• Deep Audio-Visual Speech Recognition
• Multimodal Machine Learning: A Survey and Taxonomy
• Non-invasive biomarkers of fetal brain development reflecting prenatal stress: an integrative multi-scale multi-species perspective on data collection and analysis
• Multi-task Learning Of Deep Neural Networks For Audio Visual Automatic Speech Recognition
• Robust Deep Multi-modal Learning Based on Gated Information Fusion Network
• Robust end-to-end deep audiovisual speech recognition
• Multi-Modal Data Augmentation for End-to-End ASR
• AVA-Speech: A Densely Labeled Dataset of Speech Activity in Movies
• Audio-Visual Speech Enhancement Using Multimodal Deep Convolutional Neural Networks
• Dynamic Temporal Alignment of Speech to Lips
• Latent Variable Algorithms for Multimodal Learning and Sensor Fusion
• Audio Visual Speech Recognition using Deep Recurrent Neural Networks