Deep Audio-Visual Speech Recognition
arXiv:1809.02108 · doi:10.1109/TPAMI.2018.2889052
Abstract
The goal of this work is to recognise phrases and sentences being spoken by a talking face, with or without the audio. Unlike previous works that have focussed on recognising a limited number of words or phrases, we tackle lip reading as an open-world problem - unconstrained natural language sentences, and in the wild videos. Our key contributions are: (1) we compare two models for lip reading, one using a CTC loss, and the other using a sequence-to-sequence loss. Both models are built on top of the transformer self-attention architecture; (2) we investigate to what extent lip reading is complementary to audio speech recognition, especially when the audio signal is noisy; (3) we introduce and publicly release a new dataset for audio-visual speech recognition, LRS2-BBC, consisting of thousands of natural sentences from British television. The models that we train surpass the performance of all previous work on a lip reading benchmark dataset by a significant margin.
arXiv commentAccepted for publication by IEEE Transactions on Pattern Analysis and Machine Intelligence
References in corpus (21)
- Adam: A Method for Stochastic Optimization
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- SSD: Single Shot MultiBox Detector
- Neural Machine Translation by Jointly Learning to Align and Translate
- Sequence to Sequence Learning with Neural Networks
- TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
- Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
- Attention-Based Models for Speech Recognition
- Sequence Transduction with Recurrent Neural Networks
- Lip Reading Sentences in the Wild
- End-to-end Continuous Speech Recognition using Attention-based Recurrent NN: First Results
- Convolutional Two-Stream Network Fusion for Video Action Recognition
- Listen, Attend and Spell
- LRS3-TED: a large-scale dataset for visual speech recognition
- Wav2Letter: an End-to-End ConvNet-based Speech Recognition System
- LipNet: End-to-End Sentence-level Lipreading
- Residual Convolutional CTC Networks for Automatic Speech Recognition
- Lipreading with Long Short-Term Memory
- Deep Multimodal Learning for Audio-Visual Speech Recognition
Cited by in corpus (87)
- A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild
- A General Survey on Attention Mechanisms in Deep Learning
- Multimodal Intelligence: Representation Learning, Information Fusion, and Applications
- Visual Speech Recognition for Multiple Languages in the Wild
- Silent Speech Interfaces for Speech Restoration: A Review
- Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection
- Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labels
- Multi-modal Multi-channel Target Speech Separation
- Recent Progress in the CUHK Dysarthric Speech Recognition System
- Emotional Speech-Driven Animation with Content-Emotion Disentanglement
- Lip-reading with Densely Connected Temporal Convolutional Networks
- Temporal Sentence Grounding in Videos: A Survey and Future Directions
- Learning Spatio-Temporal Features with Two-Stream Deep 3D CNNs for Lipreading
- On quadrature rules for solving Partial Differential Equations using Neural Networks
- How to Teach DNNs to Pay Attention to the Visual Modality in Speech Recognition
- XPipe: Efficient Pipeline Model Parallelism for Multi-GPU DNN Training
- Deep Person Generation: A Survey from the Perspective of Face, Pose and Cloth Synthesis
- VATLM: Visual-Audio-Text Pre-Training with Unified Masked Prediction for Speech Representation Learning
- LipSound2: Self-Supervised Pre-Training for Lip-to-Speech Reconstruction and Lip Reading
- Towards Realistic Visual Dubbing with Heterogeneous Sources
- Naver at ActivityNet Challenge 2019 -- Task B Active Speaker Detection (AVA)
- Listening to Sounds of Silence for Speech Denoising
- Seeing wake words: Audio-visual Keyword Spotting
- Multi-modal embeddings using multi-task learning for emotion recognition
- A multimodal dynamical variational autoencoder for audiovisual speech representation learning
- Ubi-SleepNet: Advanced Multimodal Fusion Techniques for Three-stage Sleep Classification Using Ubiquitous Sensing
- Expressive Speech-driven Facial Animation with controllable emotions
- Deep Audio-Visual Learning: A Survey
- Lip-to-Speech Synthesis for Arbitrary Speakers in the Wild
- LipReading with 3D-2D-CNN BLSTM-HMM and word-CTC models
- SwinLip: An Efficient Visual Speech Encoder for Lip Reading Using Swin Transformer
- From Inference to Generation: End-to-end Fully Self-supervised Generation of Human Face from Speech
- MM-ALT: A Multimodal Automatic Lyric Transcription System
- Convolutional Neural Networks for Speech Controlled Prosthetic Hands
- VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency
- Speech Separation with Pretrained Frontend to Minimize Domain Mismatch
- Audio-visual Recognition of Overlapped speech for the LRS2 dataset
- Attentive Fusion Enhanced Audio-Visual Encoding for Transformer Based Robust Speech Recognition
- Unsupervised Modality-Transferable Video Highlight Detection with Representation Activation Sequence Learning
- Uncertainty aware audiovisual activity recognition using deep Bayesian variational inference
- Multiresolution and Multimodal Speech Recognition with Transformers
- Pseudo-Convolutional Policy Gradient for Sequence-to-Sequence Lip-Reading
- Semantically-Aware Attentive Neural Embeddings for Image-based Visual Localization
- Can We Read Speech Beyond the Lips? Rethinking RoI Selection for Deep Visual Speech Recognition
- Visual Keyword Spotting with Attention
- Whose Emotion Matters? Speaking Activity Localisation without Prior Knowledge
- Towards Practical Lipreading with Distilled and Efficient Models
- Multimodal Integration for Large-Vocabulary Audio-Visual Speech Recognition
- Synchronous Bidirectional Learning for Multilingual Lip Reading
- Detection of Audio-Video Synchronization Errors Via Event Detection
- Distilling Audio-Visual Knowledge by Compositional Contrastive Learning
- Lip-reading with Hierarchical Pyramidal Convolution and Self-Attention
- Learning from Videos with Deep Convolutional LSTM Networks
- End-to-end Audio-visual Speech Recognition with Conformers
- Uncovering the Visual Contribution in Audio-Visual Speech Recognition
- AVE Speech: A Comprehensive Multi-Modal Dataset for Speech Recognition Integrating Audio, Visual, and Electromyographic Signals
- Discriminative Multi-modality Speech Recognition
- On the Importance of Video Action Recognition for Visual Lipreading
- Sub-word Level Lip Reading With Visual Attention
- Listen, Look and Deliberate: Visual context-aware speech recognition using pre-trained text-video representations
- Spatio-Temporal Attention Mechanism and Knowledge Distillation for Lip Reading
- Speaker-Adapted End-to-End Visual Speech Recognition for Continuous Spanish
- Fusing information streams in end-to-end audio-visual speech recognition
- Neural Dubber: Dubbing for Videos According to Scripts
- Contrastive Learning of Global-Local Video Representations
- Audio-visual Multi-channel Integration and Recognition of Overlapped Speech
- Muse: Multi-modal target speaker extraction with visual cues
- Audio-visual Multi-channel Recognition of Overlapped Speech
- LiRA: Learning Visual Speech Representations from Audio through Self-supervision
- Where and When: Space-Time Attention for Audio-Visual Explanations
- Visual Speech Enhancement Without A Real Visual Stream
- Personalized One-Shot Lipreading for an ALS Patient
- PIAVE: A Pose-Invariant Audio-Visual Speaker Extraction Network
- Augmenting Images for ASR and TTS through Single-loop and Dual-loop Multimodal Chain Framework
- Interpretable Multimodal Learning for Intelligent Regulation in Online Payment Systems
- Towards Pose-invariant Lip-Reading
- Continuous Speech Recognition using EEG and Video
- Audio-Visual Decision Fusion for WFST-based and seq2seq Models
- Joint Distribution across Representation Space for Out-of-Distribution Detection
- A Comprehensive Multi-scale Approach for Speech and Dynamics Synchrony in Talking Head Generation
- Multi-modal Speech Transformer Decoders: When Do Multiple Modalities Improve Accuracy?
- Perception Point: Identifying Critical Learning Periods in Speech for Bilingual Networks
- Interactive decoding of words from visual speech recognition models
- "Is this an example image?" -- Predicting the Relative Abstractness Level of Image and Text
- Improving the Gap in Visual Speech Recognition Between Normal and Silent Speech Based on Metric Learning
- Mixed Precision DNN Qunatization for Overlapped Speech Separation and Recognition
- Binaural Audio Generation via Multi-task Learning