LipNet: End-to-End Sentence-level Lipreading

Abstract

Lipreading is the task of decoding text from the movement of a speaker's mouth. Traditional approaches separated the problem into two stages: designing or learning visual features, and prediction. More recent deep lipreading approaches are end-to-end trainable (Wand et al., 2016; Chung & Zisserman, 2016a). However, existing work on models trained end-to-end perform only word classification, rather than sentence-level sequence prediction. Studies have shown that human lipreading performance increases for longer words (Easton & Basala, 1982), indicating the importance of features capturing temporal context in an ambiguous communication channel. Motivated by this observation, we present LipNet, a model that maps a variable-length sequence of video frames to text, making use of spatiotemporal convolutions, a recurrent network, and the connectionist temporal classification loss, trained entirely end-to-end. To the best of our knowledge, LipNet is the first end-to-end sentence-level lipreading model that simultaneously learns spatiotemporal visual features and a sequence model. On the GRID corpus, LipNet achieves 95.2% accuracy in sentence-level, overlapped speaker split task, outperforming experienced human lipreaders and the previous 86.4% word-level state-of-the-art accuracy (Gergen et al., 2016).

References in corpus (2)

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Striving for Simplicity: The All Convolutional Net

Cited by in corpus (16)

• Learning Spatio-Temporal Features with Two-Stream Deep 3D CNNs for Lipreading
• Cortical microcircuits as gated-recurrent neural networks
• Naver at ActivityNet Challenge 2019 -- Task B Active Speaker Detection (AVA)
• Recurrent Neural Network Transducer for Audio-Visual Speech Recognition
• LipReading with 3D-2D-CNN BLSTM-HMM and word-CTC models
• Attentive Fusion Enhanced Audio-Visual Encoding for Transformer Based Robust Speech Recognition
• Vid2speech: Speech Reconstruction from Silent Video
• Simulation Experiment of BCI Based on Imagined Speech EEG Decoding
• MobiVSR: A Visual Speech Recognition Solution for Mobile Devices
• "Notic My Speech" -- Blending Speech Patterns With Multimedia
• Continuous Speech Recognition using EEG and Video
• Towards Pose-invariant Lip-Reading
• Audio-Visual Decision Fusion for WFST-based and seq2seq Models
• Sensor Transformation Attention Networks
• Latent Variable Algorithms for Multimodal Learning and Sensor Fusion
• An Empirical Analysis of Deep Audio-Visual Models for Speech Recognition