Advances and Challenges in Deep Lip Reading

Abstract

Driven by deep learning techniques and large-scale datasets, recent years have witnessed a paradigm shift in automatic lip reading. While the main thrust of Visual Speech Recognition (VSR) was improving accuracy of Audio Speech Recognition systems, other potential applications, such as biometric identification, and the promised gains of VSR systems, have motivated extensive efforts on developing the lip reading technology. This paper provides a comprehensive survey of the state-of-the-art deep learning based VSR research with a focus on data challenges, task-specific complications, and the corresponding solutions. Advancements in these directions will expedite the transformation of silent speech interface from theory to practice. We also discuss the main modules of a VSR pipeline and the influential datasets. Finally, we introduce some typical VSR application concerns and impediments to real-world scenarios as well as future research directions.

References in corpus (19)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
• Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning?
• Character-Aware Neural Language Models
• Object Detection in 20 Years: A Survey
• RetinaFace: Single-stage Dense Face Localisation in the Wild
• LRS3-TED: a large-scale dataset for visual speech recognition
• LipNet: End-to-End Sentence-level Lipreading
• Understanding and Improving Knowledge Distillation
• Lip-reading with Densely Connected Temporal Convolutional Networks
• Learn an Effective Lip Reading Model without Pains
• Learning Spatio-Temporal Features with Two-Stream Deep 3D CNNs for Lipreading
• An End-to-end 3D Convolutional Neural Network for Action Detection and Segmentation in Videos
• Comparing phonemes and visemes with DNN-based lipreading
• Pseudo-Convolutional Policy Gradient for Sequence-to-Sequence Lip-Reading
• Lip-reading with Hierarchical Pyramidal Convolution and Self-Attention
• Hallucinating Optical Flow Features for Video Classification
• Disentangling Homophemes in Lip Reading using Perplexity Analysis
• Cascaded CNN-resBiLSTM-CTC: An End-to-End Acoustic Model For Speech Recognition