Learning Spatio-Temporal Features with Two-Stream Deep 3D CNNs for Lipreading

Abstract

We focus on the word-level visual lipreading, which requires recognizing the word being spoken, given only the video but not the audio. State-of-the-art methods explore the use of end-to-end neural networks, including a shallow (up to three layers) 3D convolutional neural network (CNN) + a deep 2D CNN (e.g., ResNet) as the front-end to extract visual features, and a recurrent neural network (e.g., bidirectional LSTM) as the back-end for classification. In this work, we propose to replace the shallow 3D CNNs + deep 2D CNNs front-end with recent successful deep 3D CNNs --- two-stream (i.e., grayscale video and optical flow streams) I3D. We evaluate different combinations of front-end and back-end modules with the grayscale video and optical flow inputs on the LRW dataset. The experiments show that, compared to the shallow 3D CNNs + deep 2D CNNs front-end, the deep 3D CNNs front-end with pre-training on the large-scale image and video datasets (e.g., ImageNet and Kinetics) can improve the classification accuracy. Also, we demonstrate that using the optical flow input alone can achieve comparable performance as using the grayscale video as input. Moreover, the two-stream network using both the grayscale video and optical flow inputs can further improve the performance. Overall, our two-stream I3D front-end with a Bi-LSTM back-end results in an absolute improvement of 5.3% over the previous art on the LRW dataset.

camera ready version for BMVC 2019

References in corpus (8)

• Two-Stream Convolutional Networks for Action Recognition in Videos
• Deep Audio-Visual Speech Recognition
• Convolutional Two-Stream Network Fusion for Video Action Recognition
• Beyond Short Snippets: Deep Networks for Video Classification
• Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks
• LipNet: End-to-End Sentence-level Lipreading
• Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?
• Lipreading with Long Short-Term Memory

Cited by in corpus (8)

• Lip-reading with Densely Connected Temporal Convolutional Networks
• Training Strategies for Improved Lip-reading
• LipSound2: Self-Supervised Pre-Training for Lip-to-Speech Reconstruction and Lip Reading
• Towards Practical Lipreading with Distilled and Efficient Models
• Deformation Flow Based Two-Stream Network for Lip Reading
• Contrastive Learning of Global-Local Video Representations
• Advances and Challenges in Deep Lip Reading
• TD3Net: A temporal densely connected multi-dilated convolutional network for lipreading