Lip-reading with Densely Connected Temporal Convolutional Networks

Abstract

In this work, we present the Densely Connected Temporal Convolutional Network (DC-TCN) for lip-reading of isolated words. Although Temporal Convolutional Networks (TCN) have recently demonstrated great potential in many vision tasks, its receptive fields are not dense enough to model the complex temporal dynamics in lip-reading scenarios. To address this problem, we introduce dense connections into the network to capture more robust temporal features. Moreover, our approach utilises the Squeeze-and-Excitation block, a light-weight attention mechanism, to further enhance the model's classification power. Without bells and whistles, our DC-TCN method has achieved 88.36% accuracy on the Lip Reading in the Wild (LRW) dataset and 43.65% on the LRW-1000 dataset, which has surpassed all the baseline methods and is the new state-of-the-art on both datasets.

WACV 2021. An improved code implementation is available at: https://github.com/mpc001/Lipreading_using_Temporal_Convolutional_Networks

References in corpus (13)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Neural Machine Translation by Jointly Learning to Align and Translate
• Decoupled Weight Decay Regularization
• Rethinking Atrous Convolution for Semantic Image Segmentation
• An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
• WaveNet: A Generative Model for Raw Audio
• Language Modeling with Gated Convolutional Networks
• Deep Audio-Visual Speech Recognition
• Lip Reading Sentences in the Wild
• Multi-Grained Spatio-temporal Modeling for Lip-reading
• Learning Spatio-Temporal Features with Two-Stream Deep 3D CNNs for Lipreading

Cited by in corpus (8)

• Training Strategies for Improved Lip-reading
• Do Smart Glasses Dream of Sentimental Visions? Deep Emotionship Analysis for Eyewear Devices
• SwinLip: An Efficient Visual Speech Encoder for Lip Reading Using Swin Transformer
• Expression-preserving face frontalization improves visually assisted speech processing
• Dilated Convolutions with Lateral Inhibitions for Semantic Image Segmentation
• Advances and Challenges in Deep Lip Reading
• TD3Net: A temporal densely connected multi-dilated convolutional network for lipreading
• LiRA: Learning Visual Speech Representations from Audio through Self-supervision