MobiVSR: A Visual Speech Recognition Solution for Mobile Devices
arXiv:1905.03968
Abstract
Visual speech recognition (VSR) is the task of recognizing spoken language from video input only, without any audio. VSR has many applications as an assistive technology, especially if it could be deployed in mobile devices and embedded systems. The need of intensive computational resources and large memory footprint are two of the major obstacles in developing neural network models for VSR in a resource constrained environment. We propose a novel end-to-end deep neural network architecture for word level VSR called MobiVSR with a design parameter that aids in balancing the model's accuracy and parameter count. We use depthwise-separable 3D convolution for the first time in the domain of VSR and show how it makes our model efficient. MobiVSR achieves an accuracy of 73\% on a challenging Lip Reading in the Wild dataset with 6 times fewer parameters and 20 times lesser memory footprint than the current state of the art. MobiVSR can also be compressed to 6 MB by applying post training quantization.
References in corpus (16)
- The NumPy array: a structure for efficient numerical computation
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
- MobileNetV2: Inverted Residuals and Linear Bottlenecks
- cuDNN: Efficient Primitives for Deep Learning
- Compressing Neural Networks with the Hashing Trick
- Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition
- Quasi-Recurrent Neural Networks
- Fast Training of Convolutional Networks through FFTs
- Fast Convolutional Nets With fbfft: A GPU Performance Evaluation
- Model compression via distillation and quantization
- LipNet: End-to-End Sentence-level Lipreading
- On the Compression of Recurrent Neural Networks with an Application to LVCSR acoustic modeling for Embedded Speech Recognition
- Dynamically Hierarchy Revolution: DirNet for Compressing Recurrent Neural Network on Mobile Devices
- Learning Motion in Feature Space: Locally-Consistent Deformable Convolution Networks for Fine-Grained Action Detection
- 3D Depthwise Convolution: Reducing Model Parameters in 3D Vision Tasks