Visual Keyword Spotting with Attention
arXiv:2110.15957
Abstract
In this paper, we consider the task of spotting spoken keywords in silent video sequences -- also known as visual keyword spotting. To this end, we investigate Transformer-based models that ingest two streams, a visual encoding of the video and a phonetic encoding of the keyword, and output the temporal location of the keyword if present. Our contributions are as follows: (1) We propose a novel architecture, the Transpotter, that uses full cross-modal attention between the visual and phonetic streams; (2) We show through extensive evaluations that our model outperforms the prior state-of-the-art visual keyword spotting and lip reading methods on the challenging LRW, LRS2, LRS3 datasets by a large margin; (3) We demonstrate the ability of our model to spot words under the extreme conditions of isolated mouthings in sign language videos.
arXiv commentAppears in: British Machine Vision Conference 2021 (BMVC 2021)
References in corpus (15)
- Deep Audio-Visual Speech Recognition
- Visual Transformers: Token-based Image Representation and Processing for Computer Vision
- Hello Edge: Keyword Spotting on Microcontrollers
- LRS3-TED: a large-scale dataset for visual speech recognition
- LipNet: End-to-End Sentence-level Lipreading
- Transformers with convolutional context for ASR
- Keyword Transformer: A Self-Attention Model for Keyword Spotting
- End-to-End ASR-free Keyword Search from Speech
- Online Keyword Spotting with a Character-Level Recurrent Neural Network
- An End-to-End Architecture for Keyword Spotting and Voice Activity Detection
- Dense Regression Network for Video Grounding
- Signs in time: Encoding human motion as a temporal image
- Small-footprint Keyword Spotting with Graph Convolutional Network
- Sequence-to-sequence Models for Small-Footprint Keyword Spotting
- Sub-word Level Lip Reading With Visual Attention