LRS3-TED: a large-scale dataset for visual speech recognition

Abstract

This paper introduces a new multi-modal dataset for visual and audio-visual speech recognition. It includes face tracks from over 400 hours of TED and TEDx videos, along with the corresponding subtitles and word alignment boundaries. The new dataset is substantially larger in scale compared to other public datasets that are available for general research.

arXiv commentThe audio-visual dataset can be downloaded from http://www.robots.ox.ac.uk/~vgg/data/lip_reading/

References in corpus (1)

• LipNet: End-to-End Sentence-level Lipreading

Cited by in corpus (43)

• A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild
• Deep Audio-Visual Speech Recognition
• Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection
• Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labels
• Training Strategies for Improved Lip-reading
• VATLM: Visual-Audio-Text Pre-Training with Unified Masked Prediction for Speech Representation Learning
• Deep Person Generation: A Survey from the Perspective of Face, Pose and Cloth Synthesis
• Seeing wake words: Audio-visual Keyword Spotting
• Scaling up sign spotting through sign language dictionaries
• Recurrent Neural Network Transducer for Audio-Visual Speech Recognition
• SwinLip: An Efficient Visual Speech Encoder for Lip Reading Using Swin Transformer
• MM-ALT: A Multimodal Automatic Lyric Transcription System
• Emphasizing Unseen Words: New Vocabulary Acquisition for End-to-End Speech Recognition
• LPIPS-AttnWav2Lip: Generic Audio-Driven lip synchronization for Talking Head Generation in the Wild
• Learning Individual Speaking Styles for Accurate Lip to Speech Synthesis
• Attentive Fusion Enhanced Audio-Visual Encoding for Transformer Based Robust Speech Recognition
• Leveraging Visual Supervision for Array-based Active Speaker Detection and Localization
• Mutual Information Maximization for Effective Lip Reading
• Visual Keyword Spotting with Attention
• Learning from Videos with Deep Convolutional LSTM Networks
• Talking-head Generation with Rhythmic Head Motion
• Exploring Deep Learning for Joint Audio-Visual Lip Biometrics
• End-to-end Audio-visual Speech Recognition with Conformers
• Sub-word Level Lip Reading With Visual Attention
• Discriminative Multi-modality Speech Recognition
• Animating Face using Disentangled Audio Representations
• Fusing information streams in end-to-end audio-visual speech recognition
• Speaker-Adapted End-to-End Visual Speech Recognition for Continuous Spanish
• Advances and Challenges in Deep Lip Reading
• The VVAD-LRS3 Dataset for Visual Voice Activity Detection
• Multimodal Target Speech Separation with Voice and Face References
• Contrastive Learning of Global-Local Video Representations
• Muse: Multi-modal target speaker extraction with visual cues
• "Notic My Speech" -- Blending Speech Patterns With Multimedia
• LiRA: Learning Visual Speech Representations from Audio through Self-supervision
• PIAVE: A Pose-Invariant Audio-Visual Speaker Extraction Network
• Personalized One-Shot Lipreading for an ALS Patient
• Visual Speech Enhancement Without A Real Visual Stream
• Evaluation of End-to-End Continuous Spanish Lipreading in Different Data Conditions
• Audio-Visual Speech Separation Using Cross-Modal Correspondence Loss
• Interactive decoding of words from visual speech recognition models
• An Empirical Study of Visual Features for DNN based Audio-Visual Speech Enhancement in Multi-talker Environments
• Improving the Gap in Visual Speech Recognition Between Normal and Silent Speech Based on Metric Learning