Personalized One-Shot Lipreading for an ALS Patient

Abstract

Lipreading or visually recognizing speech from the mouth movements of a speaker is a challenging and mentally taxing task. Unfortunately, multiple medical conditions force people to depend on this skill in their day-to-day lives for essential communication. Patients suffering from Amyotrophic Lateral Sclerosis (ALS) often lose muscle control, consequently their ability to generate speech and communicate via lip movements. Existing large datasets do not focus on medical patients or curate personalized vocabulary relevant to an individual. Collecting a large-scale dataset of a patient, needed to train mod-ern data-hungry deep learning models is, however, extremely challenging. In this work, we propose a personalized network to lipread an ALS patient using only one-shot examples. We depend on synthetically generated lip movements to augment the one-shot scenario. A Variational Encoder based domain adaptation technique is used to bridge the real-synthetic domain gap. Our approach significantly improves and achieves high top-5accuracy with 83.2% accuracy compared to 62.6% achieved by comparable methods for the patient. Apart from evaluating our approach on the ALS patient, we also extend it to people with hearing impairment relying extensively on lip movements to communicate.

References in corpus (8)

• GAN-based Synthetic Medical Image Augmentation for increased CNN Performance in Liver Lesion Classification
• A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild
• Deep Audio-Visual Speech Recognition
• Adversarial Discriminative Domain Adaptation
• LRS3-TED: a large-scale dataset for visual speech recognition
• Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search
• 3D Convolutional Neural Networks for Cross Audio-Visual Matching Recognition
• Learn an Effective Lip Reading Model without Pains