7 papers · 1 filter
FlashLips: 100-FPS Mask-Free Latent Lip-Sync using Reconstruction Instead of Diffusion or GANs
Andreas Zinonos, MichaÅ StypuÅkowski, Antoni Bigata +3
We present FlashLips, a two-stage, mask-free lip-sync system that decouples lips control from rendering and achieves real-time performance, with our U-Net variant running at over 1…
FaceCrafter: Identity-Conditional Diffusion with Disentangled Control over Facial Pose, Expression, and Emotion
Kazuaki Mishima, Antoni Bigata Casademunt, Stavros Petridis +2
Human facial images encode a rich spectrum of information, encompassing both stable identity-related traits and mutable attributes such as pose, expression, and emotion. While rece…
KeySync: A Robust Approach for Leakage-free Lip Synchronization in High Resolution
Antoni Bigata, Rodrigo Mira, Stella Bounareli +4
Lip synchronization, known as the task of aligning lip movements in an existing video with new input audio, is typically framed as a simpler variant of audio-driven facial animatio…
Large Language Models are Strong Audio-Visual Speech Recognition Learners
Umberto Cappellazzo, Minsu Kim, Honglie Chen +5
Multimodal large language models (MLLMs) have recently become a focal point of research due to their formidable multimodal understanding capabilities. For example, in the audio and…
Unified Speech Recognition: A Single Model for Auditory, Visual, and Audiovisual Inputs
Alexandros Haliassos, Rodrigo Mira, Honglie Chen +3
Research in auditory, visual, and audiovisual speech recognition (ASR, VSR, and AVSR, respectively) has traditionally been conducted independently. Even recent self-supervised stud…
MSRS: Training Multimodal Speech Recognition Models from Scratch with Sparse Mask Optimization
Adriana Fernandez-Lopez, Honglie Chen, Pingchuan Ma +5
Pre-trained models have been a foundational approach in speech recognition, albeit with associated additional costs. In this study, we propose a regularization technique that facil…