51 citations · 122 across the 14 of their papers we have counts for
14 papers
DBMIF: a deep balanced multimodal iterative fusion framework for air- and bone-conduction speech enhancement
Yilei Wu, Changyan Zheng, Xingyu Zhang +5
The performance of conventional speech enhancement systems degrades sharply in extremely low signal-to-noise ratio (SNR) environments where air-conduction (AC) microphones are over…
SAIL-Embedding Technical Report: Omni-modal Embedding Foundation Model
Lin Lin, Jiefeng Long, Zhihe Wan +15
Multimodal embedding models aim to yield informative unified representations that empower diverse cross-modal tasks. Despite promising developments in the evolution from CLIP-based…
GLip: A Global-Local Integrated Progressive Framework for Robust Visual Speech Recognition
Tianyue Wang, Shuang Yang, Shiguang Shan +1
Visual speech recognition (VSR), also known as lip reading, is the task of recognizing speech from silent video. Despite significant advancements in VSR over recent decades, most e…
Cooperative Dual Attention for Audio-Visual Speech Enhancement with Facial Cues
Feixiang Wang, Shuang Yang, Shiguang Shan +1
In this work, we focus on leveraging facial cues beyond the lip region for robust Audio-Visual Speech Enhancement (AVSE). The facial region, encompassing the lip region, reflects a…
Learning Separable Hidden Unit Contributions for Speaker-Adaptive Lip-Reading
Songtao Luo, Shuang Yang, Shiguang Shan +1
In this paper, we propose a novel method for speaker adaptation in lip reading, motivated by two observations. Firstly, a speaker's own characteristics can always be portrayed well…
UniCon+: ICTCAS-UCAS Submission to the AVA-ActiveSpeaker Task at ActivityNet Challenge 2022
Yuanhang Zhang, Susan Liang, Shuang Yang +1
This report presents a brief description of our winning solution to the AVA Active Speaker Detection (ASD) task at ActivityNet Challenge 2022. Our underlying model UniCon+ continue…
UniCon: Unified Context Network for Robust Active Speaker Detection
Yuanhang Zhang, Susan Liang, Shuang Yang +4
We introduce a new efficient framework, the Unified Context Network (UniCon), for robust active speaker detection (ASD). Traditional methods for ASD usually operate on each candida…
Learn an Effective Lip Reading Model without Pains
Dalu Feng, Shuang Yang, Shiguang Shan +1
Lip reading, also known as visual speech recognition, aims to recognize the speech content from videos by analyzing the lip dynamics. There have been several appealing progress in…
Synchronous Bidirectional Learning for Multilingual Lip Reading
Mingshuang Luo, Shuang Yang, Xilin Chen +2
Lip reading has received increasing attention in recent years. This paper focuses on the synergy of multilingual lip reading. There are about as many as 7000 languages in the world…
Mutual Information Maximization for Effective Lip Reading
Xing Zhao, Shuang Yang, Shiguang Shan +1
Lip reading has received an increasing research interest in recent years due to the rapid development of deep learning and its widespread potential applications. One key point to o…
Deformation Flow Based Two-Stream Network for Lip Reading
Jingyun Xiao, Shuang Yang, Yuanhang Zhang +2
Lip reading is the task of recognizing the speech content by analyzing movements in the lip region when people are speaking. Observing on the continuity in adjacent frames in the s…
Pseudo-Convolutional Policy Gradient for Sequence-to-Sequence Lip-Reading
Mingshuang Luo, Shuang Yang, Shiguang Shan +1
Lip-reading aims to infer the speech content from the lip movement sequence and can be seen as a typical sequence-to-sequence (seq2seq) problem which translates the input image seq…
Can We Read Speech Beyond the Lips? Rethinking RoI Selection for Deep Visual Speech Recognition
Yuanhang Zhang, Shuang Yang, Jingyun Xiao +2
Recent advances in deep learning have heightened interest among researchers in the field of visual speech recognition (VSR). Currently, most existing methods equate VSR with automa…
3D Feature Pyramid Attention Module for Robust Visual Speech Recognition
Jingyun Xiao
Visual speech recognition is the task to decode the speech content from a video based on visual information, especially the movements of lips. It is also referenced as lipreading.…