58 citations · 255 across the 31 of their papers we have counts for
31 papers
ReliTalk: Relightable Talking Portrait Generation from a Single Video
Haonan Qiu, Zhaoxi Chen, Yuming Jiang +5
Recent years have witnessed great progress in creating vivid audio-driven portraits from monocular videos. However, how to seamlessly adapt the created video avatars to other scena…
StyleSync: High-Fidelity Generalized and Personalized Lip Sync in Style-based Generator
Jiazhi Guan, Zhanwang Zhang, Hang Zhou +8
Despite recent advances in syncing lip movements with any audio waves, current methods still struggle to balance generation quality and the model's generalization ability. Previous…
Delicate Textured Mesh Recovery from NeRF via Adaptive Surface Refinement
Jiaxiang Tang, Hang Zhou, Xiaokang Chen +4
Neural Radiance Fields (NeRF) have constituted a remarkable breakthrough in image-based 3D reconstruction. However, their implicit volumetric representations differ significantly f…
Make Your Brief Stroke Real and Stereoscopic: 3D-Aware Simplified Sketch to Portrait Generation
Yasheng Sun, Qianyi Wu, Hang Zhou +6
Creating the photo-realistic version of people sketched portraits is useful to various entertainment purposes. Existing studies only generate portraits in the 2D plane with fixed v…
Masked Lip-Sync Prediction by Audio-Visual Contextual Exploitation in Transformers
Yasheng Sun, Hang Zhou, Kaisiyuan Wang +7
Previous studies have explored generating accurately lip-synced talking faces for arbitrary targets given audio conditions. However, most of them deform or generate the whole facia…
Audio-Driven Co-Speech Gesture Video Generation
Xian Liu, Qianyi Wu, Hang Zhou +4
Co-speech gesture is crucial for human-machine interaction and digital entertainment. While previous works mostly map speech audio to human skeletons (e.g., 2D keypoints), directly…
Real-time Neural Radiance Talking Portrait Synthesis via Audio-spatial Decomposition
Jiaxiang Tang, Kaisiyuan Wang, Hang Zhou +6
While dynamic Neural Radiance Fields (NeRF) have shown success in high-fidelity 3D modeling of talking portraits, the slow training and inference speed severely obstruct their pote…
StyleSwap: Style-Based Generator Empowers Robust Face Swapping
Zhiliang Xu, Hang Zhou, Zhibin Hong +7
Numerous attempts have been made to the task of person-agnostic face swapping given its wide applications. While existing methods mostly rely on tedious network and loss designs, t…
StyleFaceV: Face Video Generation via Decomposing and Recomposing Pretrained StyleGAN3
Haonan Qiu, Yuming Jiang, Hang Zhou +2
Realistic generative face video synthesis has long been a pursuit in both computer vision and graphics community. However, existing face video generation methods tend to produce lo…
Detecting Deepfake by Creating Spatio-Temporal Regularity Disruption
Jiazhi Guan, Hang Zhou, Mingming Gong +3
Despite encouraging progress in deepfake detection, generalization to unseen forgery types remains a significant challenge due to the limited forgery clues explored during training…
TokenMix: Rethinking Image Mixing for Data Augmentation in Vision Transformers
Jihao Liu, Boxiao Liu, Hang Zhou +2
CutMix is a popular augmentation technique commonly used for training modern convolutional and transformer vision networks. It was originally designed to encourage Convolution Neur…
EAMM: One-Shot Emotional Talking Face via Audio-Based Emotion-Aware Motion Model
Xinya Ji, Hang Zhou, Kaisiyuan Wang +4
Although significant progress has been made to audio-driven talking face generation, existing methods either neglect facial emotion or cannot be applied to arbitrary subjects. In t…
Few-Shot Head Swapping in the Wild
Changyong Shu, Hemao Wu, Hang Zhou +7
The head swapping task aims at flawlessly placing a source head onto a target body, which is of great importance to various entertainment scenarios. While face swapping has drawn m…
Joint-Modal Label Denoising for Weakly-Supervised Audio-Visual Video Parsing
Haoyue Cheng, Zhaoyang Liu, Hang Zhou +3
This paper focuses on the weakly-supervised audio-visual video parsing task, which aims to recognize all events belonging to each modality and localize their temporal boundaries. T…
SeCo: Separating Unknown Musical Visual Sounds with Consistency Guidance
Xinchi Zhou, Dongzhan Zhou, Wanli Ouyang +3
Recent years have witnessed the success of deep learning on the visual sound separation task. However, existing works follow similar settings where the training and testing dataset…
Learning Hierarchical Cross-Modal Association for Co-Speech Gesture Generation
Xian Liu, Qianyi Wu, Hang Zhou +7
Generating speech-consistent body and gesture movements is a long-standing problem in virtual avatar creation. Previous studies often synthesize pose movement in a holistic manner,…
Visual Sound Localization in the Wild by Cross-Modal Interference Erasing
Xian Liu, Rui Qian, Hang Zhou +5
The task of audio-visual sound source localization has been well studied under constrained scenes, where the audio recordings are clean. However, in real-world scenarios, audios ar…
STC: Spatio-Temporal Contrastive Learning for Video Instance Segmentation
Zhengkai Jiang, Zhangxuan Gu, Jinlong Peng +6
Video Instance Segmentation (VIS) is a task that simultaneously requires classification, segmentation, and instance association in a video. Recent VIS approaches rely on sophistica…
Semantic-Aware Implicit Neural Audio-Driven Video Portrait Generation
Xian Liu, Yinghao Xu, Qianyi Wu +3
Animating high-fidelity video portrait with speech audio is crucial for virtual reality and digital entertainment. While most previous studies rely on accurate explicit structural…
Self-Supervised Monocular Depth Estimation with Internal Feature Fusion
Hang Zhou, David Greenwood, Sarah Taylor
Self-supervised learning for depth estimation uses geometry in image sequences for supervision and shows promising results. Like many computer vision tasks, depth network performan…
Hyperspectral Mixed Noise Removal via Subspace Representation and Weighted Low-rank Tensor Regularization
Hang Zhou, Yanchi Su, Zhanshan Li
Recently, the low-rank property of different components extracted from the image has been considered in man hyperspectral image denoising methods. However, these methods usually un…
SUB-Depth: Self-distillation and Uncertainty Boosting Self-supervised Monocular Depth Estimation
Hang Zhou, Sarah Taylor, David Greenwood +1
We propose SUB-Depth, a universal multi-task training framework for self-supervised monocular depth estimation (SDE). Depth models trained with SUB-Depth outperform the same models…
Audio-Driven Emotional Video Portraits
Xinya Ji, Hang Zhou, Kaisiyuan Wang +4
Despite previous success in generating audio-driven talking heads, most of the previous studies focus on the correlation between speech content and the mouth shape. Facial emotion,…
Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation
Hang Zhou, Yasheng Sun, Wayne Wu +3
While accurate lip synchronization has been achieved for arbitrary-subject audio-driven talking face generation, the problem of how to efficiently drive the head pose remains. Prev…