47 citations · 135 across the 21 of their papers we have counts for
22 papers
Avatar V: Scaling Video-Reference Avatar Video Generation
Benjamin Liang, Ce Chen, Desmond Lin +20
Generating avatar videos that are not merely visually similar to a target individual but behaviorally recognizable, faithfully reproducing their talking rhythm, gestural tendencies…
Generate Your Talking Avatar from Video Reference
Zujin Guo, Zhenhui Ye, Yi Ren +4
Existing talking avatar methods typically adopt an image-to-video pipeline conditioned on a static reference image within the same scene as the target generation. This restricted,…
TransVLM: A Vision-Language Framework and Benchmark for Detecting Any Shot Transitions
Ce Chen, Yi Ren, Yuanming Li +5
Traditional Shot Boundary Detection (SBD) inherently struggles with complex transitions by formulating the task around isolated cut points, frequently yielding corrupted video shot…
SSGaussian: Semantic-Aware and Structure-Preserving 3D Style Transfer
Jimin Xu, Bosheng Qin, Tao Jin +4
Recent advancements in neural representations, such as Neural Radiance Fields and 3D Gaussian Splatting, have increased interest in applying style transfer to 3D scenes. While exis…
T2A-Feedback: Improving Basic Capabilities of Text-to-Audio Generation via Fine-grained AI Feedback
Zehan Wang, Ke Lei, Chen Zhu +8
Text-to-audio (T2A) generation has achieved remarkable progress in generating a variety of audio outputs from language prompts. However, current state-of-the-art T2A models still s…
MegaTTS 3: Sparse Alignment Enhanced Latent Diffusion Transformer for Zero-Shot Speech Synthesis
Ziyue Jiang, Yi Ren, Ruiqi Li +11
While recent zero-shot text-to-speech (TTS) models have significantly improved speech quality and expressiveness, mainstream systems still suffer from issues related to speech-text…
HumanDiT: Pose-Guided Diffusion Transformer for Long-form Human Motion Video Generation
Qijun Gan, Yi Ren, Chen Zhang +6
Human motion video generation has advanced significantly, while existing methods still struggle with accurately rendering detailed body parts like hands and faces, especially in lo…
MimicTalk: Mimicking a personalized and expressive 3D talking face in minutes
Zhenhui Ye, Tianyun Zhong, Yi Ren +10
Talking face generation (TFG) aims to animate a target identity's face to create realistic talking videos. Personalized TFG is a variant that emphasizes the perceptual identity sim…
MulliVC: Multi-lingual Voice Conversion With Cycle Consistency
Jiawei Huang, Chen Zhang, Yi Ren +6
Voice conversion aims to modify the source speaker's voice to resemble the target speaker while preserving the original speech content. Despite notable advancements in voice conver…
FreeBind: Free Lunch in Unified Multimodal Space via Knowledge Fusion
Zehan Wang, Ziang Zhang, Xize Cheng +8
Unified multi-model representation spaces are the foundation of multimodal understanding and generation. However, the billions of model parameters and catastrophic forgetting probl…
Real3D-Portrait: One-shot Realistic 3D Talking Portrait Synthesis
Zhenhui Ye, Tianyun Zhong, Yi Ren +11
One-shot 3D talking portrait generation aims to reconstruct a 3D avatar from an unseen image, and then animate it with a reference video or audio to generate a talking portrait vid…
Make-A-Voice: Unified Voice Synthesis With Discrete Representation
Rongjie Huang, Chunlei Zhang, Yongqi Wang +7
Various applications of voice synthesis have been developed independently despite the fact that they generate "voice" as output in common. In addition, the majority of voice synthe…
Make-An-Audio 2: Temporal-Enhanced Text-to-Audio Generation
Jiawei Huang, Yi Ren, Rongjie Huang +7
Large diffusion models have been successful in text-to-audio (T2A) synthesis tasks, but they often suffer from common issues such as semantic misalignment and poor temporal consist…
RMSSinger: Realistic-Music-Score based Singing Voice Synthesis
Jinzheng He, Jinglin Liu, Zhenhui Ye +4
We are interested in a challenging task, Realistic-Music-Score based Singing Voice Synthesis (RMS-SVS). RMS-SVS aims to generate high-quality singing voices given realistic music s…
GeneFace++: Generalized and Stable Real-Time Audio-Driven 3D Talking Face Generation
Zhenhui Ye, Jinzheng He, Ziyue Jiang +7
Generating talking person portraits with arbitrary speech audio is a crucial problem in the field of digital human and metaverse. A modern talking face generation method is expecte…
AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head
Rongjie Huang, Mingze Li, Dongchao Yang +10
Large language models (LLMs) have exhibited remarkable capabilities across a variety of domains and tasks, challenging our understanding of learning and cognition. Despite the rece…
GeneFace: Generalized and High-Fidelity Audio-Driven 3D Talking Face Synthesis
Zhenhui Ye, Ziyue Jiang, Yi Ren +3
Generating photo-realistic video portrait with arbitrary speech audio is a crucial problem in film-making and virtual reality. Recently, several works explore the usage of neural r…
Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models
Rongjie Huang, Jiawei Huang, Dongchao Yang +7
Large-scale multimodal generative modeling has created milestones in text-to-image and text-to-video generation. Its application to audio still lags behind for two main reasons: th…
DopplerBAS: Binaural Audio Synthesis Addressing Doppler Effect
Jinglin Liu, Zhenhui Ye, Qian Chen +4
Recently, binaural audio synthesis (BAS) has emerged as a promising research field for its applications in augmented and virtual realities. Binaural audio helps users orient themse…
SyntaSpeech: Syntax-Aware Generative Adversarial Text-to-Speech
Zhenhui Ye, Zhou Zhao, Yi Ren +1
The recent progress in non-autoregressive text-to-speech (NAR-TTS) has made fast and high-quality speech synthesis possible. However, current NAR-TTS models usually use phoneme seq…
Soft Hierarchical Graph Recurrent Networks for Many-Agent Partially Observable Environments
Zhenhui Ye, Xiaohong Jiang, Guanghua Song +1
The recent progress in multi-agent deep reinforcement learning(MADRL) makes it more practical in real-world tasks, but its relatively poor scalability and the partially observable…
Experience Augmentation: Boosting and Accelerating Off-Policy Multi-Agent Reinforcement Learning
Zhenhui Ye, Yining Chen, Guanghua Song +2
Exploration of the high-dimensional state action space is one of the biggest challenges in Reinforcement Learning (RL), especially in multi-agent domain. We present a novel techniq…