47 citations · 269 across the 36 of their papers we have counts for
15 papers · 1 filter
C2G2: Controllable Co-speech Gesture Generation with Latent Diffusion Model
Longbin Ji, Pengfei Wei, Yi Ren +3
Co-speech gesture generation is crucial for automatic digital avatar animation. However, existing methods suffer from issues such as unstable training and temporal inconsistency, p…
Ada-TTA: Towards Adaptive High-Quality Text-to-Talking Avatar Synthesis
Zhenhui Ye, Ziyue Jiang, Yi Ren +5
We are interested in a novel task, namely low-resource text-to-talking avatar. Given only a few-minute-long talking person video with the audio track as the training data and arbit…
Mega-TTS 2: Boosting Prompting Mechanisms for Zero-Shot Speech Synthesis
Ziyue Jiang, Jinglin Liu, Yi Ren +10
Zero-shot text-to-speech (TTS) aims to synthesize voices with unseen speech prompts, which significantly reduces the data and computation requirements for voice cloning by skipping…
Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias
Ziyue Jiang, Yi Ren, Zhenhui Ye +9
Scaling text-to-speech to a large and wild dataset has been proven to be highly effective in achieving timbre and speech style generalization, particularly in zero-shot TTS. Howeve…
Make-An-Audio 2: Temporal-Enhanced Text-to-Audio Generation
Jiawei Huang, Yi Ren, Rongjie Huang +7
Large diffusion models have been successful in text-to-audio (T2A) synthesis tasks, but they often suffer from common issues such as semantic misalignment and poor temporal consist…
AV-TranSpeech: Audio-Visual Robust Speech-to-Speech Translation
Rongjie Huang, Huadai Liu, Xize Cheng +8
Direct speech-to-speech translation (S2ST) aims to convert speech from one language into another, and has demonstrated significant progress to date. Despite the recent success, cur…
AlignSTS: Speech-to-Singing Conversion via Cross-Modal Alignment
Ruiqi Li, Rongjie Huang, Lichao Zhang +2
The speech-to-singing (STS) voice conversion task aims to generate singing samples corresponding to speech recordings while facing a major challenge: the alignment between the targ…
CLAPSpeech: Learning Prosody from Text Context with Contrastive Language-Audio Pre-training
Zhenhui Ye, Rongjie Huang, Yi Ren +5
Improving text representation has attracted much attention to achieve expressive text-to-speech (TTS). However, existing works only implicitly learn the prosody with masked token r…
RMSSinger: Realistic-Music-Score based Singing Voice Synthesis
Jinzheng He, Jinglin Liu, Zhenhui Ye +4
We are interested in a challenging task, Realistic-Music-Score based Singing Voice Synthesis (RMS-SVS). RMS-SVS aims to generate high-quality singing voices given realistic music s…
GeneFace++: Generalized and Stable Real-Time Audio-Driven 3D Talking Face Generation
Zhenhui Ye, Jinzheng He, Ziyue Jiang +7
Generating talking person portraits with arbitrary speech audio is a crucial problem in the field of digital human and metaverse. A modern talking face generation method is expecte…
AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head
Rongjie Huang, Mingze Li, Dongchao Yang +10
Large language models (LLMs) have exhibited remarkable capabilities across a variety of domains and tasks, challenging our understanding of learning and cognition. Despite the rece…
MUG: A General Meeting Understanding and Generation Benchmark
Qinglin Zhang, Chong Deng, Jiaqing Liu +7
Listening to long video/audio recordings from video conferencing and online courses for acquiring information is extremely inefficient. Even after ASR systems transcribe recordings…
Overview of the ICASSP 2023 General Meeting Understanding and Generation Challenge (MUG)
Qinglin Zhang, Chong Deng, Jiaqing Liu +7
ICASSP2023 General Meeting Understanding and Generation Challenge (MUG) focuses on prompting a wide range of spoken language processing (SLP) research on meeting transcripts, as SL…
GeneFace: Generalized and High-Fidelity Audio-Driven 3D Talking Face Synthesis
Zhenhui Ye, Ziyue Jiang, Yi Ren +3
Generating photo-realistic video portrait with arbitrary speech audio is a crucial problem in film-making and virtual reality. Recently, several works explore the usage of neural r…
Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models
Rongjie Huang, Jiawei Huang, Dongchao Yang +7
Large-scale multimodal generative modeling has created milestones in text-to-image and text-to-video generation. Its application to audio still lags behind for two main reasons: th…