34 citations · 121 across the 23 of their papers we have counts for
23 papers
C2G2: Controllable Co-speech Gesture Generation with Latent Diffusion Model
Longbin Ji, Pengfei Wei, Yi Ren +3
Co-speech gesture generation is crucial for automatic digital avatar animation. However, existing methods suffer from issues such as unstable training and temporal inconsistency, p…
Ada-TTA: Towards Adaptive High-Quality Text-to-Talking Avatar Synthesis
Zhenhui Ye, Ziyue Jiang, Yi Ren +5
We are interested in a novel task, namely low-resource text-to-talking avatar. Given only a few-minute-long talking person video with the audio track as the training data and arbit…
Mega-TTS 2: Boosting Prompting Mechanisms for Zero-Shot Speech Synthesis
Ziyue Jiang, Jinglin Liu, Yi Ren +10
Zero-shot text-to-speech (TTS) aims to synthesize voices with unseen speech prompts, which significantly reduces the data and computation requirements for voice cloning by skipping…
GenerTTS: Pronunciation Disentanglement for Timbre and Style Generalization in Cross-Lingual Text-to-Speech
Yahuan Cong, Haoyu Zhang, Haopeng Lin +5
Cross-lingual timbre and style generalizable text-to-speech (TTS) aims to synthesize speech with a specific reference timbre or style that is never trained in the target language.…
Towards Building Voice-based Conversational Recommender Systems: Datasets, Potential Solutions, and Prospects
Xinghua Qu, Hongyang Liu, Zhu Sun +4
Conversational recommender systems (CRSs) have become crucial emerging research topics in the field of RSs, thanks to their natural advantages of explicitly acquiring user preferen…
Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias
Ziyue Jiang, Yi Ren, Zhenhui Ye +9
Scaling text-to-speech to a large and wild dataset has been proven to be highly effective in achieving timbre and speech style generalization, particularly in zero-shot TTS. Howeve…
Make-An-Audio 2: Temporal-Enhanced Text-to-Audio Generation
Jiawei Huang, Yi Ren, Rongjie Huang +7
Large diffusion models have been successful in text-to-audio (T2A) synthesis tasks, but they often suffer from common issues such as semantic misalignment and poor temporal consist…
CLAPSpeech: Learning Prosody from Text Context with Contrastive Language-Audio Pre-training
Zhenhui Ye, Rongjie Huang, Yi Ren +5
Improving text representation has attracted much attention to achieve expressive text-to-speech (TTS). However, existing works only implicitly learn the prosody with masked token r…
StyleS2ST: Zero-shot Style Transfer for Direct Speech-to-speech Translation
Kun Song, Yi Ren, Yi Lei +5
Direct speech-to-speech translation (S2ST) has gradually become popular as it has many advantages compared with cascade S2ST. However, current research mainly focuses on the accura…
GeneFace++: Generalized and Stable Real-Time Audio-Driven 3D Talking Face Generation
Zhenhui Ye, Jinzheng He, Ziyue Jiang +7
Generating talking person portraits with arbitrary speech audio is a crucial problem in the field of digital human and metaverse. A modern talking face generation method is expecte…
LiteG2P: A fast, light and high accuracy model for grapheme-to-phoneme conversion
Chunfeng Wang, Peisong Huang, Yuxiang Zou +4
As a key component of automated speech recognition (ASR) and the front-end in text-to-speech (TTS), grapheme-to-phoneme (G2P) plays the role of converting letters to their correspo…
Direct Speech-to-speech Translation without Textual Annotation using Bottleneck Features
Junhui Zhang, Junjie Pan, Xiang Yin +1
Speech-to-speech translation directly translates a speech utterance to another between different languages, and has great potential in tasks such as simultaneous interpretation. St…
A Novel Chinese Dialect TTS Frontend with Non-Autoregressive Neural Machine Translation
Junhui Zhang, Wudi Bao, Junjie Pan +2
Chinese dialects are different variations of Chinese and can be considered as different languages in the same language family with Mandarin. Though they all use Chinese characters,…
Towards Realistic Visual Dubbing with Heterogeneous Sources
Tianyi Xie, Liucheng Liao, Cheng Bi +7
The task of few-shot visual dubbing focuses on synchronizing the lip movements with arbitrary speech input for any talking head video. Albeit moderate improvements in current appro…
Towards Using Clothes Style Transfer for Scenario-aware Person Video Generation
Jingning Xu, Benlai Tang, Mingjie Wang +4
Clothes style transfer for person video generation is a challenging task, due to drastic variations of intra-person appearance and video scenarios. To tackle this problem, most rec…
Cross-speaker Emotion Transfer Based on Speaker Condition Layer Normalization and Semi-Supervised Training in Text-To-Speech
Pengfei Wu, Junjie Pan, Chenchang Xu +4
In expressive speech synthesis, there are high requirements for emotion interpretation. However, it is time-consuming to acquire emotional audio corpus for arbitrary speakers due t…
Towards High-fidelity Singing Voice Conversion with Acoustic Reference and Contrastive Predictive Coding
Chao Wang, Zhonghao Li, Benlai Tang +4
Recently, phonetic posteriorgrams (PPGs) based methods have been quite popular in non-parallel singing voice conversion systems. However, due to the lack of acoustic information in…
PPG-based singing voice conversion with adversarial representation learning
Zhonghao Li, Benlai Tang, Xiang Yin +4
Singing voice conversion (SVC) aims to convert the voice of one singer to that of other singers while keeping the singing content and melody. On top of recent voice conversion work…
Xiaomingbot: A Multilingual Robot News Reporter
Runxin Xu, Jun Cao, Mingxuan Wang +10
This paper proposes the building of Xiaomingbot, an intelligent, multilingual and multimodal software robot equipped with four integral capabilities: news generation, news translat…
Improving Accent Conversion with Reference Encoder and End-To-End Text-To-Speech
Wenjie Li, Benlai Tang, Xiang Yin +6
Accent conversion (AC) transforms a non-native speaker's accent into a native accent while maintaining the speaker's voice timbre. In this paper, we propose approaches to improving…
ByteSing: A Chinese Singing Voice Synthesis System Using Duration Allocated Encoder-Decoder Acoustic Models and WaveRNN Vocoders
Yu Gu, Xiang Yin, Yonghui Rao +6
This paper presents ByteSing, a Chinese singing voice synthesis (SVS) system based on duration allocated Tacotron-like acoustic models and WaveRNN neural vocoders. Different from t…
A unified sequence-to-sequence front-end model for Mandarin text-to-speech synthesis
Junjie Pan, Xiang Yin, Zhiling Zhang +4
In Mandarin text-to-speech (TTS) system, the front-end text processing module significantly influences the intelligibility and naturalness of synthesized speech. Building a typical…
A hybrid text normalization system using multi-head self-attention for mandarin
Junhui Zhang, Junjie Pan, Xiang Yin +5
In this paper, we propose a hybrid text normalization system using multi-head self-attention. The system combines the advantages of a rule-based model and a neural model for text p…