34 citations · 150 across the 24 of their papers we have counts for
24 papers
UniAudio: An Audio Foundation Model Toward Universal Audio Generation
Dongchao Yang, Jinchuan Tian, Xu Tan +9
Large Language models (LLM) have demonstrated the capability to handle a variety of generative tasks. This paper presents the UniAudio system, which, unlike prior task-specific app…
VAST: Vivify Your Talking Avatar via Zero-Shot Expressive Facial Style Transfer
Liyang Chen, Zhiyong Wu, Runnan Li +4
Current talking face generation methods mainly focus on speech-lip synchronization. However, insufficient investigation on the facial talking style leads to a lifeless and monotono…
EmoGen: Eliminating Subjective Bias in Emotional Music Generation
Chenfei Kang, Peiling Lu, Botao Yu +4
Music is used to convey emotions, and thus generating emotional music is important in automatic music generation. Previous work on emotional music generation directly uses annotate…
ContextSpeech: Expressive and Efficient Text-to-Speech for Paragraph Reading
Yujia Xiao, Shaofei Zhang, Xi Wang +5
While state-of-the-art Text-to-Speech systems can generate natural speech of very high quality at sentence level, they still meet great challenges in speech generation for paragrap…
MuseCoco: Generating Symbolic Music from Text
Peiling Lu, Xin Xu, Chenfei Kang +4
Generating music from text descriptions is a user-friendly mode since the text is a relatively easy interface for user engagement. While some approaches utilize texts to control mu…
Deliberate then Generate: Enhanced Prompting Framework for Text Generation
Bei Li, Rui Wang, Junliang Guo +7
Large language models (LLMs) have shown remarkable success across a wide range of natural language generation tasks, where proper prompt designs make great impacts. While existing…
DiffusionNER: Boundary Diffusion for Named Entity Recognition
Yongliang Shen, Kaitao Song, Xu Tan +3
In this paper, we propose DiffusionNER, which formulates the named entity recognition task as a boundary-denoising diffusion process and thus generates named entities from noisy sp…
NAS-FM: Neural Architecture Search for Tunable and Interpretable Sound Synthesis based on Frequency Modulation
Zhen Ye, Wei Xue, Xu Tan +2
Developing digital sound synthesizers is crucial to the music industry as it provides a low-cost way to produce high-quality sounds with rich timbres. Existing traditional synthesi…
GETMusic: Generating Any Music Tracks with a Unified Representation and Diffusion Framework
Ang Lv, Xu Tan, Peiling Lu +4
Symbolic music generation aims to create musical notes, which can help users compose music, such as generating target instrument tracks based on provided source tracks. In practica…
CoMoSpeech: One-Step Speech and Singing Voice Synthesis via Consistency Model
Zhen Ye, Wei Xue, Xu Tan +3
Denoising diffusion probabilistic models (DDPMs) have shown promising performance for speech synthesis. However, a large number of iterative steps are required to achieve high samp…
AUDIT: Audio Editing by Following Instructions with Latent Diffusion Models
Yuancheng Wang, Zeqian Ju, Xu Tan +4
Audio editing is applicable for various purposes, such as adding background sound effects, replacing a musical instrument, and repairing damaged audio. Recently, some diffusion-bas…
Improving Few-Shot Learning for Talking Face System with TTS Data Augmentation
Qi Chen, Ziyang Ma, Tao Liu +4
Audio-driven talking face has attracted broad interest from academia and industry recently. However, data acquisition and labeling in audio-driven talking face are labor-intensive…
FoundationTTS: Text-to-Speech for ASR Customization with Generative Language Model
Ruiqing Xue, Yanqing Liu, Lei He +4
Neural text-to-speech (TTS) generally consists of cascaded architecture with separately optimized acoustic model and vocoder, or end-to-end architecture with continuous mel-spectro…
DAE-Talker: High Fidelity Speech-Driven Talking Face Generation with Diffusion Autoencoder
Chenpeng Du, Qi Chen, Tianyu He +5
While recent research has made significant progress in speech-driven talking face generation, the quality of the generated video still lags behind that of real recordings. One reas…
HiFace: High-Fidelity 3D Face Reconstruction by Learning Static and Dynamic Details
Zenghao Chai, Tianke Zhang, Tianyu He +7
3D Morphable Models (3DMMs) demonstrate great potential for reconstructing faithful and animatable 3D facial surfaces from a single image. The facial surface is influenced by the c…
A Study on ReLU and Softmax in Transformer
Kai Shen, Junliang Guo, Xu Tan +3
The Transformer architecture consists of self-attention and feed-forward networks (FFNs) which can be viewed as key-value memories according to previous works. However, FFN and tra…
ERA-Solver: Error-Robust Adams Solver for Fast Sampling of Diffusion Probabilistic Models
Shengming Li, Luping Liu, Runnan Li +1
Though denoising diffusion probabilistic models (DDPMs) have achieved remarkable generation results, the low sampling efficiency of DDPMs still limits further applications. Since D…
ResGrad: Residual Denoising Diffusion Probabilistic Models for Text to Speech
Zehua Chen, Yihan Wu, Yichong Leng +9
Denoising Diffusion Probabilistic Models (DDPMs) are emerging in text-to-speech (TTS) synthesis because of their strong capability of generating high-fidelity samples. However, the…
Empowering Diffusion Models on the Embedding Space for Text Generation
Zhujin Gao, Junliang Guo, Xu Tan +4
Diffusion models have achieved state-of-the-art synthesis quality on both visual and audio tasks, and recent works further adapt them to textual data by diffusing on the embedding…
Memories are One-to-Many Mapping Alleviators in Talking Face Generation
Anni Tang, Tianyu He, Xu Tan +2
Talking face generation aims at generating photo-realistic video portraits of a target person driven by input audio. Due to its nature of one-to-many mapping from the input audio t…
Mask the Correct Tokens: An Embarrassingly Simple Approach for Error Correction
Kai Shen, Yichong Leng, Xu Tan +4
Text error correction aims to correct the errors in text sequences such as those typed by humans or generated by speech recognition models. Previous error correction methods usuall…
PromptTTS: Controllable Text-to-Speech with Text Descriptions
Zhifang Guo, Yichong Leng, Yihan Wu +2
Using a text description as prompt to guide the generation of text or images (e.g., GPT-3 or DALLE-2) has drawn wide attention recently. Beyond text and image generation, in this w…
VideoDubber: Machine Translation with Speech-Aware Length Control for Video Dubbing
Yihan Wu, Junliang Guo, Xu Tan +7
Video dubbing aims to translate the original speech in a film or television program into the speech in a target language, which can be achieved with a cascaded system consisting of…
StableFace: Analyzing and Improving Motion Stability for Talking Face Generation
Jun Ling, Xu Tan, Liyang Chen +4
While previous speech-driven talking face generation methods have made significant progress in improving the visual quality and lip-sync quality of the synthesized videos, they pay…