38 citations · 220 across the 24 of their papers we have counts for
26 papers
Large Language Model-Powered Evolutionary Code Optimization on a Phylogenetic Tree
Leyi Zhao, Weijie Huang, Yitong Guo +3
Optimizing scientific computing algorithms for modern GPUs is a labor-intensive and iterative process involving repeated code modification, benchmarking, and tuning across complex…
Make Your Actor Talk: Generalizable and High-Fidelity Lip Sync with Motion and Appearance Disentanglement
Runyi Yu, Tianyu He, Ailing Zhang +6
We aim to edit the lip movements in talking video according to the given speech while preserving the personal identity and visual details. The task can be decomposed into two sub-p…
InstructAvatar: Text-Guided Emotion and Motion Control for Avatar Generation
Yuchi Wang, Junliang Guo, Jianhong Bai +5
Recent talking avatar generation models have made strides in achieving realistic and accurate lip synchronization with the audio, but often fall short in controlling and conveying…
NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models
Zeqian Ju, Yuancheng Wang, Kai Shen +16
While recent large-scale text-to-speech (TTS) models have achieved significant progress, they still fall short in speech quality, similarity, and prosody. Considering speech intric…
UniEdit: A Unified Tuning-Free Framework for Video Motion and Appearance Editing
Jianhong Bai, Tianyu He, Yuchi Wang +4
Recent advances in text-guided video editing have showcased promising results in appearance editing (e.g., stylization). However, video motion editing in the temporal dimension (e.…
GAIA: Zero-shot Talking Avatar Generation
Tianyu He, Junliang Guo, Runyi Yu +10
Zero-shot talking avatar generation aims at synthesizing natural talking videos from speech and a single portrait image. Previous methods have relied on domain-specific heuristics…
MusicAgent: An AI Agent for Music Understanding and Generation with Large Language Models
Dingyao Yu, Kaitao Song, Peiling Lu +5
AI-empowered music processing is a diverse field that encompasses dozens of tasks, ranging from generation tasks (e.g., timbre synthesis) to comprehension tasks (e.g., music classi…
PromptTTS 2: Describing and Generating Voices with Text Prompt
Yichong Leng, Zhifang Guo, Kai Shen +12
Speech conveys more information than text, as the same word can be uttered in various voices to convey diverse information. Compared to traditional text-to-speech (TTS) methods rel…
UniAudio: An Audio Foundation Model Toward Universal Audio Generation
Dongchao Yang, Jinchuan Tian, Xu Tan +9
Large Language models (LLM) have demonstrated the capability to handle a variety of generative tasks. This paper presents the UniAudio system, which, unlike prior task-specific app…
EvoPrompt: Connecting LLMs with Evolutionary Algorithms Yields Powerful Prompt Optimizers
Qingyan Guo, Rui Wang, Junliang Guo +6
Large Language Models (LLMs) excel in various tasks, but they rely on carefully crafted prompts that often demand substantial human effort. To automate this process, in this paper,…
Apple Vision Pro for Healthcare: "The Ultimate Display"? -- Entering the Wonderland of Precision Medicine
Jan Egger, Christina Gsaxner, Xiaojun Chen +3
At the Worldwide Developers Conference (WWDC) in June 2023, Apple introduced the Vision Pro. The Vision Pro is a Mixed Reality (MR) headset, more specifically it is a Virtual Reali…
EmoGen: Eliminating Subjective Bias in Emotional Music Generation
Chenfei Kang, Peiling Lu, Botao Yu +4
Music is used to convey emotions, and thus generating emotional music is important in automatic music generation. Previous work on emotional music generation directly uses annotate…
MuseCoco: Generating Symbolic Music from Text
Peiling Lu, Xin Xu, Chenfei Kang +4
Generating music from text descriptions is a user-friendly mode since the text is a relatively easy interface for user engagement. While some approaches utilize texts to control mu…
Deliberate then Generate: Enhanced Prompting Framework for Text Generation
Bei Li, Rui Wang, Junliang Guo +7
Large language models (LLMs) have shown remarkable success across a wide range of natural language generation tasks, where proper prompt designs make great impacts. While existing…
NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers
Kai Shen, Zeqian Ju, Xu Tan +6
Scaling text-to-speech (TTS) to large-scale, multi-speaker, and in-the-wild datasets is important to capture the diversity in human speech such as speaker identities, prosodies, an…
GETMusic: Generating Any Music Tracks with a Unified Representation and Diffusion Framework
Ang Lv, Xu Tan, Peiling Lu +4
Symbolic music generation aims to create musical notes, which can help users compose music, such as generating target instrument tracks based on provided source tracks. In practica…
ResiDual: Transformer with Dual Residual Connections
Shufang Xie, Huishuai Zhang, Junliang Guo +6
Transformer networks have become the preferred architecture for many tasks due to their state-of-the-art performance. However, the optimal way to implement residual connections in…
AUDIT: Audio Editing by Following Instructions with Latent Diffusion Models
Yuancheng Wang, Zeqian Ju, Xu Tan +4
Audio editing is applicable for various purposes, such as adding background sound effects, replacing a musical instrument, and repairing damaged audio. Recently, some diffusion-bas…
DAE-Talker: High Fidelity Speech-Driven Talking Face Generation with Diffusion Autoencoder
Chenpeng Du, Qi Chen, Tianyu He +5
While recent research has made significant progress in speech-driven talking face generation, the quality of the generated video still lags behind that of real recordings. One reas…
HiFace: High-Fidelity 3D Face Reconstruction by Learning Static and Dynamic Details
Zenghao Chai, Tianke Zhang, Tianyu He +7
3D Morphable Models (3DMMs) demonstrate great potential for reconstructing faithful and animatable 3D facial surfaces from a single image. The facial surface is influenced by the c…
A Study on ReLU and Softmax in Transformer
Kai Shen, Junliang Guo, Xu Tan +3
The Transformer architecture consists of self-attention and feed-forward networks (FFNs) which can be viewed as key-value memories according to previous works. However, FFN and tra…
Regeneration Learning: A Learning Paradigm for Data Generation
Xu Tan, Tao Qin, Jiang Bian +2
Machine learning methods for conditional data generation usually build a mapping from source conditional data X to target data Y. The target Y (e.g., text, speech, music, image, vi…
ResGrad: Residual Denoising Diffusion Probabilistic Models for Text to Speech
Zehua Chen, Yihan Wu, Yichong Leng +9
Denoising Diffusion Probabilistic Models (DDPMs) are emerging in text-to-speech (TTS) synthesis because of their strong capability of generating high-fidelity samples. However, the…
Empowering Diffusion Models on the Embedding Space for Text Generation
Zhujin Gao, Junliang Guo, Xu Tan +4
Diffusion models have achieved state-of-the-art synthesis quality on both visual and audio tasks, and recent works further adapt them to textual data by diffusing on the embedding…