5 citations · 22 across the 25 of their papers we have counts for
25 papers
UniSH: Unifying Scene and Human Reconstruction in a Feed-Forward Pass
Mengfei Li, Peng Li, Zheng Zhang +9
We present UniSH, a unified, feed-forward framework for joint metric-scale 3D scene and human reconstruction. A key challenge in this domain is the scarcity of large-scale, annotat…
VividListener: Expressive and Controllable Listener Dynamics Modeling for Multi-Modal Responsive Interaction
Shiying Li, Xingqun Qi, Bingkun Yang +6
Generating responsive listener head dynamics with nuanced emotions and expressive reactions is crucial for practical dialogue modeling in various virtual avatar animations. Previou…
CoGesture: Towards Coherent Concurrent Co-speech 3D Gesture Generation with Interactive Diffusion
Xingqun Qi, Yatian Wang, Hengyuan Zhang +6
Generating gestures from human speech has gained tremendous progress in animating virtual avatars. While the existing methods enable synthesizing gestures cooperated by individual…
VFX Creator: Animated Visual Effect Generation with Controllable Diffusion Transformer
Xinyu Liu, Ailing Zeng, Wei Xue +4
Crafting magic and illusions is one of the most thrilling aspects of filmmaking, with visual effects (VFX) serving as the powerhouse behind unforgettable cinematic experiences. Whi…
Diffusion as Shader: 3D-aware Video Diffusion for Versatile Video Generation Control
Zekai Gu, Rui Yan, Jiahao Lu +9
Diffusion models have demonstrated impressive performance in generating high-quality videos from text prompts or images. However, precise control over the video generation process,…
Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Model
Zhen Ye, Peiwen Sun, Jiahe Lei +9
Recent advancements in audio generation have been significantly propelled by the capabilities of Large Language Models (LLMs). The existing research on audio LLM has primarily focu…
Importance Weighting Can Help Large Language Models Self-Improve
Chunyang Jiang, Chi-min Chan, Wei Xue +2
Large language models (LLMs) have shown remarkable capability in numerous tasks and applications. However, fine-tuning LLMs using high-quality datasets under external supervision r…
STBLLM: Breaking the 1-Bit Barrier with Structured Binary LLMs
Peijie Dong, Lujun Li, Yuedong Zhong +8
In this paper, we present the first structural binarization method for LLM compression to less than 1-bit precision. Although LLMs have achieved remarkable performance, their memor…
MMTrail: A Multimodal Trailer Video Dataset with Language and Music Descriptions
Xiaowei Chi, Yatian Wang, Aosong Cheng +16
Massive multi-modality datasets play a significant role in facilitating the success of large video-language models. However, current video-language datasets primarily provide text…
LLMs Meet Multimodal Generation and Editing: A Survey
Yingqing He, Zhaoyang Liu, Jingye Chen +13
With the recent advancement in large language models (LLMs), there is a growing interest in combining LLMs with multimodal learning. Previous surveys of multimodal large language m…
Multi-View Large Reconstruction Model via Geometry-Aware Positional Encoding and Attention
Mengfei Li, Xiaoxiao Long, Yixun Liang +6
Despite recent advancements in the Large Reconstruction Model (LRM) demonstrating impressive results, when extending its input from single image to multiple images, it exhibits ine…
VidMuse: A Simple Video-to-Music Generation Framework with Long-Short-Term Modeling
Zeyue Tian, Zhaoyang Liu, Ruibin Yuan +6
In this work, we systematically study music generation conditioned solely on the video. First, we present a large-scale dataset comprising 360K video-music pairs, including various…
FastSAG: Towards Fast Non-Autoregressive Singing Accompaniment Generation
Jianyi Chen, Wei Xue, Xu Tan +3
Singing Accompaniment Generation (SAG), which generates instrumental music to accompany input vocals, is crucial to developing human-AI symbiotic art creation systems. The state-of…
CoCoGesture: Toward Coherent Co-speech 3D Gesture Generation in the Wild
Xingqun Qi, Hengyuan Zhang, Yatian Wang +10
Deriving co-speech 3D gestures has seen tremendous progress in virtual avatar animation. Yet, the existing methods often produce stiff and unreasonable gestures with unseen human s…
Era3D: High-Resolution Multiview Diffusion using Efficient Row-wise Attention
Peng Li, Yuan Liu, Xiaoxiao Long +10
In this paper, we introduce Era3D, a novel multiview diffusion method that generates high-resolution multiview images from a single-view image. Despite significant advancements in…
FlashSpeech: Efficient Zero-Shot Speech Synthesis
Zhen Ye, Zeqian Ju, Haohe Liu +10
Recent progress in large-scale zero-shot speech synthesis has been significantly advanced by language models and diffusion models. However, the generation process of both methods i…
ChatMusician: Understanding and Generating Music Intrinsically with LLM
Ruibin Yuan, Hanfeng Lin, Yi Wang +32
While Large Language Models (LLMs) demonstrate impressive capabilities in text generation, we find that their ability has yet to be generalized to music, humanity's creative langua…
CoMoSVC: Consistency Model-based Singing Voice Conversion
Yiwen Lu, Zhen Ye, Wei Xue +3
The diffusion-based Singing Voice Conversion (SVC) methods have achieved remarkable performances, producing natural audios with high similarity to the target timbre. However, the i…
MChat: Empowering VLM for Multimodal LLM Interleaved Text-Image Generation
Xiaowei Chi, Junbo Qi, Rongyu Zhang +3
While current LLM chatbots like GPT-4V bridge the gap between human instructions and visual representations to enable text-image generations, they still lack efficient alignment me…
Weakly-Supervised Emotion Transition Learning for Diverse 3D Co-speech Gesture Generation
Xingqun Qi, Jiahao Pan, Peng Li +8
Generating vivid and emotional 3D co-speech gestures is crucial for virtual avatar animation in human-machine interaction applications. While the existing methods enable generating…
Continual Learning with Dirichlet Generative-based Rehearsal
Min Zeng, Wei Xue, Qifeng Liu +1
Recent advancements in data-driven task-oriented dialogue systems (ToDs) struggle with incremental learning due to computational constraints and time-consuming issues. Continual Le…
NAS-FM: Neural Architecture Search for Tunable and Interpretable Sound Synthesis based on Frequency Modulation
Zhen Ye, Wei Xue, Xu Tan +2
Developing digital sound synthesizers is crucial to the music industry as it provides a low-cost way to produce high-quality sounds with rich timbres. Existing traditional synthesi…
Learn to Sing by Listening: Building Controllable Virtual Singer by Unsupervised Learning from Voice Recordings
Wei Xue, Yiwen Wang, Qifeng Liu +1
The virtual world is being established in which digital humans are created indistinguishable from real humans. Producing their audio-related capabilities is crucial since voice con…
CoMoSpeech: One-Step Speech and Singing Voice Synthesis via Consistency Model
Zhen Ye, Wei Xue, Xu Tan +3
Denoising diffusion probabilistic models (DDPMs) have shown promising performance for speech synthesis. However, a large number of iterative steps are required to achieve high samp…