290 citations · 981 across the 34 of their papers we have counts for
34 papers
Vidu S2: Real-Time Interactive, Editable, and Spatial Video Generation
Jintao Zhang, Kai Jiang, Jintao Chen +32
We present Vidu S2, which comprises Vidu S2-Avatar, a real-time interactive digital-character model, and Vidu S2-Editing, a real-time video editing model. Moreover, we explore the…
Motus2: A Self-Evolving General World Model for Dexterous Manipulation
Hongzhe Bi, Zihao Zhou, Yihang Tang +16
General embodied agents should perceive, predict, act, evaluate, and improve within a unified system. World models have shown great promise in building such agents, yet existing mo…
Vidu S1: A Real-Time Interactive Video Generation Model
Jintao Zhang, Kai Jiang, Jintao Chen +24
We introduce Vidu S1, a real-time interactive video generation model supporting voice control of digital characters. Users can control video generation content at any moment throug…
minWM: A Full-Stack Open-Source Framework for Real-Time Interactive Video World Models
Min Zhao, Hongzhou Zhu, Bokai Yan +13
Recent video diffusion foundation models have achieved remarkable progress in high-quality video generation, yet turning them into real-time interactive video world models remains…
Motubrain: An Advanced World Action Model for Robot Control
Motubrain Team, Chendong Xiang, Fan Bao +17
Vision-Language-Action (VLA) models generalize semantically well but often lack fine-grained modeling of world dynamics. We present Motubrain, a unified World Action Model that joi…
DiffGAP: A Lightweight Diffusion Module in Contrastive Space for Bridging Cross-Model Gap
Shentong Mo, Zehua Chen, Fan Bao +1
Recent works in cross-modal understanding and generation, notably through models like CLAP (Contrastive Language-Audio Pretraining) and CAVP (Contrastive Audio-Visual Pretraining),…
Elucidating the Preconditioning in Consistency Distillation
Kaiwen Zheng, Guande He, Jianfei Chen +2
Consistency distillation is a prevalent way for accelerating diffusion models adopted in consistency (trajectory) models, in which a student model is trained to traverse backward o…
Bridge-SR: Schrödinger Bridge for Efficient SR
Chang Li, Zehua Chen, Fan Bao +1
Speech super-resolution (SR), which generates a waveform at a higher sampling rate from its low-resolution version, is a long-standing critical task in speech restoration. Previous…
Consistency Diffusion Bridge Models
Guande He, Kaiwen Zheng, Jianfei Chen +2
Diffusion models (DMs) have become the dominant paradigm of generative modeling in a variety of domains by learning stochastic processes from noise to data. Recently, diffusion den…
Vidu: a Highly Consistent, Dynamic and Skilled Text-to-Video Generator with Diffusion Models
Fan Bao, Chendong Xiang, Gang Yue +7
We introduce Vidu, a high-performance text-to-video generator that is capable of producing 1080p videos up to 16 seconds in a single generation. Vidu is a diffusion model with U-Vi…
Diffusion Bridge Implicit Models
Kaiwen Zheng, Guande He, Jianfei Chen +2
Denoising diffusion bridge models (DDBMs) are a powerful variant of diffusion models for interpolating between two arbitrary paired distributions given as endpoints. Despite their…
Gaussian Mixture Solvers for Diffusion Models
Hanzhong Guo, Cheng Lu, Fan Bao +4
Recently, diffusion models have achieved great success in generative tasks. Sampling from diffusion models is equivalent to solving the reverse diffusion stochastic differential eq…
ControlVideo: Conditional Control for One-shot Text-driven Video Editing and Beyond
Min Zhao, Rongzhen Wang, Fan Bao +2
This paper presents \emph{ControlVideo} for text-driven video editing -- generating a video that aligns with a given text while preserving the structure of the source video. Buildi…
ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation
Zhengyi Wang, Cheng Lu, Yikai Wang +4
Score distillation sampling (SDS) has shown great promise in text-to-3D generation by distilling pretrained large-scale text-to-image diffusion models, but suffers from over-satura…
A Closer Look at Parameter-Efficient Tuning in Diffusion Models
Chendong Xiang, Fan Bao, Chongxuan Li +2
Large-scale diffusion models like Stable Diffusion are powerful and find various real-world applications while customizing such models by fine-tuning is both memory and time ineffi…
One Transformer Fits All Distributions in Multi-Modal Diffusion at Scale
Fan Bao, Shen Nie, Kaiwen Xue +7
This paper proposes a unified diffusion framework (dubbed UniDiffuser) to fit all distributions relevant to a set of multi-modal data in one model. Our key insight is -- learning d…
Diffusion Models and Semi-Supervised Learners Benefit Mutually with Few Labels
Zebin You, Yong Zhong, Fan Bao +3
In an effort to further advance semi-supervised generative and classification tasks, we propose a simple yet effective training strategy called dual pseudo training (DPT), built up…
Revisiting Discriminative vs. Generative Classifiers: Theory and Implications
Chenyu Zheng, Guoqiang Wu, Fan Bao +3
A large-scale deep model pre-trained on massive labeled or unlabeled data transfers well to downstream tasks. Linear evaluation freezes parameters in the pre-trained model and trai…
Why Are Conditional Generative Models Better Than Unconditional Ones?
Fan Bao, Chongxuan Li, Jiacheng Sun +1
Extensive empirical evidence demonstrates that conditional generative models are easier to train and perform better than unconditional ones by exploiting the labels of data. So do…
DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models
Cheng Lu, Yuhao Zhou, Fan Bao +3
Diffusion probabilistic models (DPMs) have achieved impressive success in high-resolution image synthesis, especially in recent large-scale text-to-image generation applications. A…
Equivariant Energy-Guided SDE for Inverse Molecular Design
Fan Bao, Min Zhao, Zhongkai Hao +3
Inverse molecular design is critical in material science and drug discovery, where the generated molecules should satisfy certain desirable properties. In this paper, we propose eq…
All are Worth Words: A ViT Backbone for Diffusion Models
Fan Bao, Shen Nie, Kaiwen Xue +4
Vision transformers (ViT) have shown promise in various vision tasks while the U-Net based on a convolutional neural network (CNN) remains dominant in diffusion models. We design a…
Deep Generative Modeling on Limited Data with Regularization by Nontransferable Pre-trained Models
Yong Zhong, Hongtao Liu, Xiaodong Liu +3
Deep generative models (DGMs) are data-eager because learning a complex model on limited data suffers from a large variance and easily overfits. Inspired by the classical perspecti…
EGSDE: Unpaired Image-to-Image Translation via Energy-Guided Stochastic Differential Equations
Min Zhao, Fan Bao, Chongxuan Li +1
Score-based diffusion models (SBDMs) have achieved the SOTA FID results in unpaired image-to-image translation (I2I). However, we notice that existing methods totally ignore the tr…