112 citations · 552 across the 46 of their papers we have counts for
21 papers · 1 filter
Flex-Forcing: Towards a Unified Autoregressive and Bidirectional Video Diffusion Model
Xinyin Ma, Julius Berner, Chao Liu +3
Recent progress in large-scale generative models has substantially advanced video generation, yet existing methods remain constrained by a rigid inference paradigm. Bidirectional d…
Mode Seeking meets Mean Seeking for Fast Long Video Generation
Shengqu Cai, Weili Nie, Chao Liu +8
Scaling video generation from seconds to minutes faces a critical bottleneck: while short-video data is abundant and high-fidelity, coherent long-form data is scarce and limited to…
Transition Matching Distillation for Fast Video Generation
Weili Nie, Julius Berner, Nanye Ma +3
Large video diffusion and flow models have achieved remarkable success in high-quality video generation, but their use in real-time interactive applications remains limited due to…
PixelDiT: Pixel Diffusion Transformers for Image Generation
Yongsheng Yu, Wei Xiong, Weili Nie +3
Latent-space modeling has been the standard for Diffusion Transformers (DiTs). However, it relies on a two-stage pipeline where the pretrained autoencoder introduces lossy reconstr…
BlobGEN-Vid: Compositional Text-to-Video Generation with Blob Video Representations
Weixi Feng, Chao Liu, Sifei Liu +3
Existing video generation models struggle to follow complex text prompts and synthesize multiple objects, raising the need for additional grounding input for improved controllabili…
Warped Diffusion: Solving Video Inverse Problems with Image Diffusion Models
Giannis Daras, Weili Nie, Karsten Kreis +4
Using image models naively for solving inverse video problems often suffers from flickering, texture-sticking, and temporal inconsistency in generated videos. To tackle these probl…
CamCo: Camera-Controllable 3D-Consistent Image-to-Video Generation
Dejia Xu, Weili Nie, Chao Liu +4
Recently video diffusion models have emerged as expressive generative tools for high-quality video content creation readily available to general users. However, these models often…
DiffUHaul: A Training-Free Method for Object Dragging in Images
Omri Avrahami, Rinon Gal, Gal Chechik +4
Text-to-image diffusion models have proven effective for solving many image editing tasks. However, the seemingly straightforward task of seamlessly relocating objects within a sce…
Compositional Text-to-Image Generation with Dense Blob Representations
Weili Nie, Sifei Liu, Morteza Mardani +3
Existing text-to-image models struggle to follow complex text prompts, raising the need for extra grounding inputs for better controllability. In this work, we propose to decompose…
Efficient Video Diffusion Models via Content-Frame Motion-Latent Decomposition
Sihyun Yu, Weili Nie, De-An Huang +3
Video diffusion models have recently made great progress in generation quality, but are still limited by the high memory and computational requirements. This is because current vid…
T-Stitch: Accelerating Sampling in Pre-Trained Diffusion Models with Trajectory Stitching
Zizheng Pan, Bohan Zhuang, De-An Huang +5
Sampling from diffusion probabilistic models (DPMs) is often expensive for high-quality image generation and typically requires many steps with a large model. In this paper, we int…
Fast Training of Diffusion Models with Masked Transformers
Hongkai Zheng, Weili Nie, Arash Vahdat +1
We propose an efficient approach to train large diffusion models with masked transformers. While masked transformers have been extensively explored for representation learning, the…
ISB: Image-to-Image Schrödinger Bridge
Guan-Horng Liu, Arash Vahdat, De-An Huang +3
We propose Image-to-Image Schrödinger Bridge (ISB), a new class of conditional diffusion models that directly learn the nonlinear diffusion processes between two given distribu…
Re-ViLM: Retrieval-Augmented Visual Language Model for Zero and Few-Shot Image Captioning
Zhuolin Yang, Wei Ping, Zihan Liu +13
Augmenting pretrained language models (LMs) with a vision encoder (e.g., Flamingo) has obtained the state-of-the-art results in image-to-text generation. However, these models stor…
Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models
Manli Shu, Weili Nie, De-An Huang +4
Pre-trained vision-language models (e.g., CLIP) have shown promising zero-shot generalization in many downstream tasks with properly designed text prompts. Instead of relying on ha…
PointDP: Diffusion-driven Purification against Adversarial Attacks on 3D Point Cloud Recognition
Jiachen Sun, Weili Nie, Zhiding Yu +2
3D Point cloud is becoming a critical data representation in many real-world applications like autonomous driving, robotics, and medical imaging. Although the success of deep learn…
Bongard-HOI: Benchmarking Few-Shot Visual Reasoning for Human-Object Interactions
Huaizu Jiang, Xiaojian Ma, Weili Nie +4
A significant gap remains between today's visual pattern recognition models and human-level visual cognition especially when it comes to few-shot learning and compositional reasoni…
RelViT: Concept-guided Vision Transformer for Visual Relational Reasoning
Xiaojian Ma, Weili Nie, Zhiding Yu +5
Reasoning about visual relationships is central to how humans interpret the visual world. This task remains challenging for current deep learning algorithms since it requires addre…
Controllable and Compositional Generation with Latent-Space Energy-Based Models
Weili Nie, Arash Vahdat, Anima Anandkumar
Controllable generation is one of the key requirements for successful adoption of deep generative models in real-world applications, but it still remains as a great challenge. In p…
Semi-Supervised StyleGAN for Disentanglement Learning
Weili Nie, Tero Karras, Animesh Garg +4
Disentanglement learning is crucial for obtaining disentangled representations and controllable generation. Current disentanglement methods face several inherent limitations: diffi…
A Theoretical Explanation for Perplexing Behaviors of Backpropagation-based Visualizations
Weili Nie, Yang Zhang, Ankit Patel
Backpropagation-based visualizations have been proposed to interpret convolutional neural networks (CNNs), however a theory is missing to justify their behaviors: Guided backpropag…