334 citations · 883 across the 97 of their papers we have counts for
12 papers · 1 filter
Characterizing Motion Encoding in Video Diffusion Timesteps
Vatsal Baherwani, Yixuan Ren, Abhinav Shrivastava
Text-to-video diffusion models synthesize temporal motion and spatial appearance through iterative denoising, yet how motion is encoded across timesteps remains poorly understood.…
Growing Visual Generative Capacity for Pre-Trained MLLMs
Hanyu Wang, Jiaming Han, Ziyan Yang +6
Multimodal large language models (MLLMs) extend the success of language models to visual understanding, and recent efforts have sought to build unified MLLMs that support both unde…
NeRV-Diffusion: Diffuse Implicit Neural Representations for Video Synthesis
Yixuan Ren, Hanyu Wang, Hao Chen +2
We present NeRV-Diffusion, an implicit latent video diffusion model that synthesizes videos via generating neural network weights. The generated weights can be rearranged as the pa…
Trokens: Semantic-Aware Relational Trajectory Tokens for Few-Shot Action Recognition
Pulkit Kumar, Shuaiyi Huang, Matthew Walmer +2
Video understanding requires effective modeling of both motion and appearance information, particularly for few-shot action recognition. While recent advances in point tracking hav…
Towards Multimodal Understanding via Stable Diffusion as a Task-Aware Feature Extractor
Vatsal Agarwal, Matthew Gwilliam, Gefen Kohavi +3
Recent advances in multimodal large language models (MLLMs) have enabled image-based question-answering capabilities. However, a key limitation is the use of CLIP as the visual enc…
How to Design and Train Your Implicit Neural Representation for Video Compression
Matthew Gwilliam, Roy Zhang, Namitha Padmanabhan +2
Implicit neural representation (INR) methods for video compression have recently achieved visual quality and compression ratios that are competitive with traditional pipelines. How…
SIEDD: Shared-Implicit Encoder with Discrete Decoders
Vikram Rangarajan, Shishira Maiya, Max Ehrlich +1
Implicit Neural Representations (INRs) offer exceptional fidelity for video compression by learning per-video optimized functions, but their adoption is crippled by impractically s…
Evolutionary Caching to Accelerate Your Off-the-Shelf Diffusion Model
Anirud Aggarwal, Abhinav Shrivastava, Matthew Gwilliam
Diffusion-based image generation models excel at producing high-quality synthetic content, but suffer from slow and computationally expensive inference. Prior work has attempted to…
TREND: Tri-teaching for Robust Preference-based Reinforcement Learning with Demonstrations
Shuaiyi Huang, Mara Levy, Anubhav Gupta +3
Preference feedback collected by human or VLM annotators is often noisy, presenting a significant challenge for preference-based reinforcement learning that relies on accurate pref…
Utilization of Neighbor Information for Image Classification with Different Levels of Supervision
Gihan Jayatilaka, Abhinav Shrivastava, Matthew Gwilliam
We propose to bridge the gap between semi-supervised and unsupervised image recognition with a flexible method that performs well for both generalized category discovery (GCD) and…
Mitigating Hallucinations in Diffusion Models through Adaptive Attention Modulation
Trevine Oorloff, Yaser Yacoob, Abhinav Shrivastava
Diffusion models, while increasingly adept at generating realistic images, are notably hindered by hallucinations -- unrealistic or incorrect features inconsistent with the trained…
Accelerate High-Quality Diffusion Models with Inner Loop Feedback
Matthew Gwilliam, Han Cai, Di Wu +2
We propose Inner Loop Feedback (ILF), a novel approach to accelerate diffusion models' inference. ILF trains a lightweight module to predict future features in the denoising proces…