334 citations · 883 across the 97 of their papers we have counts for
110 papers · 1 filter
TeCoNeRV: Leveraging Temporal Coherence for Compressible Neural Representations for Videos
Namitha Padmanabhan, Matthew Gwilliam, Abhinav Shrivastava
Implicit Neural Representations (INRs) have recently demonstrated impressive performance for video compression. However, since a separate INR must be overfit for each video, scalin…
Towards Understanding Best Practices for Quantization of Vision-Language Models
Gautom Das, Vincent La, Ethan Lau +2
Large language models (LLMs) deliver impressive results for a variety of tasks, but state-of-the-art systems require fast GPUs with large amounts of memory. To reduce both the memo…
UPLiFT: Efficient Pixel-Dense Feature Upsampling with Local Attenders
Matthew Walmer, Saksham Suri, Anirud Aggarwal +1
The space of task-agnostic feature upsampling has emerged as a promising area of research to efficiently create denser features from pre-trained visual backbones. These methods act…
Characterizing Motion Encoding in Video Diffusion Timesteps
Vatsal Baherwani, Yixuan Ren, Abhinav Shrivastava
Text-to-video diffusion models synthesize temporal motion and spatial appearance through iterative denoising, yet how motion is encoded across timesteps remains poorly understood.…
Growing Visual Generative Capacity for Pre-Trained MLLMs
Hanyu Wang, Jiaming Han, Ziyan Yang +6
Multimodal large language models (MLLMs) extend the success of language models to visual understanding, and recent efforts have sought to build unified MLLMs that support both unde…
NeRV-Diffusion: Diffuse Implicit Neural Representations for Video Synthesis
Yixuan Ren, Hanyu Wang, Hao Chen +2
We present NeRV-Diffusion, an implicit latent video diffusion model that synthesizes videos via generating neural network weights. The generated weights can be rearranged as the pa…
Trokens: Semantic-Aware Relational Trajectory Tokens for Few-Shot Action Recognition
Pulkit Kumar, Shuaiyi Huang, Matthew Walmer +2
Video understanding requires effective modeling of both motion and appearance information, particularly for few-shot action recognition. While recent advances in point tracking hav…
Towards Multimodal Understanding via Stable Diffusion as a Task-Aware Feature Extractor
Vatsal Agarwal, Matthew Gwilliam, Gefen Kohavi +3
Recent advances in multimodal large language models (MLLMs) have enabled image-based question-answering capabilities. However, a key limitation is the use of CLIP as the visual enc…
How to Design and Train Your Implicit Neural Representation for Video Compression
Matthew Gwilliam, Roy Zhang, Namitha Padmanabhan +2
Implicit neural representation (INR) methods for video compression have recently achieved visual quality and compression ratios that are competitive with traditional pipelines. How…
SIEDD: Shared-Implicit Encoder with Discrete Decoders
Vikram Rangarajan, Shishira Maiya, Max Ehrlich +1
Implicit Neural Representations (INRs) offer exceptional fidelity for video compression by learning per-video optimized functions, but their adoption is crippled by impractically s…
Evolutionary Caching to Accelerate Your Off-the-Shelf Diffusion Model
Anirud Aggarwal, Abhinav Shrivastava, Matthew Gwilliam
Diffusion-based image generation models excel at producing high-quality synthetic content, but suffer from slow and computationally expensive inference. Prior work has attempted to…
Utilization of Neighbor Information for Image Classification with Different Levels of Supervision
Gihan Jayatilaka, Abhinav Shrivastava, Matthew Gwilliam
We propose to bridge the gap between semi-supervised and unsupervised image recognition with a flexible method that performs well for both generalized category discovery (GCD) and…
Mitigating Hallucinations in Diffusion Models through Adaptive Attention Modulation
Trevine Oorloff, Yaser Yacoob, Abhinav Shrivastava
Diffusion models, while increasingly adept at generating realistic images, are notably hindered by hallucinations -- unrealistic or incorrect features inconsistent with the trained…
Accelerate High-Quality Diffusion Models with Inner Loop Feedback
Matthew Gwilliam, Han Cai, Di Wu +2
We propose Inner Loop Feedback (ILF), a novel approach to accelerate diffusion models' inference. ILF trains a lightweight module to predict future features in the denoising proces…
LARP: Tokenizing Videos with a Learned Autoregressive Generative Prior
Hanyu Wang, Saksham Suri, Yixuan Ren +2
We present LARP, a novel video tokenizer designed to overcome limitations in current video tokenization methods for autoregressive (AR) generative models. Unlike traditional patchw…
LEIA: Latent View-invariant Embeddings for Implicit 3D Articulation
Archana Swaminathan, Anubhav Gupta, Kamal Gupta +3
Neural Radiance Fields (NeRFs) have revolutionized the reconstruction of static scenes and objects in 3D, offering unprecedented quality. However, extending NeRFs to model dynamic…
Latent-INR: A Flexible Framework for Implicit Representations of Videos with Discriminative Semantics
Shishira R Maiya, Anubhav Gupta, Matthew Gwilliam +2
Implicit Neural Networks (INRs) have emerged as powerful representations to encode all forms of data, including images, videos, audios, and scenes. With video, many INRs for video…
Trajectory-aligned Space-time Tokens for Few-shot Action Recognition
Pulkit Kumar, Namitha Padmanabhan, Luke Luo +2
We propose a simple yet effective approach for few-shot action recognition, emphasizing the disentanglement of motion and appearance representations. By harnessing recent progress…
InVi: Object Insertion In Videos Using Off-the-Shelf Diffusion Models
Nirat Saini, Navaneeth Bodla, Ashish Shrivastava +4
We introduce InVi, an approach for inserting or replacing objects within videos (referred to as inpainting) using off-the-shelf, text-to-image latent diffusion models. InVi targets…
V-VIPE: Variational View Invariant Pose Embedding
Mara Levy, Abhinav Shrivastava
Learning to represent three dimensional (3D) human pose given a two dimensional (2D) image of a person, is a challenging problem. In order to make the problem less ambiguous it has…
UVIS: Unsupervised Video Instance Segmentation
Shuaiyi Huang, Saksham Suri, Kamal Gupta +3
Video instance segmentation requires classifying, segmenting, and tracking every object across video frames. Unlike existing approaches that rely on masks, boxes, or category label…
ARDuP: Active Region Video Diffusion for Universal Policies
Shuaiyi Huang, Mara Levy, Zhenyu Jiang +5
Sequential decision-making can be formulated as a text-conditioned video generation problem, where a video planner, guided by a text-defined goal, generates future frames visualizi…
AutoHallusion: Automatic Generation of Hallucination Benchmarks for Vision-Language Models
Xiyang Wu, Tianrui Guan, Dianqi Li +9
Large vision-language models (LVLMs) are prone to hallucinations, where certain contextual cues in an image can trigger the language module to produce overconfident and incorrect r…
MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understanding
Bo He, Hengduo Li, Young Kyun Jang +5
With the success of large language models (LLMs), integrating the vision model into LLMs to build vision-language foundation models has gained much more interest recently. However,…