708 citations · 965 across the 21 of their papers we have counts for
23 papers
Some Modalities are More Equal Than Others: Decoding and Architecting Multimodal Integration in MLLMs
Tianle Chen, Chaitanya Chakka, Arjun Reddy Akula +2
Despite remarkable advancements in Multimodal Large Language Models (MLLMs), a fundamental question remains: are MLLMs robust to contradicting modalities? To rigorously study this,…
Generative Action Tell-Tales: Assessing Human Motion in Synthesized Videos
Xavier Thomas, Youngsun Lim, Ananya Srinivasan +2
Despite rapid advances in video generative models, robust metrics for evaluating visual and temporal correctness of complex human actions remain elusive. Critically, existing pure-…
Improving Physical Object State Representation in Text-to-Image Generative Systems
Tianle Chen, Chaitanya Chakka, Deepti Ghadiyaram
Current text-to-image generative models struggle to accurately represent object states (e.g., "a table without a bottle," "an empty tumbler"). In this work, we first design a fully…
What's in a Latent? Leveraging Diffusion Latent Space for Domain Generalization
Xavier Thomas, Deepti Ghadiyaram
Domain Generalization aims to develop models that can generalize to novel and unseen data distributions. In this work, we study how model architectures and pre-training objectives…
Progressive Prompt Detailing for Improved Alignment in Text-to-Image Generative Models
Ketan Suhaas Saichandran, Xavier Thomas, Prakhar Kaushik +1
Text-to-image generative models often struggle with long prompts detailing complex scenes, diverse objects with distinct visual characteristics and spatial relationships. In this w…
DebiasPI: Inference-time Debiasing by Prompt Iteration of a Text-to-Image Generative Model
Sarah Bonna, Yu-Cheng Huang, Ekaterina Novozhilova +11
Ethical intervention prompting has emerged as a tool to counter demographic biases of text-to-image generative AI models. Existing solutions either require to retrain the model or…
: Interpreting and leveraging semantic information in diffusion models
Dahye Kim, Xavier Thomas, Deepti Ghadiyaram
We study rich visual semantic information is represented within various layers and denoising timesteps of different diffusion architectures. We uncover monosemantic…
Mitigating stereotypical biases in text to image generative systems
Piero Esposito, Parmida Atighehchian, Anastasis Germanidis +1
State-of-the-art generative text-to-image models are known to exhibit social biases and over-represent certain groups like people of perceived lighter skin tones and men in their o…
Helping Visually Impaired People Take Better Quality Pictures
Maniratnam Mandal, Deepti Ghadiyaram, Danna Gurari +1
Perception-based image analysis technologies can be used to help visually impaired people take better quality pictures by providing automated guidance, thereby empowering them to i…
GeoDE: a Geographically Diverse Evaluation Dataset for Object Recognition
Vikram V. Ramaswamy, Sing Yu Lin, Dora Zhao +4
Current dataset collection methods typically scrape large amounts of data from the web. While this technique is extremely scalable, data collected in this way tends to reinforce st…
Telepresence Video Quality Assessment
Zhenqiang Ying, Deepti Ghadiyaram, Alan Bovik
Video conferencing, which includes both video and audio content, has contributed to dramatic increases in Internet traffic, as the COVID-19 pandemic forced millions of people to wo…
Making Heads or Tails: Towards Semantically Consistent Visual Counterfactuals
Simon Vandenhende, Dhruv Mahajan, Filip Radenovic +1
A visual counterfactual explanation replaces image regions in a query image with regions from a distractor image such that the system's decision on the transformed image changes to…
Generic Event Boundary Detection: A Benchmark for Event Segmentation
Mike Zheng Shou, Stan Weixian Lei, Weiyao Wang +2
This paper presents a novel task together with a new benchmark for detecting generic, taxonomy-free event boundaries that segment a whole video into chunks. Conventional work in te…
Patch-VQ: 'Patching Up' the Video Quality Problem
Zhenqiang Ying, Maniratnam Mandal, Deepti Ghadiyaram +1
No-reference (NR) perceptual video quality assessment (VQA) is a complex, unsolved, and important problem to social and streaming media applications. Efficient and accurate video q…
How2Sign: A Large-scale Multimodal Dataset for Continuous American Sign Language
Amanda Duarte, Shruti Palaskar, Lucas Ventura +5
One of the factors that have hindered progress in the areas of sign language recognition, translation, and production is the absence of large annotated datasets. Towards this end,…
Don't Judge an Object by Its Context: Learning to Overcome Contextual Bias
Krishna Kumar Singh, Dhruv Mahajan, Kristen Grauman +3
Existing models often leverage co-occurrences between objects and their context to improve recognition accuracy. However, strongly relying on context risks a model's generalizabili…
From Patches to Pictures (PaQ-2-PiQ): Mapping the Perceptual Space of Picture Quality
Zhenqiang Ying, Haoran Niu, Praful Gupta +3
Blind or no-reference (NR) perceptual picture quality prediction is a difficult, unsolved problem of great consequence to the social and streaming media industries that impacts bil…
ClusterFit: Improving Generalization of Visual Representations
Xueting Yan, Ishan Misra, Abhinav Gupta +2
Pre-training convolutional neural networks with weakly-supervised and self-supervised strategies is becoming increasingly popular for several computer vision tasks. However, due to…
Large-scale weakly-supervised pre-training for video action recognition
Deepti Ghadiyaram, Matt Feiszli, Du Tran +3
Current fully-supervised video datasets consist of only a few hundred thousand videos and fewer than a thousand domain-specific labels. This hinders the progress towards advanced v…
Activity Driven Weakly Supervised Object Detection
Zhenheng Yang, Dhruv Mahajan, Deepti Ghadiyaram +2
Weakly supervised object detection aims at reducing the amount of supervision required to train detection models. Such models are traditionally learned from images/videos labelled…
Less is More: Learning Highlight Detection from Video Duration
Bo Xiong, Yannis Kalantidis, Deepti Ghadiyaram +1
Highlight detection has the potential to significantly ease video browsing, but existing methods often suffer from expensive supervision requirements, where human viewers must manu…
Perceptual Quality Prediction on Authentically Distorted Images Using a Bag of Features Approach
Deepti Ghadiyaram, Alan C. Bovik
Current top-performing blind perceptual image quality prediction models are generally trained on legacy databases of human quality opinion scores on synthetically distorted images.…
Massive Online Crowdsourced Study of Subjective and Objective Picture Quality
Deepti Ghadiyaram, Alan C. Bovik
Most publicly available image quality databases have been created under highly controlled conditions by introducing graded simulated distortions onto high-quality photographs. Howe…