6 papers
Evolution of Accuracy and Visual-Cognitive Errors in a Decade of Vision-Language AI Models
Shravan Murlidaran, Miguel P. Eckstein
Vision language models (VLMs) have made remarkable progress in visual reasoning during the last decade. Most evaluations have used simple scenes (MS-COCO) that do not showcase comp…
Revealing the Gap in Human and VLM Scene Perception through Counterfactual Semantic Saliency
Ziqi Wen, Parsa Madinei, Miguel P. Eckstein
Evaluating whether large vision-language models (VLMs) align with human perception for high-level semantic scene comprehension remains a challenge. Traditional white-box interpreta…
IRIS: Intent Resolution via Inference-time Saccades for Open-Ended VQA in Large Vision-Language Models
Parsa Madinei, Srijita Karmakar, Russell Cohen Hoffing +2
We introduce IRIS (Intent Resolution via Inference-time Saccades), a novel training-free approach that uses eye-tracking data in real-time to resolve ambiguity in open-ended VQA. T…
INTERLACE: Interleaved Layer Pruning and Efficient Adaptation in Large Vision-Language Models
Parsa Madinei, Ryan Solgi, Ziqi Wen +3
We introduce INTERLACE, a novel framework that prunes redundant layers in VLMs while maintaining performance through sample-efficient finetuning. Existing layer pruning methods lea…
DReX: Pure Vision Fusion of Self-Supervised and Convolutional Representations for Image Complexity Prediction
Jonathan Skaza, Parsa Madinei, Ziqi Wen +1
Visual complexity prediction is a fundamental problem in computer vision with applications in image compression, retrieval, and classification. Understanding what makes humans perc…
Predicting Reaction Time to Comprehend Scenes with Foveated Scene Understanding Maps
Ziqi Wen, Jonathan Skaza, Shravan Murlidaran +2
Although models exist that predict human response times (RTs) in tasks such as target search and visual discrimination, the development of image-computable predictors for scene und…