30 citations · 37 across the 14 of their papers we have counts for
16 papers
SPELUNKA: A Framework for Finding Unanticipated Associations in Vision-Language Models
Cassandra Parent, Isha Agarwal, Kumail Alhamoud +4
Vision-Language Models (VLMs) are the backbone of many state-of-the-art multimodal systems, but often encode spurious and harmful associations from their training data. While prior…
MatrAIx: Simulating the World with 8.3 Billion Persona Agents
Xiaomin Li, Yuexing Hao, Jianheng Hou +90
Human evaluation of AI systems and digital products is costly, slow, and difficult to scale. Offline evaluations are more scalable but often abstract away human diversity and inter…
CurveShift: Is Agent Progress Scalar? Separating Level from Shape
Hanwen Xing, Pengyun Wang, BingXu Meng +8
Progress in large language models is often summarized using a single scalar measure, such as a time horizon, a latent ability estimate, or an aggregate benchmark score. These summa…
Running the Gauntlet: Hard Agentic Tasks
Mykola Vysotskyi, Runqi Lin, Grzegorz Biziel +21
As agentic systems continue to evolve and are widely deployed in real-world scenarios, there is a growing demand to faithfully evaluate their capabilities. However, current benchma…
Disparities In Negation Understanding Across Languages In Vision-Language Models
Charikleia Moraitaki, Sarah Pan, Skyler Pulling +3
Vision-language models (VLMs) exhibit affirmation bias: a systematic tendency to select positive captions ("X is present") even when the correct description contains negation ("no…
SpaceVLM: Sub-Space Modeling of Negation in Vision-Language Models
Sepehr Kazemi Ranjbar, Kumail Alhamoud, Marzyeh Ghassemi
Vision-Language Models (VLMs) struggle with negation. Given a prompt like "retrieve (or generate) a street scene without pedestrians," they often fail to respect the "not." Existin…
Aggregation Hides Out-of-Distribution Generalization Failures from Spurious Correlations
Olawale Salaudeen, Haoran Zhang, Kumail Alhamoud +2
Benchmarks for out-of-distribution (OOD) generalization frequently show a strong positive correlation between in-distribution (ID) and OOD accuracy across models, termed "accuracy-…
Train Long, Think Short: Curriculum Learning for Efficient Reasoning
Hasan Abed Al Kader Hammoud, Kumail Alhamoud, Abed Hammoud +3
Recent work on enhancing the reasoning abilities of large language models (LLMs) has introduced explicit length control as a means of constraining computational cost while preservi…
MedPAIR: Measuring Physicians and AI Relevance Alignment in Medical Question Answering
Yuexing Hao, Kumail Alhamoud, Hyewon Jeong +6
Large Language Models (LLMs) have demonstrated remarkable performance on various medical question-answering (QA) benchmarks, including standardized medical exams. However, correct…
Medical Hallucinations in Foundation Models and Their Impact on Healthcare
Yubin Kim, Hyewon Jeong, Shan Chen +24
Hallucinations in foundation models arise from autoregressive training objectives that prioritize token-likelihood optimization over epistemic accuracy, fostering overconfidence an…
Vision-Language Models Do Not Understand Negation
Kumail Alhamoud, Shaden Alshammari, Yonglong Tian +4
Many practical vision-language applications require models that understand negation, e.g., when using natural language to retrieve images which contain certain objects but not othe…
FedMedICL: Towards Holistic Evaluation of Distribution Shifts in Federated Medical Imaging
Kumail Alhamoud, Yasir Ghunaim, Motasem Alfarra +5
For medical imaging AI models to be clinically impactful, they must generalize. However, this goal is hindered by (i) diverse types of distribution shifts, such as temporal, demogr…
Real-Time Evaluation in Online Continual Learning: A New Hope
Yasir Ghunaim, Adel Bibi, Kumail Alhamoud +5
Current evaluations of Continual Learning (CL) methods typically assume that there is no constraint on training time and computation. This is an unrealistic assumption for any real…
PIVOT: Prompting for Video Continual Learning
Andrés Villa, Juan León Alcázar, Motasem Alfarra +5
Modern machine learning pipelines are limited due to data availability, storage quotas, privacy regulations, and expensive annotation processes. These constraints make it difficult…
Generalizability of Adversarial Robustness Under Distribution Shifts
Kumail Alhamoud, Hasan Abed Al Kader Hammoud, Motasem Alfarra +1
Recent progress in empirical and certified robustness promises to deliver reliable and deployable Deep Neural Networks (DNNs). Despite that success, most existing evaluations of DN…
vCLIMB: A Novel Video Class Incremental Learning Benchmark
Andrés Villa, Kumail Alhamoud, Juan León Alcázar +3
Continual learning (CL) is under-explored in the video domain. The few existing works contain splits with imbalanced class distributions over the tasks, or study the problem in uns…