9 papers
FAM-Bench: Benchmarking Condition-Aware Food-as-Medicine Reasoning in Vision-Language Models
Mingyang Mao, Bhargav Rishi Medisetti, Utkarsh Grover +4
Food AI can recognize dishes, ingredients, and nutrients. However, these abilities do not show whether a dish is suitable for a person with a specific health condition. They also d…
ExpAlign: Expectation-Guided Vision-Language Alignment for Open-Vocabulary Grounding
Junyi Hu, Tian Bai, Fengyi Wu +3
Open-vocabulary grounding requires accurate vision-language alignment under weak supervision, yet existing methods either rely on global sentence embeddings that lack fine-grained…
CultureCLIP: Empowering CLIP with Cultural Awareness through Synthetic Images and Contextualized Captions
Yuchen Huang, Zhiyuan Fan, Zhitao He +3
Pretrained vision-language models (VLMs) such as CLIP excel in general multimodal comprehension but often struggle to capture nuanced, context-dependent visual cues. This makes it…
Hanfu-Bench: A Multimodal Benchmark on Cross-Temporal Cultural Understanding and Transcreation
Li Zhou, Lutong Yu, Dongchu Xie +3
Culture is a rich and dynamic domain that evolves across both geography and time. However, existing studies on cultural understanding with vision-language models (VLMs) primarily e…
RAVENEA: A Benchmark for Multimodal Retrieval-Augmented Visual Culture Understanding
Jiaang Li, Yifei Yuan, Wenyan Li +8
As vision-language models (VLMs) become increasingly integrated into daily life, the need for accurate visual culture understanding is becoming critical. Yet, these models frequent…
FoodieQA: A Multimodal Dataset for Fine-Grained Understanding of Chinese Food Culture
Wenyan Li, Xinyu Zhang, Jiaang Li +9
Food is a rich and varied dimension of cultural heritage, crucial to both individuals and social groups. To bridge the gap in the literature on the often-overlooked regional divers…
Words Worth a Thousand Pictures: Measuring and Understanding Perceptual Variability in Text-to-Image Generation
Raphael Tang, Xinyu Zhang, Lixinyu Xu +5
Diffusion models are the state of the art in text-to-image generation, but their perceptual variability remains understudied. In this paper, we examine how prompts affect image var…
Understanding Retrieval Robustness for Retrieval-Augmented Image Captioning
Wenyan Li, Jiaang Li, Rita Ramos +2
Recent advances in retrieval-augmented models for image captioning highlight the benefit of retrieving related captions for efficient, lightweight models with strong domain-transfe…
Exploring Visual Culture Awareness in GPT-4V: A Comprehensive Probing
Yong Cao, Wenyan Li, Jiaang Li +3
Pretrained large Vision-Language models have drawn considerable interest in recent years due to their remarkable performance. Despite considerable efforts to assess these models fr…