10 papers
Steering LLMs for Culturally Localized Generation
Simran Khanuja, Hongbin Liu, Shujian Zhang +4
LLMs are deployed globally, yet produce responses biased towards cultures with abundant training data. Existing cultural localization approaches such as prompting or post-training…
Linguistically Informed Evaluation of Multilingual ASR for African Languages
Fei-Yueh Chen, Lateef Adeleke, C. M. Downey
Word Error Rate (WER) mischaracterizes ASR models' performance for African languages by combining phonological, tone, and other linguistic errors into a single lexical error. By co…
CAIRe: Cultural Attribution of Images by Retrieval-Augmented Evaluation
Arnav Yayavaram, Siddharth Yayavaram, Simran Khanuja +2
As text-to-image models become increasingly prevalent, ensuring their equitable performance across diverse cultural contexts is critical. Efforts to mitigate cross-cultural biases…
MERLIN: A Testbed for Multilingual Multimodal Entity Recognition and Linking
Sathyanarayanan Ramamoorthy, Vishwa Shah, Simran Khanuja +5
This paper introduces MERLIN, a novel testbed system for the task of Multilingual Multimodal Entity Linking. The created dataset includes BBC news article titles, paired with corre…
Grounding Multilingual Multimodal LLMs With Cultural Knowledge
Jean de Dieu Nyandwi, Yueqi Song, Simran Khanuja +1
Multimodal Large Language Models excel in high-resource settings, but often misinterpret long-tail cultural entities and underperform in low-resource languages. To address this gap…
NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samples
Baiqi Li, Zhiqiu Lin, Wenxuan Peng +7
Vision-language models (VLMs) have made significant progress in recent visual-question-answering (VQA) benchmarks that evaluate complex visio-linguistic reasoning. However, are the…