6 papers
JuICE: A Benchmark for Evaluating LLM-Judge in Identifying Cultural Errors
Jiho Jin, Junho Myung, Juhyun Oh +5
As large language models (LLMs) are increasingly deployed to users around the world, they are integrated into everyday tasks across diverse cultural contexts, from drafting persona…
A Unified Framework to Quantify Cultural Intelligence of AI
Sunipa Dev, Vinodkumar Prabhakaran, Rutledge Chin Feman +16
As generative AI technologies are increasingly being launched across the globe, assessing their competence to operate in different cultural contexts is exigently becoming a priorit…
SAFARI: A Community-Engaged Approach and Dataset of Stereotype Resources in the Sub-Saharan African Context
Aishwarya Verma, Laud Ammah, Olivia Nercy Ndlovu Lucas +3
Stereotype repositories are critical to assess generative AI model safety, but currently lack adequate global coverage. It is imperative to prioritize targeted expansion, strategic…
Cultural Compass: A Framework for Organizing Societal Norms to Detect Violations in Human-AI Conversations
Myra Cheng, Vinodkumar Prabhakaran, Alice Oh +5
Generative AI models ought to be useful and safe across cross-cultural contexts. One critical step toward this goal is understanding how AI models adhere to sociocultural norms. Wh…
Scaling Cultural Resources for Improving Generative Models
Hayk Stepanyan, Aishwarya Verma, Andrew Zaldivar +5
Generative models are known to have reduced performance in different global cultural contexts and languages. While continual data updates have been commonly conducted to improve ov…
Adaptive Data Collection for Latin-American Community-sourced Evaluation of Stereotypes (LACES)
Guido Ivetta, Pietro Palombini, Sofía Martinelli +5
The evaluation of societal biases in NLP models is critically hindered by a geo-cultural gap, This leaves regions such as Latin America severely underserved, making it impossible t…