8 papers · 1 filter
Pluralis v0.1: Towards a Multicultural, Multimodal, Multilingual Benchmark for AI Risk and Reliability
Alicia Parrish, Rajat Shinde, Sanket Badhe +57
Current AI safety evaluation and benchmarking frameworks predominantly rely on Western-centric culture-agnostic defaults that mask critical regional laws, socio-linguistic nuances,…
JuICE: A Benchmark for Evaluating LLM-Judge in Identifying Cultural Errors
Jiho Jin, Junho Myung, Juhyun Oh +5
As large language models (LLMs) are increasingly deployed to users around the world, they are integrated into everyday tasks across diverse cultural contexts, from drafting persona…
Investigating Counterfactual Unfairness in LLMs towards Identities through Humor
Shubin Kim, Yejin Son, Junyeong Park +6
Humor holds up a mirror to social perception: what we find funny often reflects who we are and how we judge others. When language models engage with humor, their reactions expose t…
Are they lovers or friends? Evaluating LLMs' Social Reasoning in English and Korean Dialogues
Eunsu Kim, Junyeong Park, Juhyun Oh +5
As LLMs are increasingly deployed in real-world interactions, their social reasoning in interpersonal communication becomes critical. To explore their capabilities, we introduce SC…
One-Topic-Doesn't-Fit-All: Transcreating Reading Comprehension Test for Personalized Learning
Jieun Han, Daniel Lee, Haneul Yoo +5
Personalized learning has gained attention in English as a Foreign Language (EFL) education, where engagement and motivation play crucial roles in reading comprehension. We propose…
When Tom Eats Kimchi: Evaluating Cultural Bias of Multimodal Large Language Models in Cultural Mixture Contexts
Jun Seong Kim, Kyaw Ye Thu, Javad Ismayilzada +6
In a highly globalized world, it is important for multi-modal large language models (MLLMs) to recognize and respond correctly to mixed-cultural inputs. For example, a model should…