2 citations · 6 across the 10 of their papers we have counts for
11 papers
Investigating Multi-Cultural Value Selection Biases in LLMs
Siddhesh Pawar, Arnav Arora, Dustin Wright +2
Large language models (LLMs) serve users whose cultural identities are multi-layered, yet cultural evaluations commonly reduce identity to one label. In this work, we study LLMs un…
Not What, But How: A Framework for Auditing LLM Responses across Positioning, Generalization, Anthropomorphism, and Maxims
Siddhesh Milind Pawar, Sarah Masud, Haneul Yoo +2
Large language models (LLMs) are being increasingly used to answer subjective, information-seeking questions, where users are sensitive to how responses are communicated, not just…
Capabilities Ain't All You Need: Measuring Propensities in AI
Daniel Romero-Alvarado, Fernando Martínez-Plumed, Lorenzo Pacchiardi +11
AI evaluation has primarily focused on measuring capabilities, with formal approaches inspired from Item Response Theory (IRT) being increasingly applied. Yet propensities - the te…
When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation
Mubashara Akhtar, Anka Reuel, Prajna Soni +34
Artificial intelligence benchmarks are an important mechanism to measure model progress and guide deployment decisions. However, benchmarks quickly "saturate", making it difficult…
CulTrace: Tracing Internal Cultural Reasoning in Large Language Models
Haeun Yu, Arnav Arora, Seogyeong Jeong +7
The growing deployment of large language models (LLMs) across diverse cultural contexts necessitates a deeper understanding of models' hidden representations of different cultures.…
BiasGym: A Simple and Generalizable Framework for Analyzing and Removing Biases through Injection
Sekh Mainul Islam, Nadav Borenstein, Siddhesh Milind Pawar +3
Understanding biases and stereotypes encoded in the weights of Large Language Models (LLMs) is crucial for developing effective mitigation strategies. However, biased behavior is o…
Evaluation of Cultural Competence of Vision-Language Models
Srishti Yadav, Lauren Tilton, Maria Antoniak +10
Modern vision-language models (VLMs) often fail at cultural competency evaluations and benchmarks. Given the diversity of applications built upon VLMs, there is renewed interest in…
Presumed Cultural Identity: How Names Shape LLM Responses
Siddhesh Pawar, Arnav Arora, Lucie-Aimée Kaffee +1
Names are deeply tied to human identity. They can serve as markers of individuality, cultural heritage, and personal history. However, using names as a core indicator of identity c…
Survey of Cultural Awareness in Language Models: Text and Beyond
Siddhesh Pawar, Junyeong Park, Jiho Jin +7
Large-scale deployment of large language models (LLMs) in various applications, such as chatbots and virtual assistants, requires LLMs to be culturally sensitive to the user to ens…
Agree To Disagree
Abhinav Raghuvanshi, Siddhesh Pawar, Anirudh Mittal
How frequently do individuals thoroughly review terms and conditions before proceeding to register for a service, install software, or access a website? The majority of internet us…
Tapping BERT for Preposition Sense Disambiguation
Siddhesh Pawar, Shyam Thombre, Anirudh Mittal +2
Prepositions are frequently occurring polysemous words. Disambiguation of prepositions is crucial in tasks like semantic role labelling, question answering, text entailment, and no…