59 citations · 82 across the 16 of their papers we have counts for
17 papers
CulturalFrames: Assessing Cultural Expectation Alignment in Text-to-Image Models and Evaluation Metrics
Shravan Nayak, Mehar Bhatia, Xiaofeng Zhang +6
The increasing ubiquity of text-to-image (T2I) models as tools for visual content generation raises concerns about their ability to accurately represent diverse cultural contexts -…
AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories
Xing Han Lù, Amirhossein Kazemnejad, Nicholas Meade +7
Web agents enable users to perform tasks on web browsers through natural language interaction. Evaluating web agents trajectories is an important problem, since it helps us determi…
DeepSeek-R1 Thoughtology: Let's think about LLM Reasoning
Sara Vera Marjanović, Arkil Patel, Vaibhav Adlakha +14
Large Reasoning Models like DeepSeek-R1 mark a fundamental shift in how LLMs approach complex problems. Instead of directly producing an answer for a given input, DeepSeek-R1 creat…
SafeArena: Evaluating the Safety of Autonomous Web Agents
Ada Defne Tur, Nicholas Meade, Xing Han Lù +6
LLM-based agents are becoming increasingly proficient at solving web-based tasks. With this capability comes a greater risk of misuse for malicious purposes, such as posting misinf…
Societal Alignment Frameworks Can Improve LLM Alignment
Karolina Stańczak, Nicholas Meade, Mehar Bhatia +14
Recent progress in large language models (LLMs) has focused on producing responses that meet human expectations and align with shared values - a process coined alignment. However,…
Benchmarking Vision Language Models for Cultural Understanding
Shravan Nayak, Kanishk Jain, Rabiul Awal +5
Foundation models and vision-language pre-training have notably advanced Vision Language Models (VLMs), enabling multimodal processing of visual and linguistic data. However, their…
A Multilingual Perspective on Probing Gender Bias
Karolina Stańczak
Gender bias represents a form of systematic negative treatment that targets individuals based on their gender. This discrimination can range from subtle sexist remarks and gendered…
The Causal Influence of Grammatical Gender on Distributional Semantics
Karolina Stańczak, Kevin Du, Adina Williams +2
How much meaning influences gender assignment across languages is an active area of research in linguistics and cognitive science. We can view current approaches as aiming to deter…
Invisible Women in Digital Diplomacy: A Multidimensional Framework for Online Gender Bias Against Women Ambassadors Worldwide
Yevgeniy Golovchenko, Karolina Stańczak, Rebecca Adler-Nissen +2
Despite mounting evidence that women in foreign policy often bear the brunt of online hostility, the extent of online gender bias against diplomats remains unexplored. This paper o…
Social Bias Probing: Fairness Benchmarking for Language Models
Marta Marchiori Manerba, Karolina Stańczak, Riccardo Guidotti +1
While the impact of social biases in language models has been recognized, prior methods for bias evaluation have been limited to binary association tests on small datasets, limitin…
Measuring Gender Bias in West Slavic Language Models
Sandra Martinková, Karolina Stańczak, Isabelle Augenstein
Pre-trained language models have been known to perpetuate biases from the underlying datasets to downstream tasks. However, these findings are predominantly based on monolingual la…
Measuring Intersectional Biases in Historical Documents
Nadav Borenstein, Karolina Stańczak, Thea Rolskov +3
Data-driven analyses of biases in historical texts can help illuminate the origin and development of biases prevailing in modern society. However, digitised historical documents po…
Same Neurons, Different Languages: Probing Morphosyntax in Multilingual Pre-trained Models
Karolina Stańczak, Edoardo Ponti, Lucas Torroba Hennigen +2
The success of multilingual pre-trained models is underpinned by their ability to learn representations shared by multiple languages even in absence of any explicit supervision. Ho…
A Latent-Variable Model for Intrinsic Probing
Karolina Stańczak, Lucas Torroba Hennigen, Adina Williams +2
The success of pre-trained contextualized representations has prompted researchers to analyze them for the presence of linguistic information. Indeed, it is natural to assume that…
A Survey on Gender Bias in Natural Language Processing
Karolina Stanczak, Isabelle Augenstein
Language can be used as a means of reproducing and enforcing harmful stereotypes and biases and has been analysed as such in numerous research. In this paper, we present a survey o…
Quantifying Gender Biases Towards Politicians on Reddit
Sara Marjanovic, Karolina Stańczak, Isabelle Augenstein
Despite attempts to increase gender parity in politics, global efforts have struggled to ensure equal female representation. This is likely tied to implicit gender biases against w…
Quantifying Gender Bias Towards Politicians in Cross-Lingual Language Models
Karolina Stańczak, Sagnik Ray Choudhury, Tiago Pimentel +2
Recent research has demonstrated that large pre-trained language models reflect societal biases expressed in natural language. The present paper introduces a simple method for prob…