9 citations · 49 across the 32 of their papers we have counts for
38 papers
Investigating Multi-Cultural Value Selection Biases in LLMs
Siddhesh Pawar, Arnav Arora, Dustin Wright +2
Large language models (LLMs) serve users whose cultural identities are multi-layered, yet cultural evaluations commonly reduce identity to one label. In this work, we study LLMs un…
The Role of Implicit and Explicit Demographic Signals in Large Language Model-based Student Assessment
Donya Rooein, Luca Benedetto, Dirk Hovy
Large Language Models are now common in student assessment, but we know little about how student demographics affect their use. Sometimes, considering student demographics may be n…
WildSEEK: Evaluating Language Models for Information-Seeking
Tanise Ceron, Joachim Baumann, Elisa Bassignana +3
Language models are increasingly mediating information access to end users, urging a systematic evaluation of their responses for a fair and reliable information ecosystem. Existin…
Stop Automating Peer Review Without Rigorous Evaluation
Joachim Baumann, Jiaxin Pei, Sanmi Koyejo +1
Large language models offer a tempting solution to address the peer review crisis. This position paper argues that today's AI systems should not be used to produce paper reviews. W…
Diffusion Language Models Are Natively Length-Aware
Vittorio Rossi, Giacomo Cirò, Davide Beltrame +3
Unlike autoregressive language models, which terminate variable-length generation upon predicting an End-of-Sequence (EoS) token, Diffusion Language Models (DLMs) operate over a fi…
Do Large Language Models Adapt to Language Variation across Socioeconomic Status?
Elisa Bassignana, Mike Zhang, Dirk Hovy +1
Humans adjust their linguistic style to the audience they are addressing. However, the extent to which LLMs adapt to different social contexts is largely unknown. As these models i…
Responsible Evaluation of AI for Mental Health
Hiba Arnaout, Anmol Goel, H. Andrew Schwartz +13
Although artificial intelligence (AI) shows growing promise for mental health care, current approaches to evaluating AI tools in this domain remain fragmented and poorly aligned wi…
PATS: Personality-Aware Teaching Strategies with Large Language Model Tutors
Donya Rooein, Sankalan Pal Chowdhury, Mariia Eremeeva +4
Recent advances in large language models (LLMs) demonstrate their potential as educational tutors. However, different tutoring strategies benefit different student personalities, a…
Do Prompts Reshape Representations? An Empirical Study of Prompting Effects on Embeddings
Cesar Gonzalez-Gutierrez, Dirk Hovy
Prompting is a common approach for leveraging LMs in zero-shot settings. However, the underlying mechanisms that enable LMs to perform diverse tasks without task-specific supervisi…
Large Language Model Hacking: Quantifying the Hidden Risks of Using LLMs for Text Annotation
Joachim Baumann, Paul Röttger, Aleksandra Urman +4
Large language models are rapidly transforming social science research by enabling the automation of labor-intensive tasks like data annotation and text analysis. However, LLM outp…
No for Some, Yes for Others: Persona Prompts and Other Sources of False Refusal in Language Models
Flor Miriam Plaza-del-Arco, Paul Röttger, Nino Scherrer +3
Large language models (LLMs) are increasingly integrated into our daily lives and personalized. However, LLM personalization might also increase unintended side effects. Recent wor…
Biased Tales: Cultural and Topic Bias in Generating Children's Stories
Donya Rooein, Vilém Zouhar, Debora Nozza +1
Stories play a pivotal role in human communication, shaping beliefs and morals, particularly in children. As parents increasingly rely on large language models (LLMs) to craft bedt…
Exploring Subjective Tasks in Farsi: A Survey Analysis and Evaluation of Language Models
Donya Rooein, Flor Miriam Plaza-del-Arco, Debora Nozza +1
Given Farsi's speaker base of over 127 million people and the growing availability of digital text, including more than 1.3 million articles on Wikipedia, it is considered a middle…
Principled Personas: Defining and Measuring the Intended Effects of Persona Prompting on Task Performance
Pedro Henrique Luz de Araujo, Paul Röttger, Dirk Hovy +1
Expert persona prompting -- assigning roles such as expert in math to language models -- is widely used for task improvement. However, prior work shows mixed results on its effecti…
The Pluralistic Moral Gap: Understanding Judgment and Value Differences between Humans and Large Language Models
Giuseppe Russo, Debora Nozza, Paul Röttger +1
People increasingly rely on Large Language Models (LLMs) for moral advice, which may influence humans' decisions. Yet, little is known about how closely LLMs align with human moral…
Co-DETECT: Collaborative Discovery of Edge Cases in Text Classification
Chenfei Xiong, Jingwei Ni, Yu Fan +10
We introduce Co-DETECT (Collaborative Discovery of Edge cases in TExt ClassificaTion), a novel mixed-initiative annotation framework that integrates human expertise with automatic…
Educators' Perceptions of Large Language Models as Tutors: Comparing Human and AI Tutors in a Blind Text-only Setting
Sankalan Pal Chowdhury, Terry Jingchen Zhang, Donya Rooein +3
The rapid development of Large Language Models (LLMs) opens up the possibility of using them as personal tutors. This has led to the development of several intelligent tutoring sys…
Can Reasoning Help Large Language Models Capture Human Annotator Disagreement?
Jingwei Ni, Yu Fan, Vilém Zouhar +6
Variation in human annotation (i.e., disagreements) is common in NLP, often reflecting important information like task subjectivity and sample ambiguity. Modeling this variation is…
The AI Gap: How Socioeconomic Status Affects Language Technology Interactions
Elisa Bassignana, Amanda Cercas Curry, Dirk Hovy
Socioeconomic status (SES) fundamentally influences how people interact with each other and more recently, with digital technologies like Large Language Models (LLMs). While previo…
Beyond Demographics: Fine-tuning Large Language Models to Predict Individuals' Subjective Text Perceptions
Matthias Orlikowski, Jiaxin Pei, Paul Röttger +3
People naturally vary in their annotations for subjective questions and some of this variation is thought to be due to the person's sociodemographic characteristics. LLMs have also…
IssueBench: Millions of Realistic Prompts for Measuring Issue Bias in LLM Writing Assistance
Paul Röttger, Musashi Hinck, Valentin Hofmann +4
Large language models (LLMs) are helping millions of users write texts about diverse issues, and in doing so expose users to different ideas and perspectives. This creates concerns…
MSTS: A Multimodal Safety Test Suite for Vision-Language Models
Paul Röttger, Giuseppe Attanasio, Felix Friedrich +19
Vision-language models (VLMs), which process image and text inputs, are increasingly integrated into chat assistants and other consumer AI applications. Without proper safeguards,…
Compromesso! Italian Many-Shot Jailbreaks Undermine the Safety of Large Language Models
Fabio Pernisi, Dirk Hovy, Paul Röttger
As diverse linguistic communities and users adopt large language models (LLMs), assessing their safety across languages becomes critical. Despite ongoing efforts to make LLMs safe,…
Divine LLaMAs: Bias, Stereotypes, Stigmatization, and Emotion Representation of Religion in Large Language Models
Flor Miriam Plaza-del-Arco, Amanda Cercas Curry, Susanna Paoli +2
Emotions play important epistemological and cognitive roles in our lives, revealing our values and guiding our actions. Previous work has shown that LLMs display biases in emotion…