9 citations · 49 across the 32 of their papers we have counts for
35 papers · 1 filter
Investigating Multi-Cultural Value Selection Biases in LLMs
Siddhesh Pawar, Arnav Arora, Dustin Wright +2
Large language models (LLMs) serve users whose cultural identities are multi-layered, yet cultural evaluations commonly reduce identity to one label. In this work, we study LLMs un…
The Role of Implicit and Explicit Demographic Signals in Large Language Model-based Student Assessment
Donya Rooein, Luca Benedetto, Dirk Hovy
Large Language Models are now common in student assessment, but we know little about how student demographics affect their use. Sometimes, considering student demographics may be n…
WildSEEK: Evaluating Language Models for Information-Seeking
Tanise Ceron, Joachim Baumann, Elisa Bassignana +3
Language models are increasingly mediating information access to end users, urging a systematic evaluation of their responses for a fair and reliable information ecosystem. Existin…
Diffusion Language Models Are Natively Length-Aware
Vittorio Rossi, Giacomo Cirò, Davide Beltrame +3
Unlike autoregressive language models, which terminate variable-length generation upon predicting an End-of-Sequence (EoS) token, Diffusion Language Models (DLMs) operate over a fi…
Do Large Language Models Adapt to Language Variation across Socioeconomic Status?
Elisa Bassignana, Mike Zhang, Dirk Hovy +1
Humans adjust their linguistic style to the audience they are addressing. However, the extent to which LLMs adapt to different social contexts is largely unknown. As these models i…
PATS: Personality-Aware Teaching Strategies with Large Language Model Tutors
Donya Rooein, Sankalan Pal Chowdhury, Mariia Eremeeva +4
Recent advances in large language models (LLMs) demonstrate their potential as educational tutors. However, different tutoring strategies benefit different student personalities, a…
Do Prompts Reshape Representations? An Empirical Study of Prompting Effects on Embeddings
Cesar Gonzalez-Gutierrez, Dirk Hovy
Prompting is a common approach for leveraging LMs in zero-shot settings. However, the underlying mechanisms that enable LMs to perform diverse tasks without task-specific supervisi…
Large Language Model Hacking: Quantifying the Hidden Risks of Using LLMs for Text Annotation
Joachim Baumann, Paul Röttger, Aleksandra Urman +4
Large language models are rapidly transforming social science research by enabling the automation of labor-intensive tasks like data annotation and text analysis. However, LLM outp…
No for Some, Yes for Others: Persona Prompts and Other Sources of False Refusal in Language Models
Flor Miriam Plaza-del-Arco, Paul Röttger, Nino Scherrer +3
Large language models (LLMs) are increasingly integrated into our daily lives and personalized. However, LLM personalization might also increase unintended side effects. Recent wor…
Biased Tales: Cultural and Topic Bias in Generating Children's Stories
Donya Rooein, Vilém Zouhar, Debora Nozza +1
Stories play a pivotal role in human communication, shaping beliefs and morals, particularly in children. As parents increasingly rely on large language models (LLMs) to craft bedt…
Exploring Subjective Tasks in Farsi: A Survey Analysis and Evaluation of Language Models
Donya Rooein, Flor Miriam Plaza-del-Arco, Debora Nozza +1
Given Farsi's speaker base of over 127 million people and the growing availability of digital text, including more than 1.3 million articles on Wikipedia, it is considered a middle…
Principled Personas: Defining and Measuring the Intended Effects of Persona Prompting on Task Performance
Pedro Henrique Luz de Araujo, Paul Röttger, Dirk Hovy +1
Expert persona prompting -- assigning roles such as expert in math to language models -- is widely used for task improvement. However, prior work shows mixed results on its effecti…
The Pluralistic Moral Gap: Understanding Judgment and Value Differences between Humans and Large Language Models
Giuseppe Russo, Debora Nozza, Paul Röttger +1
People increasingly rely on Large Language Models (LLMs) for moral advice, which may influence humans' decisions. Yet, little is known about how closely LLMs align with human moral…
Co-DETECT: Collaborative Discovery of Edge Cases in Text Classification
Chenfei Xiong, Jingwei Ni, Yu Fan +10
We introduce Co-DETECT (Collaborative Discovery of Edge cases in TExt ClassificaTion), a novel mixed-initiative annotation framework that integrates human expertise with automatic…
Can Reasoning Help Large Language Models Capture Human Annotator Disagreement?
Jingwei Ni, Yu Fan, Vilém Zouhar +6
Variation in human annotation (i.e., disagreements) is common in NLP, often reflecting important information like task subjectivity and sample ambiguity. Modeling this variation is…
The AI Gap: How Socioeconomic Status Affects Language Technology Interactions
Elisa Bassignana, Amanda Cercas Curry, Dirk Hovy
Socioeconomic status (SES) fundamentally influences how people interact with each other and more recently, with digital technologies like Large Language Models (LLMs). While previo…
Beyond Demographics: Fine-tuning Large Language Models to Predict Individuals' Subjective Text Perceptions
Matthias Orlikowski, Jiaxin Pei, Paul Röttger +3
People naturally vary in their annotations for subjective questions and some of this variation is thought to be due to the person's sociodemographic characteristics. LLMs have also…
IssueBench: Millions of Realistic Prompts for Measuring Issue Bias in LLM Writing Assistance
Paul Röttger, Musashi Hinck, Valentin Hofmann +4
Large language models (LLMs) are helping millions of users write texts about diverse issues, and in doing so expose users to different ideas and perspectives. This creates concerns…
MSTS: A Multimodal Safety Test Suite for Vision-Language Models
Paul Röttger, Giuseppe Attanasio, Felix Friedrich +19
Vision-language models (VLMs), which process image and text inputs, are increasingly integrated into chat assistants and other consumer AI applications. Without proper safeguards,…
Compromesso! Italian Many-Shot Jailbreaks Undermine the Safety of Large Language Models
Fabio Pernisi, Dirk Hovy, Paul Röttger
As diverse linguistic communities and users adopt large language models (LLMs), assessing their safety across languages becomes critical. Despite ongoing efforts to make LLMs safe,…
Divine LLaMAs: Bias, Stereotypes, Stigmatization, and Emotion Representation of Religion in Large Language Models
Flor Miriam Plaza-del-Arco, Amanda Cercas Curry, Susanna Paoli +2
Emotions play important epistemological and cognitive roles in our lives, revealing our values and guiding our actions. Previous work has shown that LLMs display biases in emotion…
Beyond Flesch-Kincaid: Prompt-based Metrics Improve Difficulty Classification of Educational Texts
Donya Rooein, Paul Rottger, Anastassia Shaitarova +1
Using large language models (LLMs) for educational applications like dialogue-based teaching is a hot topic. Effective teaching, however, requires teachers to adapt the difficulty…
What Can Natural Language Processing Do for Peer Review?
Ilia Kuznetsov, Osama Mohammed Afzal, Koen Dercksen +21
The number of scientific articles produced every year is growing rapidly. Providing quality control over them is crucial for scientists and, ultimately, for the public good. In mod…
Conversations as a Source for Teaching Scientific Concepts at Different Education Levels
Donya Rooein, Dirk Hovy
Open conversations are one of the most engaging forms of teaching. However, creating those conversations in educational software is a complex endeavor, especially if we want to add…