62 citations · 145 across the 23 of their papers we have counts for
24 papers
When Does AI Augment Work? A Workflow-Level Framework for Human-Agent Collaboration
AI Collaboration, Jiaying Wu, Caleb Ziems +19
We aim to characterise the value of artificial intelligence in the workplace. Current studies largely measure this value in terms of the current automation capabilities and public…
Will Scaling Improve Social Simulation with LLMs?
Caleb Ziems, William Held, Su Doga Karaca +3
Large Language Model (LLM) social simulations are a promising research method, but they are not yet faithful enough to be adopted widely. In this work, we investigate whether the c…
Reflections and New Directions for Human-Centered Large Language Models
Caleb Ziems, Dora Zhao, Rose E. Wang +55
Large Language Models (LLMs) are increasingly shaping the private and professional lives of users, with numerous applications in business, education, finance, healthcare, law, and…
Culture Cartography: Mapping the Landscape of Cultural Knowledge
Caleb Ziems, William Held, Jane Yu +3
To serve global users safely and productively, LLMs need culture-specific knowledge that might not be learned during pre-training. How do we find such knowledge that is (1) salient…
EgoNormia: Benchmarking Physical Social Norm Understanding
MohammadHossein Rezaei, Yicheng Fu, Phil Cuvin +4
Human activity is moderated by norms; however, supervision for normative reasoning is sparse, particularly where norms are physically- or socially-grounded. We thus present EGONORM…
Silent Signals, Loud Impact: LLMs for Word-Sense Disambiguation of Coded Dog Whistles
Julia Kruk, Michela Marchini, Rijul Magu +3
A dog whistle is a form of coded communication that carries a secondary meaning to specific audiences and is often weaponized for racial and socioeconomic discrimination. Dog whist…
Measuring and Addressing Indexical Bias in Information Retrieval
Caleb Ziems, William Held, Jane Dwivedi-Yu +1
Information Retrieval (IR) systems are designed to deliver relevant content, but traditional systems may not optimize rankings for fairness, neutrality, or the balance of ideas. Co…
CultureBank: An Online Community-Driven Knowledge Base Towards Culturally Aware Language Technologies
Weiyan Shi, Ryan Li, Yutong Zhang +5
To enhance language models' cultural awareness, we design a generalizable pipeline to construct cultural knowledge bases from different online communities on a massive scale. With…
Social Skill Training with Large Language Models
Diyi Yang, Caleb Ziems, William Held +3
People rely on social skills like conflict resolution to communicate effectively and to thrive in both work and personal life. However, practice environments for social skills are…
Social Intelligence Data Infrastructure: Structuring the Present and Navigating the Future
Minzhi Li, Weiyan Shi, Caleb Ziems +1
As Natural Language Processing (NLP) systems become increasingly integrated into human social life, these technologies will need to increasingly rely on social intelligence. Althou…
Impressions: Understanding Visual Semiotics and Aesthetic Impact
Julia Kruk, Caleb Ziems, Diyi Yang
Is aesthetic impact different from beauty? Is visual salience a reflection of its capacity for effective communication? We present Impressions, a novel dataset through which to inv…
CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotation
Minzhi Li, Taiwei Shi, Caleb Ziems +4
Annotated data plays a critical role in Natural Language Processing (NLP) in training models and evaluating their performance. Given recent developments in Large Language Models (L…
Modeling Cross-Cultural Pragmatic Inference with Codenames Duet
Omar Shaikh, Caleb Ziems, William Held +3
Pragmatic reference enables efficient interpersonal communication. Prior work uses simple reference games to test models of pragmatic reasoning, often with unidentified speakers an…
TADA: Task-Agnostic Dialect Adapters for English
Will Held, Caleb Ziems, Diyi Yang
Large Language Models, the dominant starting point for Natural Language Processing (NLP) applications, fail at a higher rate for speakers of English dialects other than Standard Am…
NormBank: A Knowledge Bank of Situational Social Norms
Caleb Ziems, Jane Dwivedi-Yu, Yi-Chia Wang +2
We present NormBank, a knowledge bank of 155k situational norms. This resource is designed to ground flexible normative reasoning for interactive, assistive, and collaborative AI s…
Can Large Language Models Transform Computational Social Science?
Caleb Ziems, William Held, Omar Shaikh +3
Large Language Models (LLMs) are capable of successfully performing many language processing tasks zero-shot (without training data). If zero-shot LLMs can also reliably classify a…
Multi-VALUE: A Framework for Cross-Dialectal English NLP
Caleb Ziems, William Held, Jingfeng Yang +3
Dialect differences caused by regional, social, and economic factors cause performance discrepancies for many groups of language technology users. Inclusive and equitable language…
The Moral Integrity Corpus: A Benchmark for Ethical Dialogue Systems
Caleb Ziems, Jane A. Yu, Yi-Chia Wang +2
Conversational agents have come increasingly closer to human competence in open-domain dialogue settings; however, such models can reflect insensitive, hurtful, or entirely incoher…
Inducing Positive Perspectives with Text Reframing
Caleb Ziems, Minzhi Li, Anthony Zhang +1
Sentiment transfer is one popular example of a text style transfer task, where the goal is to reverse the sentiment polarity of a text. With a sentiment reversal comes also a rever…
VALUE: Understanding Dialect Disparity in NLU
Caleb Ziems, Jiaao Chen, Camille Harris +2
English Natural Language Understanding (NLU) systems have achieved great performances and even outperformed humans on benchmarks like GLUE and SuperGLUE. However, these benchmarks…
To Protect and To Serve? Analyzing Entity-Centric Framing of Police Violence
Caleb Ziems, Diyi Yang
Framing has significant but subtle effects on public opinion and policy. We propose an NLP framework to measure entity-centric frames. We use it to understand media coverage on pol…
Latent Hatred: A Benchmark for Understanding Implicit Hate Speech
Mai ElSherief, Caleb Ziems, David Muchlinski +4
Hate speech has grown significantly on social media, causing serious consequences for victims of all demographics. Despite much attention being paid to characterize and detect disc…
Racism is a Virus: Anti-Asian Hate and Counterspeech in Social Media during the COVID-19 Crisis
Bing He, Caleb Ziems, Sandeep Soni +3
The spread of COVID-19 has sparked racism and hate on social media targeted towards Asian communities. However, little is known about how racial hate spreads during a pandemic and…
Aggressive, Repetitive, Intentional, Visible, and Imbalanced: Refining Representations for Cyberbullying Classification
Caleb Ziems, Ymir Vigfusson, Fred Morstatter
Cyberbullying is a pervasive problem in online communities. To identify cyberbullying cases in large-scale social networks, content moderators depend on machine learning classifier…