131 citations · 319 across the 29 of their papers we have counts for
19 papers · 1 filter
Turning Speech Language Models into Multilingual Listeners
Tolúlopé Ògúnrèmí, Dan Jurafsky, Chris Manning +2
Speech Language Models (SLMs) that understand spoken language questions support only a few high-resource languages, limiting access to millions of people worldwide. This gap stems…
string2string Studio: An Interactive, In-Browser Platform for String-to-String Algorithms
Mirac Suzgun, James Zou, Stuart M. Shieber +1
We present string2string Studio, an interactive in-browser platform for string-to-string analysis across natural language processing, computational biology, and the digital humanit…
Verbalizing LLMs' assumptions to explain and control sycophancy
Myra Cheng, Isabel Sieh, Humishka Zope +7
LLMs can be socially sycophantic, affirming users when they ask questions like "am I in the wrong?" rather than providing genuine assessment. We hypothesize that this behavior aris…
Beyond Tokens: Concept-Level Training Objectives for LLMs
Laya Iyer, Pranav Somani, Alice Guo +2
The next-token prediction (NTP) objective has been foundational in the development of modern large language models (LLMs), driving advances in fluency and generalization. However,…
Accommodation and Epistemic Vigilance: A Pragmatic Account of Why LLMs Fail to Challenge Harmful Beliefs
Myra Cheng, Robert D. Hawkins, Dan Jurafsky
Large language models (LLMs) frequently fail to challenge users' harmful beliefs in domains ranging from medical advice to social reasoning. We argue that these failures can be und…
Generation Space Size: Understanding and Calibrating Open-Endedness of LLM Generations
Sunny Yu, Ahmad Jabbar, Robert Hawkins +2
Different open-ended generation tasks require different degrees of output diversity. However, current LLMs are often miscalibrated. They collapse to overly homogeneous outputs for…
The ML-SUPERB 2.0 Challenge: Towards Inclusive ASR Benchmarking for All Language Varieties
William Chen, Chutong Meng, Jiatong Shi +10
Recent improvements in multilingual ASR have not been equally distributed across languages and language varieties. To advance state-of-the-art (SOTA) ASR models, we present the Int…
Humans overrely on overconfident language models, across languages
Neil Rathi, Dan Jurafsky, Kaitlyn Zhou
As large language models (LLMs) are deployed globally, it is crucial that their responses are calibrated across languages to accurately convey uncertainty and limitations. Prior wo…
ELEPHANT: Measuring and understanding social sycophancy in LLMs
Myra Cheng, Sunny Yu, Cinoo Lee +3
LLMs are known to exhibit sycophancy: agreeing with and flattering users, even at the cost of correctness. Prior work measures sycophancy only as direct agreement with users' expli…
Cooking Up Creativity: Enhancing LLM Creativity through Structured Recombination
Moran Mizrahi, Chen Shani, Gabriel Stanovsky +2
Large Language Models (LLMs) excel at many tasks, yet they struggle to produce truly creative, diverse ideas. In this paper, we introduce a novel approach that enhances LLM creativ…
Belief in the Machine: Investigating Epistemological Blind Spots of Language Models
Mirac Suzgun, Tayfun Gur, Federico Bianchi +4
As language models (LMs) become integral to fields like healthcare, law, and journalism, their ability to differentiate between fact, belief, and knowledge is essential for reliabl…
Data Checklist: On Unit-Testing Datasets with Usable Information
Heidi C. Zhang, Shabnam Behzad, Kawin Ethayarajh +1
Model checklists (Ribeiro et al., 2020) have emerged as a useful tool for understanding the behavior of LLMs, analogous to unit-testing in software engineering. However, despite da…
Rel-A.I.: An Interaction-Centered Approach To Measuring Human-LM Reliance
Kaitlyn Zhou, Jena D. Hwang, Xiang Ren +3
The ability to communicate uncertainty, risk, and limitation is crucial for the safety of large language models. However, current evaluations of these abilities rely on simple cali…
NLP Systems That Can't Tell Use from Mention Censor Counterspeech, but Teaching the Distinction Helps
Kristina Gligoric, Myra Cheng, Lucia Zheng +2
The use of words to convey speaker's intent is traditionally distinguished from the `mention' of words for quoting what someone said, or pointing out properties of a word. Here we…
Dialect prejudice predicts AI decisions about people's character, employability, and criminality
Valentin Hofmann, Pratyusha Ria Kalluri, Dan Jurafsky +1
Hundreds of millions of people now interact with language models, with uses ranging from serving as a writing aid to informing hiring decisions. Yet these language models are known…
AnthroScore: A Computational Linguistic Measure of Anthropomorphism
Myra Cheng, Kristina Gligoric, Tiziano Piccardi +1
Anthropomorphism, or the attribution of human-like characteristics to non-human entities, has shaped conversations about the impacts and possibilities of technology. We present Ant…
Grounding Gaps in Language Model Generations
Omar Shaikh, Kristina Gligorić, Ashna Khetan +3
Effective conversation requires common ground: a shared understanding between the participants. Common ground, however, does not emerge spontaneously in conversation. Speakers and…
Navigating the Grey Area: How Expressions of Uncertainty and Overconfidence Affect Language Models
Kaitlyn Zhou, Dan Jurafsky, Tatsunori Hashimoto
The increased deployment of LMs for real-world tasks involving knowledge and facts makes it important to understand model epistemology: what LMs think they know, and how their atti…
Multilingual BERT has an accent: Evaluating English influences on fluency in multilingual models
Isabel Papadimitriou, Kezia Lopez, Dan Jurafsky
While multilingual language models can improve NLP performance on low-resource languages by leveraging higher-resource languages, they also reduce average performance on all langua…