7 citations · 8 across the 11 of their papers we have counts for
13 papers
Keeping an Eye on AI: A Framework for Effective Human Oversight of AI Systems
Susanne Gaube, Markus Langer, Tim Miller +17
The use of Artificial Intelligence (AI) in high-risk, decision-making scenarios presents technical, safety, and normative challenges; problems that may only be ameliorated by human…
The Story is Not the Science: Execution-Grounded Evaluation of Mechanistic Interpretability Research
Xiaoyan Bai, Alexander Baumgartner, Haojia Sun +2
Reproducibility crises across sciences highlight the limitations of the paper-centric review system in assessing the rigor and reproducibility of research. AI agents that autonomou…
Human-AI Collaboration in Radiology: The Case of Pulmonary Embolism
Paul Goldsmith-Pinkham, Chenhao Tan, Alexander K. Zentefis
We study how radiologists use AI to diagnose pulmonary embolism (PE), tracking over 100,000 scans interpreted by nearly 400 radiologists during the staggered rollout of a real-worl…
Know Thyself? On the Incapability and Implications of AI Self-Recognition
Xiaoyan Bai, Aryan Shrivastava, Ari Holtzman +1
Self-recognition is a crucial metacognitive capability for AI systems, relevant not only for psychological analysis but also for safety, particularly in evaluative scenarios. Motiv…
Why Can't Transformers Learn Multiplication? Reverse-Engineering Reveals Long-Range Dependency Pitfalls
Xiaoyan Bai, Itamar Pres, Yuntian Deng +5
Language models are increasingly capable, yet still fail at a seemingly simple task of multi-digit multiplication. In this work, we study why, by reverse-engineering a model that s…
MoVa: Towards Generalizable Classification of Human Morals and Values
Ziyu Chen, Junfei Sun, Chenxi Li +6
Identifying human morals and values embedded in language is essential to empirical studies of communication. However, researchers often face substantial difficulty navigating the d…
Prompting as Scientific Inquiry
Ari Holtzman, Chenhao Tan
Prompting is the primary method by which we study and control large language models. It is also one of the most powerful: nearly every major capability attributed to LLMs-few-shot…
"I Cannot Write This Because It Violates Our Content Policy": Understanding Content Moderation Policies and User Experiences in Generative AI Products
Lan Gao, Oscar Chen, Rachel Lee +3
While recent research has focused on developing safeguards for generative AI (GAI) model-level content safety, little is known about how content moderation to prevent malicious con…
AbsenceBench: Language Models Can't Tell What's Missing
Harvey Yiyun Fu, Aryan Shrivastava, Jared Moore +3
Large language models (LLMs) are increasingly capable of processing long inputs and locating specific information within them, as evidenced by their performance on the Needle in a…
Concept Incongruence: An Exploration of Time and Death in Role Playing
Xiaoyan Bai, Ike Peng, Aditya Singh +1
Consider this prompt "Draw a unicorn with two horns". Should large language models (LLMs) recognize that a unicorn has only one horn by definition and ask users for clarifications,…
Towards a Client-Centered Assessment of LLM Therapists by Client Simulation
Jiashuo Wang, Yang Xiao, Yanran Li +4
Although there is a growing belief that LLMs can be used as therapists, exploring LLMs' capabilities and inefficacy, particularly from the client's perspective, is limited. This wo…
The Impossibility of Fair LLMs
Jacy Anthis, Kristian Lum, Michael Ekstrand +2
The rise of general-purpose artificial intelligence (AI) systems, particularly large language models (LLMs), has raised pressing moral questions about how to reduce bias and ensure…
Ecologically Valid Explanations for Label Variation in NLI
Nan-Jiang Jiang, Chenhao Tan, Marie-Catherine de Marneffe
Human label variation, or annotation disagreement, exists in many natural language processing (NLP) tasks, including natural language inference (NLI). To gain direct evidence of ho…