2.3k citations · 4.6k across the 38 of their papers we have counts for
38 papers
LLM Detection as an Intervention: Downstream Impact under Strategic User Behavior
Meena Jagadeesan, Tatsunori Hashimoto, Jon Kleinberg
As LLM adoption becomes more widespread, there is a growing interest in detecting LLM-generated content, for example through LLM detection tools and through heuristics based on lan…
The Troubling Emergence of Hallucination in Large Language Models -- An Extensive Definition, Quantification, and Prescriptive Remediations
Vipula Rawte, Swagata Chakraborty, Agnibh Pathak +5
The recent advancements in Large Language Models (LLMs) have garnered widespread acclaim for their remarkable emerging capabilities. However, the issue of hallucination has paralle…
Identifying the Risks of LM Agents with an LM-Emulated Sandbox
Yangjun Ruan, Honghua Dong, Andrew Wang +6
Recent advances in Language Model (LM) agents and tool use, exemplified by applications like ChatGPT Plugins, enable a rich set of capabilities but also amplify potential risks - s…
Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions
Federico Bianchi, Mirac Suzgun, Giuseppe Attanasio +4
Training large language models to follow instructions makes them perform better on a wide range of tasks and generally become more helpful. However, a perfectly helpful model will…
Where's the Liability in Harmful AI Speech?
Peter Henderson, Tatsunori Hashimoto, Mark Lemley
Generative AI, in particular text-based "foundation models" (large models trained on a huge variety of information including the internet), can generate speech that could be proble…
Identifying and Mitigating the Security Risks of Generative AI
Clark Barrett, Brad Boyd, Elie Burzstein +20
Every major technical invention resurfaces the dual-use dilemma -- the new technology has the potential to be used for good as well as for harm. Generative AI (GenAI) techniques, s…
One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attention
Arvind Mahankali, Tatsunori B. Hashimoto, Tengyu Ma
Recent works have empirically analyzed in-context learning and shown that transformers trained on synthetic linear regression tasks can learn to implement ridge regression, which i…
Robust Distortion-free Watermarks for Language Models
Rohith Kuditipudi, John Thickstun, Tatsunori Hashimoto +1
We propose a methodology for planting watermarks in text from an autoregressive language model that are robust to perturbations without changing the distribution over text up to a…
Likelihood-Based Diffusion Language Models
Ishaan Gulrajani, Tatsunori B. Hashimoto
Despite a growing interest in diffusion-based language models, existing work has not shown that these models can attain nontrivial likelihoods on standard language modeling benchma…
AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback
Yann Dubois, Xuechen Li, Rohan Taori +6
Large language models (LLMs) such as ChatGPT have seen widespread adoption due to their strong instruction-following abilities. Developing these LLMs involves a complex yet poorly…
Whose Opinions Do Language Models Reflect?
Shibani Santurkar, Esin Durmus, Faisal Ladhak +3
Language models (LMs) are increasingly being used in open-ended contexts, where the opinions reflected by LMs in response to subjective queries can have a profound impact, both on…
Foundation Models and Fair Use
Peter Henderson, Xuechen Li, Dan Jurafsky +3
Existing foundation models are trained on copyrighted material. Deploying these models can pose both legal and ethical risks when data creators fail to receive appropriate attribut…
Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attacks
Daniel Kang, Xuechen Li, Ion Stoica +3
Recent advances in instruction-following large language models (LLMs) have led to dramatic improvements in a range of NLP tasks. Unfortunately, we find that the same improved capab…
Out-of-Domain Robustness via Targeted Augmentations
Irena Gao, Shiori Sagawa, Pang Wei Koh +2
Models trained on one set of domains often suffer performance drops on unseen domains, e.g., when wildlife monitoring models are deployed in new camera locations. In this work, we…
Evaluating Self-Supervised Learning via Risk Decomposition
Yann Dubois, Tatsunori Hashimoto, Percy Liang
Self-supervised learning (SSL) pipelines differ in many design choices such as the architecture, augmentations, or pretraining data. Yet SSL is typically evaluated using a single m…
Benchmarking Large Language Models for News Summarization
Tianyi Zhang, Faisal Ladhak, Esin Durmus +3
Large language models (LLMs) have shown promise for automatic summarization but the reasons behind their successes are poorly understood. By conducting a human evaluation on ten LL…
Contrastive Error Attribution for Finetuned Language Models
Faisal Ladhak, Esin Durmus, Tatsunori Hashimoto
Recent work has identified noisy and misannotated data as a core cause of hallucinations and unfaithful outputs in Natural Language Generation (NLG) tasks. Consequently, identifyin…
Privacy-Preserving Domain Adaptation of Semantic Parsers
Fatemehsadat Mireshghallah, Yu Su, Tatsunori Hashimoto +2
Task-oriented dialogue systems often assist users with personal or confidential matters. For this reason, the developers of such a system are generally prohibited from observing ac…
Coder Reviewer Reranking for Code Generation
Tianyi Zhang, Tao Yu, Tatsunori B. Hashimoto +4
Sampling diverse programs from a code language model and reranking with model likelihood is a popular method for code generation but it is prone to preferring degenerate solutions.…
A Closer Look at the Calibration of Differentially Private Learners
Hanlin Zhang, Xuechen Li, Prithviraj Sen +2
We systematically study the calibration of classifiers trained with differentially private stochastic gradient descent (DP-SGD) and observe miscalibration across a wide range of vi…
Holistic Evaluation of Language Models
Percy Liang, Rishi Bommasani, Tony Lee +47
Language models (LMs) are becoming the foundation for almost all major language technologies, but their capabilities, limitations, and risks are not well understood. We present Hol…
Easily Accessible Text-to-Image Generation Amplifies Demographic Stereotypes at Large Scale
Federico Bianchi, Pratyusha Kalluri, Esin Durmus +7
Machine learning models that convert user-written text descriptions into images are now widely available online and used by millions of users to generate millions of images a day.…
Contrastive Decoding: Open-ended Text Generation as Optimization
Xiang Lisa Li, Ari Holtzman, Daniel Fried +5
Given a language model (LM), maximum probability is a poor decoding objective for open-ended generation, because it produces short and repetitive text. On the other hand, sampling…
Data Feedback Loops: Model-driven Amplification of Dataset Biases
Rohan Taori, Tatsunori B. Hashimoto
Datasets scraped from the internet have been critical to the successes of large-scale machine learning. Yet, this very success puts the utility of future internet-derived datasets…