2.3k citations · 11.9k across the 167 of their papers we have counts for
24 papers · 1 filter
Trustworthy Social Bias Measurement
Rishi Bommasani, Percy Liang
How do we design measures of social bias that we trust? While prior work has introduced several measures, no measure has gained widespread trust: instead, mounting evidence argues…
Melody transcription via generative pre-training
Chris Donahue, John Thickstun, Percy Liang
Despite the central role that melody plays in music perception, it remains an open challenge in music information retrieval to reliably detect the notes of the melody present in an…
Evaluating Human-Language Model Interaction
Mina Lee, Megha Srivastava, Amelia Hardy +15
Many real-world applications of language models (LMs), such as writing assistance and code autocomplete, involve human-LM interaction. However, most benchmarks are non-interactive…
Picking on the Same Person: Does Algorithmic Monoculture lead to Outcome Homogenization?
Rishi Bommasani, Kathleen A. Creel, Ananya Kumar +2
As the scope of machine learning broadens, we observe a recurring theme of algorithmic monoculture: the same systems, or systems that share components (e.g. training data), are dep…
How do Authors' Perceptions of their Papers Compare with Co-authors' Perceptions and Peer-review Decisions?
Charvi Rastogi, Ivan Stelmakh, Alina Beygelzimer +7
How do author perceptions match up to the outcomes of the peer-review process and perceptions of others? In a top-tier computer science conference (NeurIPS 2021) with more than 23,…
Retrieval-Augmented Multimodal Language Modeling
Michihiro Yasunaga, Armen Aghajanyan, Weijia Shi +6
Recent multimodal models such as DALL-E and CM3 have achieved remarkable progress in text-to-image and image-to-text generation. However, these models store all learned knowledge (…
Truncation Sampling as Language Model Desmoothing
John Hewitt, Christopher D. Manning, Percy Liang
Long samples of text from neural language models can be of poor quality. Truncation sampling algorithms--like top- or top- -- address this by setting some words' probabilitie…
Deep Bidirectional Language-Knowledge Graph Pretraining
Michihiro Yasunaga, Antoine Bosselut, Hongyu Ren +4
Pretraining a language model (LM) on text has been shown to help various downstream NLP tasks. Recent works show that a knowledge graph (KG) can complement text data, offering stru…
Contrastive Decoding: Open-ended Text Generation as Optimization
Xiang Lisa Li, Ari Holtzman, Daniel Fried +5
Given a language model (LM), maximum probability is a poor decoding objective for open-ended generation, because it produces short and repetitive text. On the other hand, sampling…
Surgical Fine-Tuning Improves Adaptation to Distribution Shifts
Yoonho Lee, Annie S. Chen, Fahim Tajwar +4
A common approach to transfer learning under distribution shift is to fine-tune the last few layers of a pre-trained model, preserving learned features while also adapting to the n…
Are Sample-Efficient NLP Models More Robust?
Nelson F. Liu, Ananya Kumar, Percy Liang +1
Recent results in image classification and extractive question answering have observed that pre-trained models trained on less in-distribution data have better out-of-distribution…
Improving Self-Supervised Learning by Characterizing Idealized Representations
Yann Dubois, Tatsunori Hashimoto, Stefano Ermon +1
Despite the empirical successes of self-supervised learning (SSL) methods, it is unclear what characteristics of their representations lead to high downstream accuracies. In this w…
Social Simulacra: Creating Populated Prototypes for Social Computing Systems
Joon Sung Park, Lindsay Popowski, Carrie J. Cai +3
Social computing prototypes probe the social behaviors that may arise in an envisioned system design. This prototyping practice is currently limited to recruiting small groups of p…
What Can Transformers Learn In-Context? A Case Study of Simple Function Classes
Shivam Garg, Dimitris Tsipras, Percy Liang +1
In-context learning refers to the ability of a model to condition on a prompt sequence consisting of in-context examples (input-output pairs corresponding to some task) along with…
Calibrated ensembles can mitigate accuracy tradeoffs under distribution shift
Ananya Kumar, Tengyu Ma, Percy Liang +1
We often see undesirable tradeoffs in robust machine learning where out-of-distribution (OOD) accuracy is at odds with in-distribution (ID) accuracy: a robust classifier obtained v…
Is a Caption Worth a Thousand Images? A Controlled Study for Representation Learning
Shibani Santurkar, Yann Dubois, Rohan Taori +2
The development of CLIP [Radford et al., 2021] has sparked a debate on whether language supervision can result in vision models with more transferable representations than traditio…
Insights into Pre-training via Simpler Synthetic Tasks
Yuhuai Wu, Felix Li, Percy Liang
Pre-training produces representations that are effective for a wide range of downstream tasks, but it is still unclear what properties of pre-training are necessary for effective g…
Decentralized Training of Foundation Models in Heterogeneous Environments
Binhang Yuan, Yongjun He, Jared Quincy Davis +6
Training foundation models, such as GPT-3 and PaLM, can be extremely expensive, often involving tens of thousands of GPUs running continuously for months. These models are typicall…
Diffusion-LM Improves Controllable Text Generation
Xiang Lisa Li, John Thickstun, Ishaan Gulrajani +2
Controlling the behavior of language models (LMs) without re-training is a major open problem in natural language generation. While recent works have demonstrated successes on cont…
Connect, Not Collapse: Explaining Contrastive Learning for Unsupervised Domain Adaptation
Kendrick Shen, Robbie Jones, Ananya Kumar +4
We consider unsupervised domain adaptation (UDA), where labeled data from a source domain (e.g., photographs) and unlabeled data from a target domain (e.g., sketches) are used to l…
LinkBERT: Pretraining Language Models with Document Links
Michihiro Yasunaga, Jure Leskovec, Percy Liang
Language model (LM) pretraining can learn various knowledge from text corpora, helping downstream tasks. However, existing methods such as BERT model a single document, and do not…
Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution
Ananya Kumar, Aditi Raghunathan, Robbie Jones +2
When transferring a pretrained model to a downstream task, two popular methods are full fine-tuning (updating all the model parameters) and linear probing (updating only the last l…
CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities
Mina Lee, Percy Liang, Qian Yang
Large language models (LMs) offer unprecedented language generation capabilities and exciting opportunities for interaction design. However, their highly context-dependent capabili…
GreaseLM: Graph REASoning Enhanced Language Models for Question Answering
Xikun Zhang, Antoine Bosselut, Michihiro Yasunaga +4
Answering complex questions about textual narratives requires reasoning over both stated context and the world knowledge that underlies it. However, pretrained language models (LM)…