Publications (7)
Economic Evaluations of Language Models
Alexander Wan, Stephane Hatgis-Kessell, Tomás Aguirre +2
Language models perform economically valuable work, yet they are not currently assessed for how well they perform every economically valuable task. We introduce EconEvals as an ope…
Estimating time spent on work tasks
Stephane Hatgis-Kessell, Tomás Aguirre, Alexander Wan +1
The task-based framework in economics models occupations as bundles of tasks. It is the standard lens for understanding how technology affects work: a new technology changes the co…
The 2025 Foundation Model Transparency Index
Alexander Wan, Kevin Klyman, Sayash Kapoor +5
Foundation model developers are among the world's most important companies. As these companies become increasingly consequential, how do their transparency practices evolve? The 20…
What Evidence Do Language Models Find Convincing?
Alexander Wan, Eric Wallace, Dan Klein
Retrieval-augmented language models are being increasingly tasked with subjective, contentious, and conflicting queries such as "is aspartame linked to cancer". To resolve these am…
Poisoning Language Models During Instruction Tuning
Alexander Wan, Eric Wallace, Sheng Shen +1
Instruction-tuned LMs such as ChatGPT, FLAN, and InstructGPT are finetuned on datasets that contain user-submitted examples, e.g., FLAN aggregates numerous open-source datasets and…
The California Report on Frontier AI Policy
Rishi Bommasani, Scott R. Singer, Ruth E. Appel +20
The innovations emerging at the frontier of artificial intelligence (AI) are poised to create historic opportunities for humanity but also raise complex policy challenges. Continue…
GLUECons: A Generic Benchmark for Learning Under Constraints
Hossein Rajaby Faghihi, Aliakbar Nafar, Chen Zheng +7
Recent research has shown that integrating domain knowledge into deep learning architectures is effective -- it helps reduce the amount of required data, improves the accuracy of t…