7 citations · 16 across the 8 of their papers we have counts for
9 papers
AI Behavioral Science: A Framework and Agenda
Matthew O. Jackson, Qiaozhu Me, Stephanie W. Wang +16
We discuss the challenges and opportunities present in the rapidly emerging area of ``AI Behavioral Science.'' We frame it via three subfields. First, as AI becomes ubiquitous and…
What Has a Foundation Model Found? Using Inductive Bias to Probe for World Models
Keyon Vafa, Peter G. Chang, Ashesh Rambachan +1
Foundation models are premised on the idea that sequence prediction can uncover deeper domain understanding, much like how Kepler's predictions of planetary motion later led to the…
Potemkin Understanding in Large Language Models
Marina Mancoridis, Bec Weeks, Keyon Vafa +1
Large language models (LLMs) are regularly evaluated using benchmark datasets. But what justifies making inferences about an LLM's capabilities based on its answers to a curated se…
What's Producible May Not Be Reachable: Measuring the Steerability of Generative Models
Keyon Vafa, Sarah Bentley, Jon Kleinberg +1
How should we evaluate the quality of generative models? Many existing metrics focus on a model's producibility, i.e. the quality and breadth of outputs it can generate. However, t…
Large Language Models: An Applied Econometric Framework
Jens Ludwig, Sendhil Mullainathan, Ashesh Rambachan
Large language models (LLMs) enable researchers to analyze text at unprecedented scale and minimal cost. Researchers can now revisit old questions and tackle novel ones with rich d…
Do Large Language Models Perform the Way People Expect? Measuring the Human Generalization Function
Keyon Vafa, Ashesh Rambachan, Sendhil Mullainathan
What makes large language models (LLMs) impressive is also what makes them hard to evaluate: their diversity of uses. To evaluate these models, we must understand the purposes they…
Evaluating the World Model Implicit in a Generative Model
Keyon Vafa, Justin Y. Chen, Ashesh Rambachan +2
Recent work suggests that large language models may implicitly learn world models. How should we assess this possibility? We formalize this question for the case where the underlyi…
Language Generation in the Limit
Jon Kleinberg, Sendhil Mullainathan
Although current large language models are complex, the most basic specifications of the underlying language generation problem itself are simple to state: given a finite set of tr…
Using large language models to promote health equity
Emma Pierson, Divya Shanmugam, Rajiv Movva +12
Advances in large language models (LLMs) have driven an explosion of interest about their societal impacts. Much of the discourse around how they will impact social equity has been…