9 papers
Levels of Analysis for Large Language Models
Alexander Y. Ku, Declan Campbell, Xuechunzi Bai +10
Modern artificial intelligence systems, such as large language models, are increasingly powerful but also increasingly hard to understand. Recognizing this problem as analogous to…
Evaluating Language Model Agency through Negotiations
Tim R. Davidson, Veniamin Veselovsky, Martin Josifoski +4
We introduce an approach to evaluate language model (LM) agency using negotiation games. This approach better reflects real-world use cases and addresses some of the shortcomings o…
Capturing Dynamics in Online Public Discourse: A Case Study of Universal Basic Income Discussions on Reddit
Rachel Kim, Veniamin Veselovsky, Ashton Anderson
Societal change is often driven by shifts in public opinion. As citizens evolve in their norms, beliefs, and values, public policies change too. While traditional opinion polling a…
Separating Tongue from Thought: Activation Patching Reveals Language-Agnostic Concept Representations in Transformers
Clément Dumas, Chris Wendler, Veniamin Veselovsky +2
A central question in multilingual language modeling is whether large language models (LLMs) develop a universal concept representation, disentangled from specific languages. In th…
Web2Wiki: Characterizing Wikipedia Linking Across the Web
Veniamin Veselovsky, Tiziano Piccardi, Ashton Anderson +2
Wikipedia is one of the most visited websites globally, yet its role beyond its own platform remains largely unexplored. In this paper, we present the first large-scale analysis of…
Identifying and Mitigating the Influence of the Prior Distribution in Large Language Models
Liyi Zhang, Veniamin Veselovsky, R. Thomas McCoy +1
Large language models (LLMs) sometimes fail to respond appropriately to deterministic tasks -- such as counting or forming acronyms -- because the implicit prior distribution they…