45 citations · 96 across the 11 of their papers we have counts for
12 papers
Reasoning Models are Test Exploiters: Rethinking Multiple-Choice
Narun Raman, Taylor Lundy, Kevin Leyton-Brown
When evaluating Large Language Models (LLMs) in question answering domains, it is common to ask the model to choose among a fixed set of choices (so-called multiple-choice question…
How malicious AI swarms can threaten democracy: The fusion of agentic AI and LLMs marks a new frontier in information warfare
Daniel Thilo Schroeder, Meeyoung Cha, Andrea Baronchelli +19
Advances in AI offer the prospect of manipulating beliefs and behaviors on a population-wide level. Large language models and autonomous agents now let influence campaigns reach un…
NFTs as a Data-Rich Test Bed: Conspicuous Consumption and its Determinants
Taylor Lundy, Narun Raman, Scott Duke Kominers +1
Conspicuous consumption occurs when a consumer derives value from a good based on its social meaning as a signal of wealth, taste, and/or community affiliation. Common conspicuous…
STEER-ME: Assessing the Microeconomic Reasoning of Large Language Models
Narun Raman, Taylor Lundy, Thiago Amin +2
How should one judge whether a given large language model (LLM) can reliably perform economic reasoning? Most existing LLM benchmarks focus on specific applications and fail to pre…
Understanding Understanding: A Pragmatic Framework Motivated by Large Language Models
Kevin Leyton-Brown, Yoav Shoham
Motivated by the rapid ascent of Large Language Models (LLMs) and debates about the extent to which they possess human-level qualities, we propose a framework for testing whether a…
What Can Natural Language Processing Do for Peer Review?
Ilia Kuznetsov, Osama Mohammed Afzal, Koen Dercksen +21
The number of scientific articles produced every year is growing rapidly. Providing quality control over them is crucial for scientists and, ultimately, for the public good. In mod…
STEER: Assessing the Economic Rationality of Large Language Models
Narun Raman, Taylor Lundy, Samuel Amouyal +3
There is increasing interest in using LLMs as decision-making "agents." Doing so includes many degrees of freedom: which model should be used; how should it be prompted; should it…
Pay to (Not) Play: Monetizing Impatience in Mobile Games
Taylor Lundy, Narun Raman, Hu Fu +1
Mobile gaming is a rapidly growing and incredibly profitable sector; having grown seven-fold over the past 10 years, it now grosses over $100 billion annually. This growth was due…
Generating Benchmarks for Factuality Evaluation of Language Models
Dor Muhlgay, Ori Ram, Inbal Magar +7
Before deploying a language model (LM) within a given domain, it is important to measure its tendency to generate factually incorrect information in that domain. Existing methods f…
In-Context Retrieval-Augmented Language Models
Ori Ram, Yoav Levine, Itay Dalmedigos +4
Retrieval-Augmented Language Modeling (RALM) methods, which condition a language model (LM) on relevant documents from a grounding corpus during generation, were shown to significa…
Parallel Context Windows for Large Language Models
Nir Ratner, Yoav Levine, Yonatan Belinkov +7
When applied to processing long text, Large Language Models (LLMs) are limited by their context window. Existing efforts to address this limitation involve training specialized arc…
MRKL Systems: A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoning
Ehud Karpas, Omri Abend, Yonatan Belinkov +14
Huge language models (LMs) have ushered in a new era for AI, serving as a gateway to natural-language-based knowledge tasks. Although an essential element of modern AI, LMs are als…