Publications (6)
Cost-of-Pass: An Economic Framework for Evaluating Language Models
Mehmet Hamza Erol, Batu El, Mirac Suzgun +2
Widespread adoption of AI systems hinges on their ability to generate economic value that outweighs their inference costs. Evaluating this tradeoff requires metrics accounting for…
Moloch's Bargain: Emergent Misalignment When LLMs Compete for Audiences
Batu El, James Zou
Large language models (LLMs) are increasingly shaping how information is created and disseminated, from companies using them to craft persuasive advertisements, to election campaig…
TERMS-Bench: Diagnosing LLM Negotiation Agents Beyond Deal Rate
Erica Zhang, Fangzhao Zhang, Aneesh Pappu +5
Negotiation is a central mechanism of economic exchange, shaping markets, procurement, labor agreements, and resource allocation. It is also a canonical testbed for agentic languag…
Multi-Agent Teams Hold Experts Back
Aneesh Pappu, Batu El, Hancheng Cao +4
Multi-agent LLM systems are increasingly deployed as autonomous collaborators, where agents interact freely rather than execute fixed, pre-specified workflows. In such settings, ef…
Inefficiencies of Meta Agents for Agent Design
Batu El, Mert Yuksekgonul, James Zou
Recent works began to automate the design of agentic systems using meta-agents that propose and iteratively refine new agent architectures. In this paper, we examine three key chal…
Towards Mechanistic Interpretability of Graph Transformers via Attention Graphs
Batu El, Deepro Choudhury, Pietro Liò +1
We introduce Attention Graphs, a new tool for mechanistic interpretability of Graph Neural Networks (GNNs) and Graph Transformers based on the mathematical equivalence between mess…