15 citations · 24 across the 20 of their papers we have counts for
7 papers · 1 filter
Decentralized LLM Workflows via Sequential Auctions
Ian Gemp, Shubhendu Trivedi, Adam Fisch +7
We present a scalable framework for solving multi-step tasks through a decentralized market of self-interested agents. Prior theoretical work demonstrated that agents organized thr…
Your Transformer Is Secretly Elastic
Walter Simoncini, Dheeraj Varghese, Gertjan J. Burghouts +1
Scaling up transformers has produced models with broad capabilities, deployed across heterogeneous hardware and fluctuating workloads. Yet, model families offer only a few fixed si…
Pandora's AI Model Routing Box: Efficient Allocation with Costly Value Estimation
Adam Fisch, Shubhendu Trivedi, Fantine Huot +5
Heterogeneous AI systems composed of multiple models, architectures, harnesses, or inference-time settings can improve quality and efficiency by routing queries to the specialist w…
CollabEval: Statistically Efficient Collaborative Model Evaluation via Matrix Completion
Adam Fisch, Daniel Deutsch, Joshua Maynez +5
Evaluating generative AI models is a routine, but resource-intensive, process that is conducted over and over again during the course of model development. In this work, we propose…
Multiple-Prediction-Powered Inference
Charlie Cowen-Breen, Alekh Agarwal, Stephen Bates +4
Statistical estimation often involves tradeoffs between expensive, high-quality measurements and a variety of lower-quality proxies. We introduce Multiple-Prediction-Powered Infere…
MT-PingEval: Evaluating Multi-Turn Collaboration with Private Information Games
Jacob Eisenstein, Fantine Huot, Adam Fisch +2
We present a scalable and verifiable methodology for evaluating language models in multi-turn interactions, using a suite of collaborative games that require effective communicatio…
Learning Steerable Clarification Policies with Collaborative Self-play
Jonathan Berant, Maximillian Chen, Adam Fisch +4
To handle underspecified or ambiguous queries, AI assistants need a policy for managing their uncertainty to determine (a) when to guess the user intent and answer directly, (b) wh…