13 papers
Evaluating for the long term: Learnings from industry
Leif Sigerson, Tom Cunningham, Winston Chou +22
Online platforms prioritize long-term business outcomes, yet typical experiments are far too short to measure these outcomes directly. Our goal in this paper is to collect and shar…
Offline-to-Online Creative Optimization with Generative Models and Adaptive Testing
Kevin Lee, Benjamin Letham, Zhiyuan Jerry Lin +5
Ad creative optimization is increasingly constrained by evaluation rather than generation. Generative models can produce many plausible creatives, but reliable evaluation requires…
Pitfalls and Remedies for Multi-Task Bayesian Optimization
Carl Hvarfner, Sam Daulton, Max Balandat +1
Bayesian optimization routinely warm-starts a target experiment with data from related source tasks, and the multi-task Gaussian process is the textbook surrogate for the job. We r…
-PFN: Fast Entropy Search via In-Context Learning
Herilalaina Rakotoarison, Steven Adriaensen, Tom Viering +4
Information-theoretic acquisition functions such as Entropy Search (ES) offer a principled exploration-exploitation framework for Bayesian optimization (BO). However, their practic…
Embedding by Elicitation: Dynamic Representations for Bayesian Optimization of System Prompts
Zhiyuan Jerry Lin, Benjamin Letham, Samuel Dooley +2
System prompts are a central control mechanism in modern AI systems, shaping behavior across conversations, tasks, and user populations. Yet they are difficult to tune when feedbac…
LILO: Bayesian Optimization with Natural Language Feedback
Katarzyna Kobalczyk, Zhiyuan Jerry Lin, Benjamin Letham +3
Many real-world optimization problems are guided by complex, subjective preferences that are difficult to express as explicit closed-form objectives. In response, we introduce Lang…