37 citations · 78 across the 38 of their papers we have counts for
41 papers
Always-On Experimentation
Ricardo J. Sandoval, David Arbour, Avi Feller +1
Generative AI has dramatically accelerated the rate at which new treatments---from novel pharmaceuticals to online marketing campaigns---can be conceived and deployed. As a result,…
Toward Latent Language Model Skills Steering and Optimization: An Empirical Study
Xunyi Jiang, Junda Wu, Yuxin Xiong +6
Skills, as a useful abstraction for the procedural capabilities of large language models (LLMs), capture how models perform structured, multi-step reasoning and program execution.…
BACON: Budgeted Human Calibration for Modeling and Evaluation with Multiple AI Judges
Lei Shi, Anlan Zhang, Rita Lyu +6
AI judges offer a scalable, low-cost alternative to human evaluation, but their outputs can be biased relative to human preferences and highly item-dependent, varying across judges…
The Curse of Multiple Mediators: Hidden Interaction Effects in Activation Patching
Sankaran Vaidyanathan, David Arbour, Aaron Mueller +2
Activation patching is the primary tool in mechanistic interpretability. It attributes causal responsibility for a model behavior to each of its individual components by estimating…
AI-Assisted Variance Reduction in Randomized Experiments
David Arbour, Eli Ben-Michael, Avi Feller +2
Generative AI and large language models can produce realistic predictions of human behavior from rich, unstructured inputs with little to no task-specific training data. Recent wor…
Experimentation Accelerator: Interpretable Insights and Creative Recommendations for A/B Testing with Content-Aware ranking
Zhengmian Hu, Lei Shi, Ritwik Sinha +2
Modern online experimentation faces two bottlenecks: scarce traffic forces tough choices on which variants to test, and post-hoc insight extraction is manual, inconsistent, and oft…
Leveraging semantic similarity for experimentation with AI-generated treatments
Lei Shi, David Arbour, Raghavendra Addanki +2
Large Language Models (LLMs) enable a new form of digital experimentation where treatments combine human and model-generated content in increasingly sophisticated ways. The main me…
Handling Missing Responses under Cluster Dependence with Applications to Language Model Evaluation
Zhenghao Zeng, David Arbour, Avi Feller +3
Human annotations play a crucial role in evaluating the performance of GenAI models. Two common challenges in practice, however, are missing annotations (the response variable of i…
Regularizing Extrapolation in Causal Inference
David Arbour, Harsh Parikh, Bijan Niknam +3
Many common estimators in machine learning and causal inference are linear smoothers, where the prediction is a weighted average of the training outcomes. Some estimators, such as…
Relational Causal Discovery with Latent Confounders
Matteo Negro, Andrea Piras, Ragib Ahsan +2
Estimating causal effects from real-world relational data can be challenging when the underlying causal model and potential confounders are unknown. While several causal discovery…
Sequential Causal Discovery with Noisy Language Model Priors
Prakhar Verma, David Arbour, Sunav Choudhary +3
Causal discovery from observational data typically assumes access to complete data and availability of perfect domain experts. In practice, data often arrive in batches, are subjec…
Principled Content Selection to Generate Diverse and Personalized Multi-Document Summaries
Vishakh Padmakumar, Zichao Wang, David Arbour +1
While large language models (LLMs) are increasingly capable of handling longer contexts, recent work has demonstrated that they exhibit the "lost in the middle" phenomenon (Liu et…
Evaluation and Incident Prevention in an Enterprise AI Assistant
Akash V. Maharaj, David Arbour, Daniel Lee +6
Enterprise AI Assistants are increasingly deployed in domains where accuracy is paramount, making each erroneous output a potentially significant incident. This paper presents a co…
Continuous Treatment Effects with Surrogate Outcomes
Zhenghao Zeng, David Arbour, Avi Feller +4
In many real-world causal inference applications, the primary outcomes (labels) are often partially missing, especially if they are expensive or difficult to collect. If the missin…
Leveraging Graph Diffusion Models for Network Refinement Tasks
Puja Trivedi, Ryan Rossi, David Arbour +7
Most real-world networks are noisy and incomplete samples from an unknown target distribution. Refining them by correcting corruptions or inferring unobserved regions typically imp…
Distributional Off-Policy Evaluation for Slate Recommendations
Shreyas Chaudhari, David Arbour, Georgios Theocharous +1
Recommendation strategies are typically evaluated by using previously logged data, employing off-policy evaluation methods to estimate their expected performance. However, for stra…
Dynamic Vector Bin Packing for Online Resource Allocation in the Cloud
Aniket Murhekar, David Arbour, Tung Mai +1
Several cloud-based applications, such as cloud gaming, rent servers to execute jobs which arrive in an online fashion. Each job has a resource demand and must be dispatched to a c…
Anytime-Valid Confidence Sequences in an Enterprise A/B Testing Platform
Akash V. Maharaj, Ritwik Sinha, David Arbour +7
A/B tests are the gold standard for evaluating digital experiences on the web. However, traditional "fixed-horizon" statistical methods are often incompatible with the needs of mod…
Sample Constrained Treatment Effect Estimation
Raghavendra Addanki, David Arbour, Tung Mai +2
Treatment effect estimation is a fundamental problem in causal inference. We focus on designing efficient randomized controlled trials, to accurately estimate the effect of some tr…
Learning Relational Causal Models with Cycles through Relational Acyclification
Ragib Ahsan, David Arbour, Elena Zheleva
In real-world phenomena which involve mutual influence or causal effects between interconnected units, equilibrium states are typically represented with cycles in graphical models.…
Non-Parametric Inference of Relational Dependence
Ragib Ahsan, Zahra Fatemi, David Arbour +1
Independence testing plays a central role in statistical and causal inference from observational data. Standard independence tests assume that the data samples are independent and…
Offline Evaluation of Ranked Lists using Parametric Estimation of Propensities
Vishwa Vinay, Manoj Kilaru, David Arbour
Search engines and recommendation systems attempt to continually improve the quality of the experience they afford to their users. Refining the ranker that produces the lists displ…
Bayesian Modeling of Marketing Attribution
Ritwik Sinha, David Arbour, Aahlad Manas Puli
In a multi-channel marketing world, the purchase decision journey encounters many interactions (e.g., email, mobile notifications, display advertising, social media, and so on). Th…
Online Balanced Experimental Design
David Arbour, Drew Dimmery, Tung Mai +1
e consider the experimental design problem in an online environment, an important practical task for reducing the variance of estimates in randomized experiments which allows for g…