361 citations · 1.2k across the 36 of their papers we have counts for
38 papers
When Representative Samples Produce Worse Outcomes: Scale-up Decisions and Testing in Small-Budget RCTs
Hannah Li, Hongseok Namkoong, Isaac Scheinfeld
Small randomized controlled trials are often used to screen interventions before running larger follow-up studies. This is a critical phase of experimentation, as missing effective…
Local Sensitivity Under Transport Restrictions
Hongseok Namkoong
We quantify the value of structural knowledge, restrictions a modeler places on the world before seeing data. Our analytic workhorse is the local sensitivity of an estimand to dist…
Empirical Likelihood for Nonsmooth Functionals
Hongseok Namkoong
Empirical likelihood is an attractive inferential framework that respects natural parameter boundaries, but existing approaches typically require smoothness of the functional and m…
LLM Swiss Round: Aggregating Multi-Benchmark Performance via Competitive Swiss-System Dynamics
Jiashuo Liu, Jiayun Wu, Chunjie Wu +5
The rapid proliferation of Large Language Models (LLMs) and diverse specialized benchmarks necessitates a shift from fragmented, task-specific metrics to a holistic, competitive ra…
FinSearchComp: Towards a Realistic, Expert-Level Evaluation of Financial Search and Reasoning
Liang Hu, Jianpeng Jiao, Jiashuo Liu +20
Search has emerged as core infrastructure for LLM-based agents and is widely viewed as critical on the path toward more general intelligence. Finance is a particularly demanding pr…
DRO: A Python Library for Distributionally Robust Optimization in Machine Learning
Jiashuo Liu, Tianyu Wang, Henry Lam +2
We introduce dro, an open-source Python library for distributionally robust optimization (DRO) for regression and classification problems. The library implements 15 DRO formulation…
From Models to Systems: A Comprehensive Fairness Framework for Compositional Recommender Systems
Brian Hsu, Cyrus DiCiccio, Natesh Sivasubramoniapillai +1
Fairness research in machine learning often centers on ensuring equitable performance of individual models. However, real-world recommendation systems are built on multiple models…
LLM Embeddings Improve Test-time Adaptation to Tabular -Shifts
Yibo Zeng, Jiashuo Liu, Henry Lam +1
For tabular datasets, the change in the relationship between the label and covariates (-shifts) is common due to missing variables (a.k.a. confounders). Since it is impossible…
QGym: Scalable Simulation and Benchmarking of Queuing Network Controllers
Haozhe Chen, Ang Li, Ethan Che +3
Queuing network control determines the allocation of scarce resources to manage congestion, a fundamental problem in manufacturing, communications, and healthcare. Compared to stan…
Differentiable Discrete Event Simulation for Queuing Network Control
Ethan Che, Jing Dong, Hongseok Namkoong
Queuing network control is essential for managing congestion in job-processing systems such as service systems, communication networks, and manufacturing processes. Despite growing…
PersonalLLM: Tailoring LLMs to Individual Preferences
Thomas P. Zollo, Andrew Wei Tung Siah, Naimeng Ye +2
As LLMs become capable of complex tasks, there is growing potential for personalized interactions tailored to the subtle and idiosyncratic preferences of the user. We present a pub…
AExGym: Benchmarks and Environments for Adaptive Experimentation
Jimmy Wang, Ethan Che, Daniel R. Jiang +1
Innovations across science and industry are evaluated using randomized trials (a.k.a. A/B tests). While simple and robust, such static designs are inefficient or infeasible for tes…
Optimization-Driven Adaptive Experimentation
Ethan Che, Daniel R. Jiang, Hongseok Namkoong +1
Real-world experiments involve batched & delayed feedback, non-stationarity, multiple objectives & constraints, and (often some) personalization. Tailoring adaptive methods to addr…
Exchangeable Sequence Models Quantify Uncertainty Over Latent Concepts
Naimeng Ye, Hongseok Namkoong
Intelligent agents must be able to articulate its own uncertainty. In this work, we show that pre-trained sequence models are naturally capable of probabilistic reasoning over exch…
Evaluating Model Performance Under Worst-case Subpopulations
Mike Li, Daksh Mittal, Hongseok Namkoong +1
The performance of ML models degrades when the training population is different from that seen under operation. Towards assessing distributional robustness, we study the worst-case…
Design and Scheduling of an AI-based Queueing System
Jiung Lee, Hongseok Namkoong, Yibo Zeng
To leverage prediction models to make optimal scheduling decisions in service systems, we must understand how predictive errors impact congestion due to externalities on the delay…
Active Exploration via Autoregressive Generation of Missing Data
Tiffany Tianhui Cai, Hongseok Namkoong, Daniel Russo +1
We pose uncertainty quantification and exploration in online decision-making as a problem of training and generation from an autoregressive sequence model, an area experiencing rap…
Rethinking Distribution Shifts: Empirical Analysis and Modeling for Tabular Data
Tianyu Wang, Jiashuo Liu, Peng Cui +1
Different distribution shifts require different interventions, and algorithms must be grounded in the specific shifts they address. However, methodological development for robust a…
Modeling Interference Using Experiment Roll-out
Ariel Boyarsky, Hongseok Namkoong, Jean Pouget-Abadie
Experiments on online marketplaces and social networks suffer from interference, where the outcome of a unit is impacted by the treatment status of other units. We propose a framew…
An Operational Perspective to Fairness Interventions: Where and How to Intervene
Brian Hsu, Xiaotong Chen, Ying Han +2
As AI-based decision systems proliferate, their successful operationalization requires balancing multiple desiderata: predictive performance, disparity across groups, safeguarding…
Adaptive Experimentation at Scale: A Computational Framework for Flexible Batches
Ethan Che, Hongseok Namkoong
Standard bandit algorithms that assume continual reallocation of measurement effort are challenging to implement due to delayed feedback and infrastructural/organizational difficul…
Diagnosing Model Performance Under Distribution Shift
Tiffany Tianhui Cai, Hongseok Namkoong, Steve Yadlowsky
Prediction models can perform poorly when deployed to target distributions different from the training distribution. To understand these operational failure modes, we develop a met…
Minimax Optimal Estimation of Stability Under Distribution Shift
Hongseok Namkoong, Yuanzhe Ma, Peter W. Glynn
The performance of decision policies and prediction models often deteriorates when applied to environments different from the ones seen during training. To ensure reliable operatio…
Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time
Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre +8
The conventional recipe for maximizing model accuracy is to (1) train multiple models with various hyperparameters and (2) pick the individual model which performs best on a held-o…