430 citations · 512 across the 19 of their papers we have counts for
19 papers
LearnLM: Improving Gemini for Learning
LearnLM Team, Abhinit Modi, Aditya Srikanth Veerubhotla +43
Today's generative AI systems are tuned to present information by default, rather than engage users in service of learning as a human tutor would. To address the wide range of pote…
Pretraining Data Mixtures Enable Narrow Model Selection Capabilities in Transformer Models
Steve Yadlowsky, Lyric Doshi, Nilesh Tripuraneni
Transformer models, notably large language models (LLMs), have the remarkable ability to perform in-context learning (ICL) -- to perform new tasks when prompted with unseen input-o…
Choosing a Proxy Metric from Past Experiments
Nilesh Tripuraneni, Lee Richardson, Alexander D'Amour +2
In many randomized experiments, the treatment effect of the long-term metric (i.e. the primary outcome of interest) is often difficult or infeasible to measure. Such long-term metr…
Diagnosing Model Performance Under Distribution Shift
Tiffany Tianhui Cai, Hongseok Namkoong, Steve Yadlowsky
Prediction models can perform poorly when deployed to target distributions different from the training distribution. To understand these operational failure modes, we develop a met…
Boosting the interpretability of clinical risk scores with intervention predictions
Eric Loreaux, Ke Yu, Jonas Kemp +8
Machine learning systems show significant promise for forecasting patient adverse events via risk scores. However, these risk scores implicitly encode assumptions about future inte…
Understanding the Risks and Rewards of Combining Unbiased and Possibly Biased Estimators, with Applications to Causal Inference
Michael Oberst, Alexander D'Amour, Minmin Chen +3
Several problems in statistics involve the combination of high-variance unbiased estimators with low-variance estimators that are only unbiased under strong assumptions. A notable…
Calibration Error for Heterogeneous Treatment Effects
Yizhe Xu, Steve Yadlowsky
Recently, many researchers have advanced data-driven methods for modeling heterogeneous treatment effects (HTEs). Even still, estimation of HTEs is a difficult task -- these method…
Explaining Practical Differences Between Treatment Effect Estimators with High Dimensional Asymptotics
Steve Yadlowsky
We revisit the classical causal inference problem of estimating the average treatment effect in the presence of fully observed confounding variables using two-stage semiparametric…
Evaluating Treatment Prioritization Rules via Rank-Weighted Average Treatment Effects
Steve Yadlowsky, Scott Fleming, Nigam Shah +2
There are a number of available methods for selecting whom to prioritize for treatment, including ones based on treatment effect estimation, risk scoring, and hand-crafted rules. W…
The MultiBERTs: BERT Reproductions for Robustness Analysis
Thibault Sellam, Steve Yadlowsky, Jason Wei +9
Experiments with pre-trained models such as BERT are often based on a single checkpoint. While the conclusions drawn apply to the artifact tested in the experiment (i.e., the parti…
Counterfactual Invariance to Spurious Correlations: Why and How to Pass Stress Tests
Victor Veitch, Alexander D'Amour, Steve Yadlowsky +1
Informally, a 'spurious correlation' is the dependence of a model on some aspect of the input data that an analyst thinks shouldn't matter. In machine learning, these have a know-i…
SLOE: A Faster Method for Statistical Inference in High-Dimensional Logistic Regression
Steve Yadlowsky, Taedong Yun, Cory McLean +1
Logistic regression remains one of the most widely used tools in applied statistics, machine learning and data science. However, in moderately high-dimensional problems, where the…
Deep Cox Mixtures for Survival Regression
Chirag Nagpal, Steve Yadlowsky, Negar Rostamzadeh +1
Survival analysis is a challenging variation of regression modeling because of the presence of censoring, where the outcome measurement is only partially known, due to, for example…
Underspecification Presents Challenges for Credibility in Modern Machine Learning
Alexander D'Amour, Katherine Heller, Dan Moldovan +37
ML models often exhibit unexpectedly poor behavior when they are deployed in real-world domains. We identify underspecification as a key reason for these failures. An ML pipeline i…
Using public clinical trial reports to evaluate observational study methods
Ethan Steinberg, Nikolaos Ignatiadis, Steve Yadlowsky +2
Observational studies are valuable for estimating the effects of various medical interventions, but are notoriously difficult to evaluate because the methods used in observational…
Off-policy Policy Evaluation For Sequential Decisions Under Unobserved Confounding
Hongseok Namkoong, Ramtin Keramati, Steve Yadlowsky +1
When observed decisions depend only on observed features, off-policy policy evaluation (OPE) methods for sequential decision making problems can estimate the performance of evaluat…
Estimation and Validation of Ratio-based Conditional Average Treatment Effects Using Observational Data
Steve Yadlowsky, Fabio Pellegrini, Federica Lionetto +2
While sample sizes in randomized clinical trials are large enough to estimate the average treatment effect well, they are often insufficient for estimation of treatment-covariate i…
Bounds on the conditional and average treatment effect with unobserved confounding factors
Steve Yadlowsky, Hongseok Namkoong, Sanjay Basu +2
For observational studies, we study the sensitivity of causal inference when treatment assignments may depend on unobserved confounders. We develop a loss minimization approach for…
Derivative free optimization via repeated classification
Tatsunori B. Hashimoto, Steve Yadlowsky, John C. Duchi
We develop an algorithm for minimizing a function using batched function value measurements at each of rounds by using classifiers to identify a function's sublevel set. We…