19 papers
Seq@K: Evaluating Language Models Under Sequential Feedback
Thomson Yen, Gilbert Yang, Naimeng Ye +8
Language-model agents differ fundamentally from prior intelligent systems in their ability to learn from natural language in context. This capability enables agents to iteratively…
MBABench-2.0: Evaluating AI Agents on Constructing Financial Spreadsheets Beyond Simple Verifiability
Julian Poeltl, Thomson Yen, Patrick Shea +5
Spreadsheets are the lingua franca of modern finance, driving valuation, forecast, audit, and many other core workflows of the industry. As LLM agents advance to automate knowledge…
GameBoyWorlds: A Testbed for Self-Improvement in Embodied Video Games
Dhananjay Ashok, Adam Shen, Aslan Huo Feng +10
Powered by expert guidance, agents can operate in interactive environments; however, it is unclear whether they can learn autonomously from their own experience. To evaluate such s…
Rare Event Estimation via Iterative Unalignment
Hanming Yang, Daksh Mittal, Jing Dong +1
As agents are deployed with increased autonomy, even extremely rare events along their stochastic output trajectories can occur and prove catastrophic. Safe deployment therefore do…
LatentGym: A Testbed For Cross-Task Experiential Learning With Controllable Latent Structure
Daksh Mittal, Tommaso Castellani, Thomson Yen +7
We envision continually learning agentic systems that become more useful over time: as they encounter sequences of related tasks, they should infer the hidden structure shared acro…
MBABench: Evaluating LLM Agents on End-to-End Spreadsheet Tasks in Finance
Thomson Yen, Julian Poeltl, Harshith Srinivas Gear +10
LLM agents are increasingly expected to carry out end-to-end workflows, producing complete artifacts from high-level user instructions. To meet enterprise needs, frontier AI labs h…
Benchmarking In-context Experiential Learning Through Repeated Product Recommendations
Gilbert Yang, Yaqin Chen, Thomson Yen +1
To navigate ever-shifting real-world environments, agents must grapple with incomplete knowledge and adapt their strategies through experience. However, current evaluations of LLM-…
A Sensitivity Approach to Causal Inference Under Limited Overlap
Yuanzhe Ma, Yian Huang, Hongseok Namkoong
Limited overlap between treated and control groups is a key challenge in observational analysis. Standard approaches like trimming importance weights can reduce variance but introd…
SynthTools: A Framework for Scaling Synthetic Tools for Agent Development
Tommaso Castellani, Naimeng Ye, Daksh Mittal +4
For agentic systems to use external tools to solve complex, long-horizon tasks, we need a large set of diverse and controllable tool-use environments. We introduce SynthTools, a fu…
Learning-To-Measure: In-Context Active Feature Acquisition
Yuta Kobayashi, Zilin Jing, Jiayu Yao +2
Active feature acquisition (AFA) is a sequential decision-making problem where the goal is to improve model performance for test instances by adaptively selecting which features to…
A Broader View of Thompson Sampling
Yanlin Qu, Hongseok Namkoong, Assaf Zeevi
Thompson Sampling is one of the most widely used and studied bandit algorithms, known for its simple structure, low regret performance, and solid theoretical guarantees. Yet, in st…
Data-Driven Stochastic Modeling Using Autoregressive Sequence Models: Translating Event Tables to Queueing Dynamics
Daksh Mittal, Shunri Zheng, Jing Dong +1
While queueing network models are powerful tools for analyzing service systems, they traditionally require substantial human effort and domain expertise to construct. To make this…
Adaptive Elicitation of Latent Information Using Natural Language
Jimmy Wang, Thomas Zollo, Richard Zemel +1
Eliciting information to reduce uncertainty about a latent entity is a critical task in many application domains, e.g., assessing individual student learning outcomes, diagnosing u…
Data Mixture Optimization: A Multi-fidelity Multi-scale Bayesian Framework
Thomson Yen, Andrew Wei Tung Siah, Haozhe Chen +3
Careful curation of data sources can significantly improve the performance of LLM pre-training, but predominant approaches rely heavily on intuition or costly trial-and-error, maki…
LLM Generated Persona is a Promise with a Catch
Ang Li, Haozhe Chen, Hongseok Namkoong +1
The use of large language models (LLMs) to simulate human behavior has gained significant attention, particularly through personas that approximate individual characteristics. Pers…
Architectural and Inferential Inductive Biases For Exchangeable Sequence Modeling
Daksh Mittal, Ang Li, Tzu-Ching Yen +2
Autoregressive models have emerged as a powerful framework for modeling exchangeable sequences - i.i.d. observations when conditioned on some latent factor - enabling direct modeli…
A Planning Framework for Adaptive Labeling
Daksh Mittal, Yuanzhe Ma, Shalmali Joshi +1
Ground truth labels/outcomes are critical for advancing scientific and engineering applications, e.g., evaluating the treatment effect of an intervention or performance of a predic…
Contextual Thompson Sampling via Generation of Missing Data
Kelly W. Zhang, Tiffany Tianhui Cai, Hongseok Namkoong +1
We introduce a framework for Thompson sampling (TS) contextual bandit algorithms, in which the algorithm's ability to quantify uncertainty and make decisions depends on the quality…
C-Learner: Constrained Learning for Causal Inference
Tiffany Tianhui Cai, Yuri Fonseca, Kaiwen Hou +1
Debiasing methods such as augmented inverse propensity weighting (AIPW), and targeted maximum likelihood estimation (TMLE) enjoy asymptotic properties like semiparametric efficienc…