565 citations · 588 across the 12 of their papers we have counts for
13 papers
AgentPersonaBench: Benchmarking Persona-Driven User Simulation
Jintao Huang, Yifan Wang, Hongyu Shen +43
We introduce AgentPersonaBench (APB), a benchmark evaluating whether persona conditioning faithfully steers downstream agent behavior. While language models are increasingly deploy…
Discovering High-Quality Chess Puzzles with Offline Reinforcement Learning
Allen Nie, Anirudhan Badrinath, Nicholas Tomlin +5
Learning and skill mastery require extensive and deliberate practice. In many learning settings, producing high-quality pedagogical materials can require a high level of domain exp…
Understanding the Challenges in Iterative Generative Optimization with LLMs
Allen Nie, Xavier Daull, Zhiyi Kuang +10
Generative optimization uses large language models (LLMs) to iteratively improve artifacts (such as code, workflows or prompts) using execution feedback. It is a promising approach…
Formalizing Learning from Language Feedback with Provable Guarantees
Wanqiao Xu, Allen Nie, Ruijie Zheng +3
Interactively learning from observation and language feedback is an increasingly studied area driven by the emergence of large language model (LLM) agents. Despite impressive empir…
The GPT Surprise: Offering Large Language Model Chat in a Massive Coding Class Reduced Engagement but Increased Adopters Exam Performances
Allen Nie, Yash Chandak, Miroslav Suzara +6
Large language models (LLMs) are quickly being adopted in a wide range of learning experiences, especially via ubiquitous and broadly accessible chat interfaces like ChatGPT and Co…
Rel-A.I.: An Interaction-Centered Approach To Measuring Human-LM Reliance
Kaitlyn Zhou, Jena D. Hwang, Xiang Ren +3
The ability to communicate uncertainty, risk, and limitation is crucial for the safety of large language models. However, current evaluations of these abilities rely on simple cali…
The Importance of Directional Feedback for LLM-based Optimizers
Allen Nie, Ching-An Cheng, Andrey Kolobov +1
We study the potential of using large language models (LLMs) as an interactive optimizer for solving maximization problems in a text space using natural language and numerical feed…
Trace is the Next AutoDiff: Generative Optimization with Rich Feedback, Execution Traces, and LLMs
Ching-An Cheng, Allen Nie, Adith Swaminathan
We study a class of optimization problems motivated by automating the design and update of AI systems like coding assistants, robots, and copilots. AutoDiff frameworks, like PyTorc…
OPERA: Automatic Offline Policy Evaluation with Re-weighted Aggregates of Multiple Estimators
Allen Nie, Yash Chandak, Christina J. Yuan +3
Offline policy evaluation (OPE) allows us to evaluate and estimate a new sequential decision-making policy's performance by leveraging historical interaction data collected from ot…
LLF-Bench: Benchmark for Interactive Learning from Language Feedback
Ching-An Cheng, Andrey Kolobov, Dipendra Misra +2
We introduce a new benchmark, LLF-Bench (Learning from Language Feedback Benchmark; pronounced as "elf-bench"), to evaluate the ability of AI agents to interactively learn from nat…
MoCa: Measuring Human-Language Model Alignment on Causal and Moral Judgment Tasks
Allen Nie, Yuhui Zhang, Atharva Amdekar +3
Human commonsense understanding of the physical and social world is organized around intuitive theories. These theories support making causal and moral judgments. When something ba…
Leveraging Large (Visual) Language Models for Robot 3D Scene Understanding
William Chen, Siyi Hu, Rajat Talak +1
Abstract semantic 3D scene understanding is a problem of critical importance in robotics. As robots still lack the common-sense knowledge about household objects and locations of a…
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao +448
Language models demonstrate both quantitative improvement and new qualitative capabilities with increasing scale. Despite their potentially transformative impact, these new capabil…