5.1k citations · 9.6k across the 102 of their papers we have counts for
106 papers
Interaction-as-a-Judge
Wayne Chi, I-Hung Hsu, Li Chen +6
Coding agents are tasked with producing sophisticated interactive applications such as websites, desktop applications, and games. While humans assess these applications interactive…
Learning from 53.6K Real-World Developer Edits of AI-Generated Code
Jenny T. Liang, Mihika Bairathi, Wayne Chi +3
Imperfections in AI-generated code require that software developers modify the generated code manually, or by re-prompting an AI programming assistant. Manual code edits provide mo…
Toto 2.0: Time Series Forecasting Enters the Scaling Era
Emaad Khwaja, Chris Lettieri, Gerald Woo +10
We show that time series foundation models scale: a single training recipe produces reliable forecast-quality improvements from 4M to 2.5B parameters. We release Toto 2.0, a family…
ReproRepo: Scaling Reproducibility Audits with GitHub Repository Issues
Shanda Li, Qiuhong Anna Wei, Jingwu Tang +5
Reproducing research results from papers and released code is central to scientific progress. Existing works have introduced benchmarks to evaluate whether LLM agents can assist wi…
ARFBench: Benchmarking Time Series Question Answering Ability for Software Incident Response
Stephan Xie, Ben Cohen, Mononito Goswami +6
Time series question-answering (TSQA), in which we ask natural language questions to infer and reason about properties of time series, is a promising yet underexplored capability o…
Spend Less, Fit Better: Budget-Efficient Scaling Law Fitting via Active Experiment Selection
Sijie Li, Shanda Li, Haowei Lin +3
Scaling laws are used to plan multi-million-dollar training runs, but fitting those laws can itself cost millions. In modern large-scale workflows, assembling a sufficiently inform…
Learn Hard Problems During RL with Reference Guided Fine-tuning
Yangzhen Wu, Shanda Li, Zixin Wen +5
Reinforcement learning (RL) for mathematical reasoning can suffer from reward sparsity: for challenging problems, LLM fails to sample any correct trajectories, preventing RL from r…
Comparing Developer and LLM Biases in Code Evaluation
Aditya Mittal, Ryan Shar, Zichu Wu +6
As LLMs are increasingly used as judges in code applications, they should be evaluated in realistic interactive settings that capture partial context and ambiguous intent. We prese…
Beyond the Commit: Developer Perspectives on Productivity with AI Coding Assistants
Valerie Chen, Jasmyn He, Behnjamin Williams +2
Measuring developer productivity is a topic that has attracted attention from both academic research and industrial practice. In the age of AI coding assistants, it has become even…
GameDevBench: Evaluating Agentic Capabilities Through Game Development
Wayne Chi, Yixiong Fang, Arnav Yayavaram +8
Despite rapid progress on coding agents, progress on their multimodal counterparts has lagged behind. A key challenge is the scarcity of evaluation testbeds that combine the comple…
Pre-Generating Multi-Difficulty PDE Data for Few-Shot Neural PDE Solvers
Naman Choudhary, Vedant Singh, Ameet Talwalkar +3
A key aspect of learned partial differential equation (PDE) solvers is that the main cost often comes from generating training data with classical solvers rather than learning the…
EDIT-Bench: Evaluating LLM Abilities to Perform Real-World Instructed Code Edits
Wayne Chi, Valerie Chen, Ryan Shar +8
Instructed code editing, where LLMs directly modify a developer's existing code based on a user instruction, is becoming a widely used interaction mode in AI coding assistants. How…
Completion Collaboration: Scaling Collaborative Effort with Agents
Shannon Zejiang Shen, Valerie Chen, Ken Gu +11
Current evaluations of agents remain centered around one-shot task completion, failing to account for the inherently iterative and collaborative nature of many real-world problems,…
RECODE: Reasoning Through Code Generation for Visual Question Answering
Junhong Shen, Mu Cai, Bo Hu +4
Multimodal Large Language Models (MLLMs) struggle with precise reasoning for structured visuals like charts and diagrams, as pixel-based perception lacks a mechanism for verificati…
How can we assess human-agent interactions? Case studies in software agent design
Valerie Chen, Rohit Malhotra, Xingyao Wang +7
While benchmarks measure the accuracy of LLM-powered agents, they mostly assume full automation, failing to represent the collaborative nature of real-world use cases. In this pape…
Why Do Decision Makers (Not) Use AI? A Cross-Domain Analysis of Factors Impacting AI Adoption
Rebecca Yu, Valerie Chen, Ameet Talwalkar +1
Growing excitement around deploying AI across various domains calls for a careful assessment of how human decision-makers interact with AI-powered systems. In particular, it is ess…
Code with Me or for Me? How Increasing AI Automation Transforms Developer Workflows
Valerie Chen, Ameet Talwalkar, Robert Brennan +1
Developers now have access to a growing array of increasingly autonomous AI tools for software development. While many studies examine copilots that provide chat assistance or code…
Sample Complexity and Representation Ability of Test-time Scaling Paradigms
Baihe Huang, Shanda Li, Tianhao Wu +5
Test-time scaling paradigms have significantly advanced the capabilities of large language models (LLMs) on complex tasks. Despite their empirical success, theoretical understandin…
Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interaction
Junhong Shen, Hao Bai, Lunjun Zhang +8
The current paradigm of test-time scaling relies on generating long reasoning traces ("thinking" more) before producing a response. In agent problems that require interaction, this…
CoMind: Towards Community-Driven Agents for Machine Learning Engineering
Sijie Li, Weiwei Sun, Shanda Li +2
Large language model (LLM) agents show promise in automating machine learning (ML) engineering. However, existing agents typically operate in isolation on a given research problem,…
FrontierCO: Real-World and Large-Scale Evaluation of Machine Learning Solvers for Combinatorial Optimization
Shengyu Feng, Weiwei Sun, Shanda Li +2
Machine learning (ML) has shown promise for tackling combinatorial optimization (CO), but much of the reported progress relies on small-scale, synthetic benchmarks that fail to cap…
This Time is Different: An Observability Perspective on Time Series Foundation Models
Ben Cohen, Emaad Khwaja, Youssef Doubli +16
We introduce Toto, a time series forecasting foundation model with 151 million parameters. Toto uses a modern decoder-only architecture coupled with architectural innovations desig…
CodePDE: An Inference Framework for LLM-driven PDE Solver Generation
Shanda Li, Tanya Marwah, Junhong Shen +4
Partial differential equations (PDEs) are fundamental to modeling physical systems, yet solving them remains a complex challenge. Traditional numerical solvers rely on expert knowl…
CodingGenie: A Proactive LLM-Powered Programming Assistant
Sebastian Zhao, Alan Zhu, Hussein Mozannar +3
While developers increasingly adopt tools powered by large language models (LLMs) in day-to-day workflows, these tools still require explicit user invocation. To seamlessly integra…