44 citations · 104 across the 47 of their papers we have counts for
49 papers
Synthetic Hospital: An Open, Verifiable, Physician-Validated Longitudinal EHR Benchmark
Christine Park, Valerie Chen, Tim Dettmers
Frontier language models are rarely used in clinical workflows because the realistic, longitudinal benchmarks needed to develop them are scarce. Real electronic health record (EHR)…
VibeJam: A User Study Platform for Web Development with Agents
Nishant Balepur, Connor Baumler, Valerie Chen +3
Programming with AI is increasingly agentic, users prompt LLMs to directly edit their code and review the changes, with adoption growing especially for web development tasks. Despi…
(Im)Paired Programming: Coding Agents Improve Productivity but Harm Understanding
Nishant Balepur, Connor Baumler, Valerie Chen +3
Coding agents (e.g., Cursor) improve developer productivity by optimizing task completion, but shifting users from writing code to prompting and reviewing may harm their understand…
Learning from 53.6K Real-World Developer Edits of AI-Generated Code
Jenny T. Liang, Mihika Bairathi, Wayne Chi +3
Imperfections in AI-generated code require that software developers modify the generated code manually, or by re-prompting an AI programming assistant. Manual code edits provide mo…
Align AI to Dynamic Human-AI Workflows
Valerie Chen, Cleotilde Gonzalez, Anita Williams Woolley +4
Current alignment approaches typically focus on emulating human behavior using static representations of human preferences, failing to capture the dynamic, context-dependent nature…
Humans are Missing from AI Coding Agent Research
Zora Z. Wang, John Yang, Kilian Lieret +10
Recent progress in AI coding agent research has led to rapid improvements in agents' ability to autonomously perform complex software engineering tasks, from editing large codebase…
Is Agent Code Less Maintainable Than Human Code?
Shaswat Patel, Betty Li Hou, Arun Purohit +4
Maintainability is a core dimension of software engineering, shaping how code is written, reviewed, and developed over time. While coding agents have demonstrated strong performanc…
ReproRepo: Scaling Reproducibility Audits with GitHub Repository Issues
Shanda Li, Qiuhong Anna Wei, Jingwu Tang +5
Reproducing research results from papers and released code is central to scientific progress. Existing works have introduced benchmarks to evaluate whether LLM agents can assist wi…
Human agency in initial human-AI proof formalization workflows
Katherine M. Collins, Simon Frieder, Jonas Bayer +14
For centuries, human mathematicians have written proofs to substantiate their mathematical arguments; yet, the ability to automatically verify the validity of proofs has long been…
RECAP: An End-to-End Platform for Capturing, Replaying, and Analyzing AI-Assisted Programming Interactions
Keyu He, Qianou Ma, Valerie Chen +2
Understanding how developers interact with AI coding assistants requires more than chat logs or git histories in isolation; it requires reconstructing the full context: which promp…
How Well Does Agent Development Reflect Real-World Work?
Zora Zhiruo Wang, Sanidhya Vijayvargiya, Aspen Chen +7
AI agents are increasingly developed and evaluated on benchmarks relevant to human work, yet it remains unclear how representative these benchmarking efforts are of the labor marke…
A Rubric-Supervised Critic from Sparse Real-World Outcomes
Xingyao Wang, Valerie Chen, Heng Ji +1
Academic benchmarks for coding agents tend to reward autonomous task completion, measured by verifiable rewards such as unit-test success. In contrast, real-world coding agents ope…
Comparing Developer and LLM Biases in Code Evaluation
Aditya Mittal, Ryan Shar, Zichu Wu +6
As LLMs are increasingly used as judges in code applications, they should be evaluated in realistic interactive settings that capture partial context and ambiguous intent. We prese…
RCTs for Frontier AI Governance: Methodological Challenges and Solutions for Human Uplift Studies
Patricia Paskov, Kevin Wei, Shen Zhou Hong +7
Human uplift studies, or studies that measure the effects of AI access on human performance via randomized controlled trials (RCT) or similar methodologies, increasingly inform fro…
Beyond the Commit: Developer Perspectives on Productivity with AI Coding Assistants
Valerie Chen, Jasmyn He, Behnjamin Williams +2
Measuring developer productivity is a topic that has attracted attention from both academic research and industrial practice. In the age of AI coding assistants, it has become even…
GameDevBench: Evaluating Agentic Capabilities Through Game Development
Wayne Chi, Yixiong Fang, Arnav Yayavaram +8
Despite rapid progress on coding agents, progress on their multimodal counterparts has lagged behind. A key challenge is the scarcity of evaluation testbeds that combine the comple…
Developer Interaction Patterns with Proactive AI: A Five-Day Field Study
Nadine Kuo, Agnia Sergeyuk, Valerie Chen +1
Current in-IDE AI coding tools typically rely on time-consuming manual prompting and context management, whereas proactive alternatives that anticipate developer needs without expl…
SWE-Tester: Training Open-Source LLMs for Issue Reproduction in Real-World Repositories
Aditya Bharat Soni, Rajat Ghosh, Vaishnavi Bhargava +2
Software testing is crucial for ensuring the correctness and reliability of software systems. Automated generation of issue reproduction tests from natural language issue descripti…
EDIT-Bench: Evaluating LLM Abilities to Perform Real-World Instructed Code Edits
Wayne Chi, Valerie Chen, Ryan Shar +8
Instructed code editing, where LLMs directly modify a developer's existing code based on a user instruction, is becoming a widely used interaction mode in AI coding assistants. How…
The OpenHands Software Agent SDK: A Composable and Extensible Foundation for Production Agents
Xingyao Wang, Simon Rosenberg, Juan Michelini +8
Agents are now used widely in the process of software development, but building production-ready software engineering agents is a complex task. Deploying software agents effectivel…
Completion Collaboration: Scaling Collaborative Effort with Agents
Shannon Zejiang Shen, Valerie Chen, Ken Gu +11
Current evaluations of agents remain centered around one-shot task completion, failing to account for the inherently iterative and collaborative nature of many real-world problems,…
TOM-SWE: User Mental Modeling For Software Engineering Agents
Xuhui Zhou, Valerie Chen, Zora Zhiruo Wang +3
Recent advances in coding agents have made them capable of planning, editing, running, and testing complex code bases. Despite their growing ability in coding tasks, these systems…
How can we assess human-agent interactions? Case studies in software agent design
Valerie Chen, Rohit Malhotra, Xingyao Wang +7
While benchmarks measure the accuracy of LLM-powered agents, they mostly assume full automation, failing to represent the collaborative nature of real-world use cases. In this pape…
Why Do Decision Makers (Not) Use AI? A Cross-Domain Analysis of Factors Impacting AI Adoption
Rebecca Yu, Valerie Chen, Ameet Talwalkar +1
Growing excitement around deploying AI across various domains calls for a careful assessment of how human decision-makers interact with AI-powered systems. In particular, it is ess…