5.1k citations · 9.6k across the 102 of their papers we have counts for
15 papers · 1 filter
EDIT-Bench: Evaluating LLM Abilities to Perform Real-World Instructed Code Edits
Wayne Chi, Valerie Chen, Ryan Shar +8
Instructed code editing, where LLMs directly modify a developer's existing code based on a user instruction, is becoming a widely used interaction mode in AI coding assistants. How…
Completion Collaboration: Scaling Collaborative Effort with Agents
Shannon Zejiang Shen, Valerie Chen, Ken Gu +11
Current evaluations of agents remain centered around one-shot task completion, failing to account for the inherently iterative and collaborative nature of many real-world problems,…
RECODE: Reasoning Through Code Generation for Visual Question Answering
Junhong Shen, Mu Cai, Bo Hu +4
Multimodal Large Language Models (MLLMs) struggle with precise reasoning for structured visuals like charts and diagrams, as pixel-based perception lacks a mechanism for verificati…
How can we assess human-agent interactions? Case studies in software agent design
Valerie Chen, Rohit Malhotra, Xingyao Wang +7
While benchmarks measure the accuracy of LLM-powered agents, they mostly assume full automation, failing to represent the collaborative nature of real-world use cases. In this pape…
Why Do Decision Makers (Not) Use AI? A Cross-Domain Analysis of Factors Impacting AI Adoption
Rebecca Yu, Valerie Chen, Ameet Talwalkar +1
Growing excitement around deploying AI across various domains calls for a careful assessment of how human decision-makers interact with AI-powered systems. In particular, it is ess…
Code with Me or for Me? How Increasing AI Automation Transforms Developer Workflows
Valerie Chen, Ameet Talwalkar, Robert Brennan +1
Developers now have access to a growing array of increasingly autonomous AI tools for software development. While many studies examine copilots that provide chat assistance or code…
Sample Complexity and Representation Ability of Test-time Scaling Paradigms
Baihe Huang, Shanda Li, Tianhao Wu +5
Test-time scaling paradigms have significantly advanced the capabilities of large language models (LLMs) on complex tasks. Despite their empirical success, theoretical understandin…
Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interaction
Junhong Shen, Hao Bai, Lunjun Zhang +8
The current paradigm of test-time scaling relies on generating long reasoning traces ("thinking" more) before producing a response. In agent problems that require interaction, this…
CoMind: Towards Community-Driven Agents for Machine Learning Engineering
Sijie Li, Weiwei Sun, Shanda Li +2
Large language model (LLM) agents show promise in automating machine learning (ML) engineering. However, existing agents typically operate in isolation on a given research problem,…
FrontierCO: Real-World and Large-Scale Evaluation of Machine Learning Solvers for Combinatorial Optimization
Shengyu Feng, Weiwei Sun, Shanda Li +2
Machine learning (ML) has shown promise for tackling combinatorial optimization (CO), but much of the reported progress relies on small-scale, synthetic benchmarks that fail to cap…
This Time is Different: An Observability Perspective on Time Series Foundation Models
Ben Cohen, Emaad Khwaja, Youssef Doubli +16
We introduce Toto, a time series forecasting foundation model with 151 million parameters. Toto uses a modern decoder-only architecture coupled with architectural innovations desig…
CodePDE: An Inference Framework for LLM-driven PDE Solver Generation
Shanda Li, Tanya Marwah, Junhong Shen +4
Partial differential equations (PDEs) are fundamental to modeling physical systems, yet solving them remains a complex challenge. Traditional numerical solvers rely on expert knowl…
CodingGenie: A Proactive LLM-Powered Programming Assistant
Sebastian Zhao, Alan Zhu, Hussein Mozannar +3
While developers increasingly adopt tools powered by large language models (LLMs) in day-to-day workflows, these tools still require explicit user invocation. To seamlessly integra…
When Benchmarks Talk: Re-Evaluating Code LLMs with Interactive Feedback
Jane Pan, Ryan Shar, Jacob Pfau +3
Programming is a fundamentally interactive process, yet coding assistants are often evaluated using static benchmarks that fail to measure how well models collaborate with users. W…
Copilot Arena: A Platform for Code LLM Evaluation in the Wild
Wayne Chi, Valerie Chen, Anastasios Nikolas Angelopoulos +7
Evaluating in-the-wild coding capabilities of large language models (LLMs) is a challenging endeavor with no clear solution. We introduce Copilot Arena, a platform to collect user…