46 citations · 104 across the 35 of their papers we have counts for
8 papers · 1 filter
Between Spoilers and Guesswork: Avoiding Over-Specification in Coding Benchmarks
Yuanzhe Liu, Bastian Lipka, Avraham Shinnar +3
Benchmarks drive AI progress, but are hampered by task under-specification and over-specification. Task under-specification causes guesswork, where solutions have to make things up…
One Recipe, Many Harnesses: What Self-Evolution Encodes Across Languages and Models
Siqi Yang, Qianlan Yang, Yu-Xiong Wang +2
Self-evolving harnesses are closed-loop systems in which an agent inspects its own rollouts and edits its prompts, tools, and memory. They reliably improve coding agents in evaluat…
Online Monitoring and Corrective Steering of Programming Agents
Shuyang Liu, Saman Dehghan, Ji Young Kim +3
Fixing GitHub issues in large-scale projects is a long-horizon task, especially when a fix requires changes across multiple locations or the issue description lacks the information…
PPDL: LLM-Based Flows as Probabilistic Programs
Louis Mandel, Guillaume Baudart, Mandana Vaziri +1
Building reliable applications that leverage large language models (LLMs) remains a significant challenge. While LLMs offer impressive capabilities across diverse tasks, their outp…
EvoOtter: Evolutionary Reproduction Test Generator
Toufique Ahmed, Jatin Ganhotra, Avraham Shinnar +1
Before fixing an issue, it is useful to first reproduce it by generating a bug reproduction test (BRT). However, generating a BRT is itself a challenging task, because issue descri…
Reproduction Test Generation for Java SWE Issues
Toufique Ahmed, Jatin Ganhotra, Avraham Shinnar +1
Given an issue on a software repository, a reproduction test confirms its presence in the code before it gets fixed and its absence after. Reproduction tests provide crucial execut…
From Plan to Action: How Well Do Agents Follow the Plan?
Shuyang Liu, Saman Dehghan, Jatin Ganhotra +2
Agents are commonly instructed to follow a task-specific plan for guidance. However, it is unknown to what extent agents actually follow instructed plans. Without such an analysis,…
Resolving Java Code Repository Issues with iSWE Agent
Jatin Ganhotra, Sami Serhan, Antonio Abu Nassar +3
Resolving issues on code repositories is an important part of software engineering. Various recent systems automatically resolve issues using large language models and agents, ofte…