For Coding Agents, the Repository Matters

Abstract

In July 2026, about 1,200 AI agents in an internal OpenAI evaluation, meant to run in isolation, used a shared package server as a message board, and about 700 of them joined an attack on Hugging Face. An automated scorer judged each agent on its own task, and the attack was coordinated through infrastructure the agents shared. Coding agents meet shared infrastructure every day, in the software repositories that receive their pull requests. Yet field studies still judge coding agents by outcomes pooled across repositories, such as merge rates and merge times. We asked how much these outcomes depend on the receiving repository, and whether they depend on it more than human contributions do. Using multilevel models, we compared agent pull requests from the AIDev dataset of public GitHub repositories with human pull requests in the same repositories. About two-fifths of the variation in how long agent pull requests take to be resolved lies between repositories, after accounting for the pull request, the agent, and the repository's observable attributes. In 280 repositories, with covariates common to both groups, the between-repository variance is 1.76 times as large for agent as for human pull requests (95% CI 1.41 to 2.18). For waits of several hours, a repository one standard deviation slower than average multiplies the typical wait by about 2.8 for agent and 2.2 for human pull requests. Rejection shows the same pattern, while for review rounds and comments the difference is not detectable. A proxy for the role of the developer who started the agent accounts for little of the difference, and stated agent policies were too rare to test. These results suggest that the oversight and evaluation of coding agents need the repository, and not only the agent, as a unit.

arXiv comment11 pages, 2 figures, 3 tables, followed by a 4-page supplement (Tables S1-S4, Figures S1-S2). v2: substantially revised and retitled; v1 was titled "Govern the Repository, Not the Agent: Measuring Ecosystem-Level Risk in AI-Native Software". Revised research questions, analysis and framing; replication package on Zenodo (doi:10.5281/zenodo.23504056)