Software WorldSimulating GitHub with agents
We build a GitHub simulated by agents and study their collaboration under extrinsic evaluation.
Motivation: optimizing software ecosystem
The development of expert-level coding agents opens the opportunity to automatically improve the efficiency of software packages that millions people use everyday. Existing efforts tend to focus on developing and maintaining individual packages, but software is not build in a vacuum. Packages exist within a broader dependency ecosystem, in which a given package might take on others as dependencies while still others downstream might depend on itself. We believe that improving software packages require improving the broader ecosystem.
More than just the implications on software engineering, optimizing software ecosystems provide an exciting setting for studying and evaluating multi-agent coordination. Ecosystems can naturally be broken up, with each agent optimizing its own package. The social structure of an ecosystem is ripe with opportunities for agents to collaborate to exchange usage information, jointly discover goals for improvement, and independently validate each others' changes. Moreover, ecosystems enable robust evaluation. We can hold out certain downstream packages—that is, not optimize them—and use them as an extrinsic evaluation. If agents' individual and collective efforts improve the ecosystem, these held out packages, that potentially use dependencies in unique ways, should also benefit.
To study this problem, we create several software worlds of Python packages sub-sampled based on real dependency data, and we deploy agent teams focused on optimizing code efficiency. We find that agents are capable of identifying code deficiencies and come up with patches individually. Notably, depending on the base models, some agents can engage in meaningful collaboration, soliciting downstream usage patterns that lead to discovering algorithmic improvements for their code. Over long-running sessions that last from a few days to weeks, agent teams can consistently improve the code, leading to 5-10% drop-in improvements measured by held-out downstream benchmarks. Through these simulations, we begin to understand how the future of agent software engineering may look like.
Creating software worlds
Selecting Repositories in the Software World. We construct software worlds by subsampling real packages from the Python ecosystem. Our candidate catalog combines packages from prior software-engineering benchmarks (Sehgal et al., 2026; He et al., 2025; Deng et al., 2025; Ma et al., 2025; Shetty et al., 2025; Nangia et al., 2026) with popular PyPI packages, for which we collect releases, source repositories, tests, benchmarks, and dependency metadata. We organize the resulting repositories into candidate communities using their package-level dependency graph, producing a shortlist of 302 repositories. Finally, we enrich the graph with static and runtime interface-usage information and manually group packages and their dependencies into software worlds of different sizes with controlled structure, composition, and code-usage patterns.
A proxy society around requests → urllib3
Society A, aligned with the held-out consumers (world overlap 0.180): two requests-only consumers, one urllib3-only, one consumer of both — six repositories, six edges, density 0.4.
Simulating software worlds
Agent Implementation. Each repository in a software world is assigned to a persistent maintainer agent that operates asynchronously in discrete work episodes, which we call wakes. Agents receive only the high-level objective of improving software efficiency while preserving correctness. At the beginning of each wake, an agent loads its persistent memory, reviews the current state of its repository and recent collaboration activity, and independently decides what to investigate. Using Harbor (Harbor Framework Team, 2026), each agent runs in a sandboxed Docker container where it can inspect and edit code, execute tests and profilers, manage version-control state, and publish new releases.
Agent Collaboration. We host the repositories on a modified version of Forge, an open-source, GitHub-like service that provides version control, issue tracking, pull requests, and release management. Agents can inspect other repositories, open and discuss issues, propose changes, and review others' pull requests via the provided forge cli. Communication is therefore asynchronous and artifact-bound: an agent that needs something from a neighbour opens an issue on that neighbour's repository, the exchange accumulates as comments on the issue or on a pull request, and every claim carries the code, benchmark, or profile it rests on. There is no direct channel between agents; what one agent learns from another, it learns from the forge, on its next wake. Only the assigned maintainer can merge changes and publish releases for its repository. These affordances ground collaboration in concrete software artifacts; in practice, we find agents exchange usage and profiling evidence, request upstream changes, discuss proposed improvements, and independently validate releases against their own workloads.
Collaboration Examples. A few kinds of exchange recur across models. Each is shown here by one real story from the runs, one frame per event. Every run can also be watched whole, wake by wake, on the simulation page.
Two agents spent two days establishing where pluggy's time is not, and left pytest with two profilers and one real target.
About to optimize its next release, pluggy-agent asks pytest — its most important consumer on the forge — which pluggy paths show up in pytest's profiles, which hooks fire most, and what must not change, so it doesn't have to guess which paths to optimize.
“I would love to make those improvements target workloads pytest actually cares about rather than guessing hot paths.”
Analyzing agent simulations
Evaluating Agent Work with Downstream Impacts. For each software world, we curate a set of evaluation repositories with human-authored performance benchmarks that exercise one or more libraries in the world. Evaluation repositories and their benchmark tasks remain fixed and unseen to the agents in the society. Because wall-clock measurements can be noisy, we use Callgrind's instruction-read count (Ir) collected in standardized container environments as our primary performance metric. Each evaluation sweep installs the world's latest releases under the fixed evaluation repositories and reports the geometric-mean speedup of their benchmarks against the world at seed; a benchmark that breaks scores 1.0, so breakage is never rewarded.
Two central providers (requests → urllib3) and four society consumers; four frozen downstream packages — requests-cache, cachecontrol, zeep, awswrangler — are the hidden battery.
| run | harness | wakes | duration | compute | releases | cross-repo | final |
|---|---|---|---|---|---|---|---|
| Terra | codex | 600 | 9.0 h | $221 | 103 | 0 | 1.034 |
| Sonnet 5 | claude-code | 599 | 13.5 h | $510 | 81 | 4 | 1.050 |
| Opus 5 | claude-code | 385 | 6.8 d | $3,021 | 171 | 216 | 1.140 |
3 runs · adoption geomean over 17 hidden codspeed tasks (CPU instructions); red gate → 1.0. adoption geomean = exp(mean(log(seed / current))) over each counted task's CPU instruction count; a red-gated consumer scores 1.0 on all its tasks. Compute is priced at list rates. Click a row to add or remove that run from the plot; click a line to pin it.
Agent Surveys. At the end of a run, every agent fills in two questionnaires. The first is a society survey: nine questions on different aspects of the collaboration, such as ownership clarity, task discovery, and coordination, each on a five-point Likert scale, plus the bottlenecks it hit and a few written answers. The second is a peer review: the agent chooses which of the agents it worked with to review, and assesses the quality of their collaboration. The card below shows both, one arm per tab; absolute levels follow a model's temperament, so read the comparison across arms rather than any one number.
How well the society worked, rated 1–5 by every agent; one dot per arm, the mean.
Levels follow the reviewing model's temperament; read across arms, not up the scale. Where an arm was surveyed again (World 1 · Sol and Sonnet 5) the band behind its dot is the spread across rounds — differences under 0.2 are noise.
Simulation safety
Agents that edit, review, and release real software can pose serious safety risks to the public infrastructure that many rely on: OpenAI's report on the Hugging Face incident showed what autonomous agents can do once they reach shared infrastructure, and a society of maintainer agents is exactly that kind of system. In our simulation, we emphasize safety by running agents in isolated environments; and we find that the collaborative social structure can potentially mitigate malicious behavior, catch bugs, and improve code robustness collectively.
Isolated Sandboxes. As we choose key Python libraries, direct changes to dependencies have the potential to introduce security vulnerabilities or alter the behavior in ways that harm downstream dependents. We're careful to not directly contribute to this problem in the live Python package ecosystem. In our research prototype, we sandbox all of the agents' work and use a self-hosted GitHub alternative to completely separate the optimized repos from any live package distribution channels. The sandboxes do reach the public index to install dependencies; what they cannot do is publish to it.
Catching Bugs with Collaborative Agent Society. Another surprising finding is that, when you put agents in a collaborative society of agents, other contributors can help identify and report bugs introduced by the agents. In one run, for example, a urllib3 release that sped up header lookups on CPython 3.13 but slowed them on 3.10 was reported by a downstream agent within the hour, with urllib3's own benchmark as evidence, and fixed and re-released twenty-six minutes later.
References
- Deng, X., Da, J., Pan, E., He, Y. Y., Ide, C., Garg, K., Lauffer, N., Park, A., Pasari, N., Rane, C., Sampath, K., Krishnan, M., et al. (2025). SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks? arXiv:2509.16941. arxiv.org
- Harbor Framework Team (2026). Harbor: A Framework for Evaluating and Optimizing Agents and Models in Container Environments. Software, version v0.16.1. doi.org
- He, X., Liu, Q., Du, M., Yan, L., Fan, Z., Huang, Y., Zheng, Y., Yuan, Z., and Ma, Z. (2025). SWE-Perf: Can Language Models Optimize Code Performance on Real-World Repositories? arXiv:2507.12415. arxiv.org
- Ma, J. J., Hashemi, M., Yazdanbakhsh, A., Swersky, K., Press, O., Li, E., Reddi, V. J., and Ranganathan, P. (2025). SWE-fficiency: Can Language Models Optimize Real-World Repositories on Real Workloads? arXiv:2511.06090. arxiv.org
- Nangia, A., Mishra, S., Gokrani, A., and Chopra, P. (2026). ISO-Bench: Can Coding Agents Optimize Real-World Inference Workloads? arXiv:2602.19594. arxiv.org
- Sehgal, A., Hou, J., Sarkar, A., Mantripragada, I., Chaudhuri, S., Sun, J. J., and Yue, Y. (2026). FormulaCode: Evaluating Agentic Optimization on Large Codebases. arXiv:2603.16011. arxiv.org
- Shetty, M., Jain, N., Liu, J., Kethanaboyina, V., Sen, K., and Stoica, I. (2025). GSO: Challenging Software Optimization Tasks for Evaluating SWE-Agents. arXiv:2505.23671. arxiv.org