Hypothesis & findings

What we set out to measure

The hypothesis was never that an agent can patch CVEs. Frontier models already can, often. The claim here is narrower and harder to fake. Everything below is measured against it, the methodology that produced each number, then the numbers, and what they don't cover.

A model's patch is worth a maintainer's attention when it arrives with proof the maintainer can check without trusting the model.

Methodology

How these numbers were produced

Three separate measurements sit behind the results on this page, each checked against a different kind of ground truth.

Blind authoring, graded against the upstream fix

For real nixpkgs security backports where the upstream fix is known, a frontier model sees only the vulnerability description and the source before the fix, never the upstream patch itself. Its own fix is then graded against what upstream actually shipped. 43 backports were eligible; 41 could be screened this way. Grading was done by an LLM judge checking semantic equivalence, not a build that runs the fix from failing to passing.

Build verification, the signal cert-green describes

Every candidate that reaches the Builder is compiled twice in a hermetic Nix sandbox at a pinned nixpkgs revision, once without the fix and once with it. Where a reproducer fixture exists, its signal is checked against both builds: the unpatched one has to fail the way the advisory describes, the patched one has to survive. Where no reproducer exists, the bundle is reviewer-attested instead, approved on review alone with no fixture built against it.

Discovery telemetry, a single dated snapshot

The discovery funnel below comes from the pipeline's own hourly telemetry, read on one date rather than tracked live or deduplicated across its full history.

Results

What the signed bundles show

Against the upstream fix

Of the 41 screened backports:

Working blind, the model matched the upstream fix, or wrote an equivalent one, on roughly two of every five real backports. These fixes come from the vulnerability itself; the model was never shown the patch it was later graded against, so this is more than a version-bump bot. The other number matters as much. A model that is wrong 34 percent of the time cannot run unsupervised, and that is why the rest of the pipeline exists. A Patcher this capable and this unreliable needs a separate Reviewer pass, a build that runs and a signed gate before anything ships.

The model was opus-4-8, the successor to the opus-4-7 pinned elsewhere in the project, because the API no longer served the pinned model at run time. Measured 4 September 2026. Methodology and per-case results are in data/reports/authoring-accuracy-vs-human-2026-09-04.json.

What a passing build certifies

This page calls a bundle cert-green when its Builder record shows the patched package actually built, in CI, from a run anyone can open. As of 8 October 2026, 86 real bundles (excluding synthetic test fixtures) met that bar: 23 routed to auto-merge and 63 to human review. Cert-green means the patch applied cleanly to the pinned nixpkgs revision and the package built; that much holds for all 86. For the bundles that also carry a reproducer, cert-green additionally means the build behaves the way the advisory predicts. For the reviewer-attested rest, cert-green says only that the patch applies and the package builds. Cert-green never says whether the patch takes the same form as the upstream fix, which is what the 39% figure above measures.

Deciding not to act

By a 21 July 2026 snapshot of its own telemetry, the discovery pipeline, which watches CVE feeds broadly and only afterward narrows to what nixpkgs actually packages, had logged roughly 19,455 CVE sightings across 628 hourly runs, a running count rather than a deduplicated total. 11,801 of those had been fully processed to a verdict at that point. Of the processed set: about 9,300 affect software nixpkgs does not package, so there is nothing to fix. Roughly 1,900 had no upstream fix when they were checked. 251 were eligible but needed a human call. 233 were eligible for the agent to start on. The ratio matters more than the 233. Most of what this system produces is a decision not to act, and the evidence behind a correct refusal costs real money. The cost page has the figures.

No frontier-parity claim

Fine-tuned open-weight models were tried as a cheaper Reviewer and Patcher. They come nowhere close to the frontier models in those roles, and the project published that result as it stood. Retired, in the open covers the best fine-tuned result in detail, including why its strong early score did not hold up on a larger sample. That work improved the project's own measurement, not its standing against frontier models. A fine-tuned 14B authoring model never produces a byte-exact match to the held-out fix in a single try. What works instead is sampling many candidate patches from that same model and keeping the first one a structural and behavioral verifier accepts: across 16 samples, that lifts correct-addition recall from zero on a single try to roughly 0.36.

Limitations

What these numbers don't cover

• The 39% figure describes 43 curated, workable authoring tasks, not a random sample of nixpkgs backports; a random sample would likely score lower. Grading was an LLM judge checking semantic equivalence, not a build; a version gated on a build running the fix from failing to passing would probably move the number, in either direction.
• Cert-green is two different claims sharing one label. For bundles with a reproducer it means the build behaves the way the advisory predicts; for reviewer-attested bundles it means only that the patch applies and the package builds. The 86 figure above counts both without distinguishing which.
• The discovery-funnel breakdown is a single dated snapshot, not a live total, and 19,455 is a running count across hourly runs rather than a deduplicated figure.
• Patcher and Reviewer both currently run on Anthropic models. The architecture page covers why a separate Reviewer pass still has value even on the same vendor. The contract calls for a stronger, cross-vendor version of that independence; it hasn't been run yet.
• Fine-tuned open-weight models were tried as a cheaper substitute for the Patcher and Reviewer roles. The No frontier-parity claim section above covers what that measured.