← Building Glasstrace

Preregistration as Policy: The Eighty-Pair Study

Building Glasstrace, Part 5 · August 27, 2026

Artificial-intelligence developer tooling is marketed in a fog of unfalsifiable claims: “ten times faster,” measured on nothing, attributed to no one. An early decision in this program was that Glasstrace’s marketing would be generated by a different mechanism — a preregistered study: the analysis plan frozen before results existed, verdicts produced by procedure, and unflattering cells published next to the flattering ones. In July that mechanism produced its first complete output.

The design

The study paired trials on 2 admitted runtime-required bugs, MFG-RLY-009 and RPORT-004-obf, both of which passed the live gate after the events of Part 3, across 2 agent runtimes (Claude and Codex) and 2 prompt profiles, with 10 pairs per cell: 80 pairs, 160 trials. Each pair ran the same bug with and without Glasstrace’s Model Context Protocol (MCP) evidence, and the preregistration fixed the outcome definitions, the statistical tests, and the verdict rules in advance. Official verdicts were generated on July 7 at 09:59 UTC: a lift verdict on fix rate and a speed-lift verdict on time.

Preregistration is often dismissed as ceremony. Its function here was to take the most dangerous participant in any analysis — the founder who needs a particular answer — out of the room where the result is computed.

The findings

Speed, on Claude, was the consistent result. Across the 40 Claude pairs, mean agent duration fell from 585.4 seconds under control conditions to 325.8 seconds with Glasstrace, a 44.3 percent mean reduction; the median fell 46.8 percent, from 542.5 to 288.7 seconds. Cell-level means ranged from 23.9 to 53.6 percent improvement. Every Claude cell was positive.

Reliability was real but fragile. Pooled fix rate: 74 of 80 under control, 79 of 80 with Glasstrace, a risk difference of +6.3 percentage points with a bootstrap 95 percent confidence interval of 1.3 to 11.3. The primary paired test told a humbler story: exact McNemar p = 0.125. The addendum further records that the interval is sensitive to bootstrap method (a naive resampling yields an interval touching zero) and that the discordant pairs concentrate in timeout-rescue behavior on one bug. The public framing ever since has been: statistically fragile; strengthens only as the bug roster expands. The report is equally explicit about where the lift lives: the pooled gain is driven by the Claude and MFG-RLY-009 cell, and on the other bug the treatment arm went 39 of 40 against a perfect 40-of-40 control.

Codex inverted. It solved these bugs at 100 percent in both arms, a ceiling effect, and consuming evidence it did not need cost it time: minus 52.8 seconds on time to final, with a 95 percent confidence interval of minus 98.1 to minus 10.8. The 95 percent interval excludes zero (negative meaning the Glasstrace arm finished later), and all 4 Codex speed cells ran negative, from minus 11.1 to minus 30.5 percent. The mechanistic reading is evidence-consumption overhead for a solver that did not require the evidence on those bugs. This inversion appears in the same documents as the Claude improvement, because a preregistration honored only in its flattering cells would be worthless.

Retrieval was the part that held everywhere. The expected trace was found 10 of 10 times in 7 of 8 cells; precision at rank 1 was 100 percent in 6 of 8 cells, with the weakest cell at 60 percent at rank 1 and 100 percent within the top 3. The evidence pipeline — capture, storage, agent-reachable retrieval — held under study conditions, and every later study leaned on that fact.

Two rules of wording

Two conventions from this study became standing policy.

Mean first, median attached. The headline is the mean reduction (44 percent), with the median (47 percent) alongside. Choosing whichever figure is larger for a given audience is the beginning of the end; a frozen convention prevents it.

“Independently re-analyzed,” never “replicated.” An independent implementation, seeded, with 10,000 bootstrap iterations, reproduced the pooled numbers: a risk difference of 6.2 percentage points with interval 1.2 to 11.2, against the report’s 6.3 with interval 1.3 to 11.3, verdicts agreeing on every slice. That is a re-analysis — the same 80 pairs, different code. “Replicated” would imply new trials and would be false. A reader who catches an inflation of “re-analyzed” into “replicated” is entitled to disbelieve everything else, so the distinction is enforced in every document.

The concessions

Rigor narrowed the scope. Eighty pairs across two bugs in one codebase — both bugs reside in Rallly, and both defects were seeded or reconstructed for study purposes rather than encountered organically — is depth without breadth. The study’s conclusions apply to these bugs under these conditions, and no further. Part 6 takes up the roster-scarcity problem underneath.

The verdict machinery contains judgment at its joints. The lift verdict rests on the bootstrap confidence interval rather than the McNemar test. That choice was preregistered and is defensible, and the addendum’s fragility notes exist because reasonable statisticians could weight the components differently.

The speed claim aged poorly, and the system caught it. Seven weeks later, a far larger study on a harder bug (Part 9) found no duration benefit at all, and current policy has retired time-to-fix percentages from forward-looking claims entirely. I regard that as the preregistration mechanism working as designed. The 44 percent figure remains true, scoped, and historical, but it is no longer the story.

A preregistered benchmark also turned out to be content few competitors are positioned to produce: anyone can assert a speed improvement, but the preregistration, the per-trial artifacts, the inversions, and the fragility notes can be published in full, and they cost little, because the validation system generates all of them as a byproduct.

Supporting data and commit references

Evidence availability: references to glasstrace-sdk are public — the repository, commits, and files are linked in the series evidence manifest. All other repositories cited above are private; artifacts may be shared as needed, and the manifest publishes each cited artifact’s SHA-256 digest so that any shared copy can be verified against the version these articles describe.