← Building Glasstrace

Post-Hoc Analysis Under a Frozen Protocol: Mining Study E

Building Glasstrace, Part 10 · August 27, 2026

Study E (Part 9) left behind the artifacts of 256 accepted trials: transcripts, Model Context Protocol (MCP) call ledgers, token accounting, oracle payloads. I wanted to mine them, of course. Mechanism stories, cost stories, and funnel stories live in a corpus like that. But post-hoc mining is where research programs most often deceive themselves, because with enough tests against enough slices, something will always reach significance. So the mining was governed by a protocol as strict as the study.

The analysis plan came first

Before any outcome-adjacent field was read, the protocol froze exactly 14 tests — 4 package contrasts by oracle angle, 2 mechanism associations, 8 resource comparisons — under a single Holm correction across the family, with the caveats attached at the start: the exercise does not retroactively preregister a result; the correction controls only these 14 tests, not the preceding weeks of analytic exploration; and the freeze is attested by process within the workspace rather than by an external timestamp. The exploratory label holds, however attractive the numbers.

Five of the 14 tests survived Holm. The review then forced a further sentence into the report: those five significant rows represent fewer than five independent empirical patterns, because one of them is the registered Claude repair result restated and the monetary findings are structurally tied to the token findings.

What survived

The funnel survived everything. Instrument readiness 64 of 64 in both runtimes; at least one tool call in 64 of 64 (Claude) and 63 of 64 (Codex); decisive-route selection at 87.5 and 84.4 percent; verified repair at 39.1 and 46.9 percent. Control-arm leakage was 0 of 128, and the single non-engaging treatment trial is characterized down to its transcript — initialization, a tool listing, then silence: observed non-engagement with intact transport.

The cost inversion survived, with its qualifiers. Booked lifecycle cost per verified repair: for Claude, $8.97 with the package versus $40.74 under control, a 78 percent reduction; for Codex, $4.67 versus $8.23, a 43 percent reduction (Codex dollars are a flat-rate ledger proxy, reclassified below). Per-trial resource consumption, however, rose significantly: roughly $0.81 in additional spend and 1.06 million additional context tokens per Claude paired repetition (Holm p = 7.8 × 10⁻⁹), and roughly $0.36 and 0.64 million tokens for Codex. The package trades volume for hit rate. The per-repair ratios are descriptive portfolio accounting — no hypothesis test attaches to them, the Claude control denominator is 3 repairs, and they are not a customer return-on-investment figure.

What was eliminated

The mechanism narrative did not survive. The intuitive causal chain — decisive route retrieved, therefore repair follows — failed the Holm correction in both runtimes (within-arm associations, p = 0.165 and 1.0). The report’s required phrasing: the data do not show that route selection mediated the package effect. I find the mechanism story personally persuasive. The artifacts do not establish it, so the published materials do not assert it.

The server-improvement narrative did not survive either. The study was occasioned by a new production deployment, and the era-interaction analysis returned “not established” in both runtimes. We upgraded the server and cannot claim the upgrade accomplished anything. That sentence has to be publishable, or nothing else in the corpus is believable.

The Codex dollar figures were reclassified twice. The economic red-team review established that Codex reports token usage but no dollar cost; the harness prices it with a frozen flat rate ($5.00 per million uncached input tokens, $0.50 per million cache reads, $30.00 per million output tokens) applied to a single cumulative usage row, which demonstrably cannot reconstruct real billing. One hundred eleven of the 128 aggregates exceed the model’s single-request context window, cache writes and 119 web-search events are unpriced, and the official list rates differed on the audit date. I then corrected the record a second time at the program level: the true account totals for August 1 through 25, $266.95 for Codex and $1,241.08 for Claude, are conservative calibration only, never to be divided into per-trial prices. Every Codex dollar figure in the program’s materials is now labeled a flat-rate planning proxy rather than actual billing, and a standing harness item exists to repair the proxy before it is ever again described as measured spend.

Adversarial review as a product capability

The mining report went through independent adversarial review to convergence — independent in the sense defined in Part 7: separate implementations against hash-bound inputs, not disinterested third parties. Population and source integrity were replayed: 256 of 256 identities, 2,688 of 2,688 files, 128 of 128 ledgers regenerated with 587 of 587 rows matched. All 14 p-values and every interval were reproduced by an implementation written against an independent stack — CPython 3.14.7 and SciPy 1.18.0, including a 300,000-resample bootstrap — with a maximum absolute p-value disagreement of 4.5 × 10⁻¹⁴. Seven initial reporting defects (among them an unsupported “keepalive” label, dependent findings presented as independent, and a missing cost-denominator sensitivity) were corrected and re-reviewed. Final convergence returned zero surviving findings of any serious priority, one documentation-grade note, and a 23-file hash-bound evidence bundle. Publication required a separate, explicit authorization the following day.

The expense was moderate. For a one-person company, the return is that the claims no longer depend on trusting the person. The protocol, the registered analysis, and the evidence bundle are hash-bound, and their digests are published with this series; a reader who later receives the artifacts can re-derive the account and confirm that nothing changed in the interim. For a company whose founding thesis is that agents should act on evidence rather than supposition, the validation program is that thesis applied to the company itself.

The account, and the ceiling above it

Read across the series, the statistical account assembles in layers, none of them free. An installation audit made “zero-configuration” a measured 83 percent (Part 2). The live capture gate of Part 3 ended synthetic evidence, and Part 4 pinned an instrument race to 10 milliseconds. A preregistered study produced speed claims and later retired them (Parts 5 and 9). A scarcity finding narrowed the scope (Part 6), the study-execution system made the work inexpensive and auditable (Part 7), and a guidance question stays open deliberately (Part 8). And a 256-trial result — 34.4 and 26.6 percentage points, an odds ratio of 5.47, p = 4.5 × 10⁻⁸ — survived its own red team (Part 9 and this article).

Even so, the program’s current claim ceiling, self-imposed in the August 25 roadmap, authorizes no latency, time-to-debug, broad-reliability, general-efficacy, production, or marketing claims. Externally distributed wording ships only under release records binding exact text to evidence hashes, and broader claims wait for fresh, prospectively admitted held-out incidents that do not yet exist. From the outside, holding a result at p = 4.5 × 10⁻⁸ under a claim embargo may look like excessive caution. From the inside it is the position the company rests on: in a market saturated with unfalsifiable numbers, the scarce asset is a claim that survives hostile scrutiny, and the validation system exists to produce claims of that kind.

Supporting data and commit references

Evidence availability: references to glasstrace-sdk are public — the repository, commits, and files are linked in the series evidence manifest. All other repositories cited above are private; artifacts may be shared as needed, and the manifest publishes each cited artifact’s SHA-256 digest so that any shared copy can be verified against the version these articles describe.