← Building Glasstrace

Synthetic Evidence in the Validation Pipeline: The June Capture-Gap Finding

Building Glasstrace, Part 3 · August 27, 2026

On June 8, 2026, I learned that the most convincing evidence in the validation corpus was not real. It was not fraudulent — it was synthetic, in a way no one had noticed. The remedy was a change in what the program accepts as proof.

Background

By June the validation system had moved past installation questions (Part 2) to a harder one: does the system capture the decisive fact? For a bug to matter to the Glasstrace thesis, the runtime evidence must contain the element an agent could not infer from source code: a side effect, a value, a linkage. A gate existed for this property, designated NR-2, and candidate bugs had been admitted to the study roster on the strength of passing it.

The value-fidelity capture work of that period passed the gate, the roster grew, and the account appeared sound.

The finding

A side-effect linkage check was then run against a live session rather than against test fixtures. The verdict file from June 8 is blunt. Verdict: CAPTURE_GAP. Across 142 spans enumerated for the session, the count of decisive side-effect attributes — the recipientClass, anonymous-creator, and glasstrace.side_effect.field values on which the admission argument rested — was 0. Non-missing sideEffectSummary or sideEffectEvidence fields: also 0.

The captures that had passed NR-2 had been exercised against mocked side-effect evidence. The live path had never produced the decisive fact.

The root cause was mundane. The environment in which NR-2 first passed happened to have Mailpit, the development mail-catching service, running. The later live suite did not replicate that environmental detail, the mail side-effect transport silently disappeared, taking the evidence with it. The fix dispatch states it directly: NR-2 passed earlier because it ran in an environment that had Mailpit up; the live suite regressed by not replicating that.

Why this was worse than an ordinary defect

A failing test reports that something is broken. This was the opposite condition, and a worse one: a passing gate that had only ever observed synthetic data. Mock-based checks validate plumbing. They confirm that spans flow, schemas parse, and attributes land. They cannot confirm that the world produces the facts the product promises to capture.

Bug admissions had rested on that gate. The design document dated June 12 does not soften this. It names the condition (“the live capture-path gap, G-TRACE-READY-LIVE FAIL, 2026-06-08”) and draws the necessary conclusion: the June 8 finding made the live gate, not synthetic NR-2, the real admission gate. The same document describes the remediation as ending the “synthetic-NR-2 era.”

Re-testing under the live gate was promptly humbling. The first candidate expected to pass, FALLBACK-ICS-001-obf, failed, recorded as decisive_fact_missing_from_natural_query: a route-only trace with no decisive fact reachable through natural agent querying. The rest of the roster, RPORT-004-obf included, was not even run; the VR-1 capture-readiness sweep of June 12 halted with status PENDING/BLOCKED because its expected-pass baseline was broken. The one preflight inside that sweep which did pass turned out, on inspection, to be a synthetic positive control as well. The value-fidelity smoke test returned PARTIAL/WARN, with get_span_attributes returning an empty span list on the very surface intended to demonstrate value capture.

June became the validation repository’s heaviest month to that point, at 106 commits, most of them rebuilding the gate infrastructure so that “capture-ready” meant live-capture-ready. The Prisma value-capture runbooks and the owned-span recordSideEffect instrumentation landed in the days that followed the finding (glasstrace-validation@60cd87b and 14f7c4e on June 12, b908745 on June 13, DISC-1929); the Mailpit transport was repaired and pinned into the environment definition; and the live-gate admission machinery followed in commits such as 724b133 (recording the MFG-RLY-021 live capture gate) and fbeca1a (correcting its admission logic).

What the incident settled

Every evaluation-driven project faces the same trade-off. Synthetic evidence is inexpensive, deterministic, and convenient for continuous integration. Live evidence is expensive, unreliable, and slow. The incentives therefore push gates toward synthetic data, while product credibility comes from live data.

The resolution was not to abolish synthetic checks, which still catch plumbing regressions cheaply. It was a hierarchy. Synthetic checks may block, failing quickly on broken plumbing. Only live checks may admit. No bug enters the roster and no claim enters a document unless the decisive fact was captured on the real path and retrieved through natural agent queries. That blocking-versus-admitting distinction has governed every admission since, and it dates from this incident.

The hierarchy leaves one limitation untouched: the bugs themselves are still manufactured. The live gate governs the provenance of the evidence — an actual application executing a genuine failing flow, producing spans through the live capture path, retrieved through natural agent queries. It does not govern the provenance of the defect, which in the current roster is deliberately seeded or reconstructed rather than organically encountered. The two limitations are distinct and resolve on different schedules. Ending the synthetic-evidence era was a correctness repair; a capture pipeline that had never been proven against reality now has been. The manufactured-bug limitation is a generality constraint. It bounds the population any result can speak for, it appears in the scope statement of every subsequent study (“one seeded hard bug”), and the program’s response to it — the measured scarcity of admissible defects, and the plan to source organic incidents through design partners — is the subject of Part 6. June repaired only the first. The second remains open, and it is disclosed in every study since.

There is a process lesson as well. The finding did not come from re-auditing NR-2, which everyone trusted. It came from building a second, differently shaped check (live linkage against a real session) and from being willing to run that check against results already recorded and banked.

One month later the live gate passed 10 of 10 attempts on the flagship bug. Getting there involved a ten-millisecond race condition inside Next.js, which the next article covers.

Supporting data and commit references

Evidence availability: references to glasstrace-sdk are public — the repository, commits, and files are linked in the series evidence manifest. All other repositories cited above are private; artifacts may be shared as needed, and the manifest publishes each cited artifact’s SHA-256 digest so that any shared copy can be verified against the version these articles describe.