Customer effect
A normal request did not reliably produce a persistent public page.
Technical record · dated evidence
July 2026 multi-day production regression
Between roughly 25 July and 1 August 2026, the core product path on app.noetfield.com — sign in, type a normal sentence to the front person, receive a public landing page in about a minute with plain client copy — was impaired or broken for multiple independent reasons at once. This was not one bad page, one bad model call, or one eight-hour debug session. Repository and deployment records show a stack of contract, configuration, state, and verification defects layered on top of a six-day production deploy freeze while motor staging kept moving.
Failure classes
Deployment drift, runtime configuration, job state, deterministic gate contracts, and false delivery signals.
Control corrections
Matched releases, sentence-form E2E tests, independent public fetch, canonical identity resolution, hard stops on deterministic defects.
Current status
Required historical matrix rows were verified on 1 Aug 2026 against app release 2b818e41…. Generic trade path remained open. Full current closure requires a fresh same-release production E2E — do not treat this page as closed without that verify. Live tip: /v1/release.json.
Cost-of-governance lesson
Repeated paid generation did not fix unchanged deterministic metadata. Each retry spent budget against the same wrong expected value. The contract—not the content—had to be repaired.
Deploy split-brain
Gate contract bugs
Runtime 1101 dead path
False green health
Stale success UI
Internal jargon in chat
Wedged projects
False delivery receipts
Incident window
Last known-good deploy 25 Jul → recovery merges 1 Aug
Production deploy gap
Six days with zero successful Company New deploys
Motor staging drift
Multiple motor staging deploys 29–30 Jul while Company New production had zero successful deploys 26–31 Jul (unsourced aggregate commit totals omitted)
Closure rule
Fresh same-release customer-effect E2E — not commits or health alone
What worked before
Through 24–25 July, repository records and production deploys show a working
sentence-to-site path: natural language request → build completes quickly → public preview URL →
clean front-person copy without pipeline codes. Last successful Company New production deploy before
the freeze: 6e192c34 (25 Jul), including PR #316 golden-path work (Cursor co-authored).
At that SHA the app did not yet ship the later promotion gate stack.
Separately on 24 July, agents also caused UI surface damage (empty shells, wrong surfaces edited) — documented in INC-2026-07-24 UI structural damage. That incident is related trust debt but a different failure class from the July 25–August build pipeline regression below.
System picture — five layers failed together
| Layer | What broke | Why “green” lied |
|---|---|---|
| Deploy | No Company New production deploy 26 Jul–31 Jul; motor staging deployed repeatedly | App shell and runtime rules diverged for days without a matched pair shipping together |
| Runtime config | PR #476: CI overwrote scoped API keys with legacy shape → Cloudflare 1101 on all job routes | /healthz still answered; app reported dispatch ready |
| Database / jobs | PR #481: migration replay downgraded enqueue; #482–#488 wedged goals and recovery | Requests accepted; no job or no exit from stuck state |
| Publish gates | PR #322 + follow-ups: six deterministic gates on promotion; title-as-brand wiring | Verifier and unit tests passed while customer sentence metadata was wrong |
| Client truth | Stale preview state, false completion, jargon in chat, no external URL verify (#491, #493) | UI said “passed quality check” while publish path carried an error |
The 31 July recovery session merged many repairs in this stack but repeatedly inferred “fixed” from commits, health endpoints, and unit tests — not from the signed-in front-person journey producing one correct persistent public page.
Customer-visible failure modes (multiple projects, not one prompt)
1 — Builds never start (dead gateway)
After PR #476, every job route on staging runtime returned Cloudflare 1101 with an empty body
while /healthz stayed green. The app could still look ready to build. No customer sentence could
complete a path through a gateway that rejects all job traffic.
2 — Jobs accepted but nothing runs
PR #481 replayed an old migration and downgraded the enqueue function. Symptom: intake succeeds, no build job is created. Owner waits on a UI that suggests work is in progress.
3 — Entity gate compares page to wrong identity
Commit f8701f62 (26 Jul, Claude co-authored) passed project.title — the first ~60
characters of the order, often truncated mid-word — as the formal brand for entity-consistency.
Multiple real prompts hit this class:
- Meridian Tax Group — gate expected
Landing page for Meridian Tax Group, an accounting firm for - Northline Bookkeeping — gate expected
Build a landing page for Northline Bookkeeping, a bookkeepinwhile brief already held the correct company name (commit850c8030notes) - Generic trade phrases — e.g. dental booking invented as a company name and demanded verbatim in copy (#504 partial fix; #505 open)
Correct HTML could pass every model check and still be refused against an impossible or invented expectation.
4 — Brief autodraft invents the wrong business
On a live production project (project ID redacted from this public record), workspace UI showed restaurant / local diners / “See the menu and reserve a table” while the customer asked for an accounting firm with a consultation form. Keyword rules ran before reading the stated trade (PR #505 addresses). Two inventions on one page — wrong workspace fields plus gate failure in chat.
5 — Wrong project built
PR #486: opening New workspace silently rebuilt the previous brief instead of the new prompt.
6 — Page built but withheld or not public
PR #487: validation expected prompt wording echoed in page content. PR #491: cockpit announced completion
without a published page — links returned NOT_GENERATED. PR #489/#493: shareability and delivery
receipts did not match what an unauthenticated fetch of the public URL showed.
7 — Failure and success shown together
Representative chat pattern (Meridian class, repeated across poll retries):
The build did not produce a page… Stage: app publish (poll).
CONTENT_GATES_FAILED: entity-consistency: expected company name "…truncated sentence…" absent from visible copy
Your landing page passed quality check. Open Preview in Documents / Website…
Two state sources disagreed: stale previewReady from older R2 HTML vs current publish error.
Duplicate suppression failed to collapse equivalent failures. Internal commissioning telemetry
(rungs, attempts, acceptance counts) also appeared in client-facing surfaces — not operations-only copy.
8 — Recovery session tested the wrong thing
Internal harnesses used clean company names. Real customers type sentences. PR #496’s sentence-form E2E disproved earlier “fixed” declarations. Paid regeneration could not repair unchanged deterministic metadata — each retry spent budget against the same wrong expected value.
Timeline (repository + deploy evidence)
| When | Evidence | Effect |
|---|---|---|
| 24 Jul | UI structural damage chain (#276–#293); many production deploys | Surface overwrite trust debt (separate incident doc) |
| 25 Jul | Last prod deploy 6e192c34 (#316 golden path) | Last known-good app bundle before freeze |
| 26 Jul | f8701f62 — title passed as brand | Truncated orders become gate identity |
| 27 Jul | PR #322 merged — six content gates + promotion authority | New publish bar on motor/app path |
| 26–31 Jul | 0 successful Deploy Company New production runs | App frozen; motor staging deployed repeatedly (GHA runs 29–30 Jul) |
| 29–30 Jul | Repeated motor staging deploys (SinaOS / runtime churn) | Runtime rules changed under stale app shell |
| 31 Jul morning | #476 merged — API key shape overwrite | 1101 on all job routes; health still green |
| 31 Jul day | #478–#496 repair cascade; Claude Code-assisted recovery session (session ID redacted) | Many merges; owner still reproducing failures on real path |
| 1 Aug | #498–#504 recovery merges; deploy 5b069578 | Brand extraction, plain errors, false-ready fixes, CI |
| 1 Aug | Postmortem page PR #508 deployed (static asset) | Public record at this URL |
Repository repair map (31 Jul – 1 Aug)
| PR | Defect class | Customer symptom |
|---|---|---|
| #476 | Runtime API key CI writers disagreed | Opaque 1101; builds dead |
| #478 | Stale deployment identity in config | Runtime drift; routes disabled |
| #481 | Migration replay | No enqueue / no job |
| #482–#488 | Stuck goals / recovery unreachable | Projects wedged |
| #486 | New workspace replay | Wrong brief built |
| #487 | Prompt echo required in HTML | Good pages withheld |
| #489 | Shareability copy | Partner link uncertainty |
| #491 | Completion without publish | NOT_GENERATED URLs |
| #493 | No external URL verify | False delivery receipts |
| #496 | Sentence-form E2E boundary | Disproved prior “fixed” claims |
| #498 | Brand extraction + plain errors | Cursor co-authored Meridian-class fix |
| #502 | False-ready / stale R2 path | Withhold preview on publish fail |
| #504 | Generic category as brand | Trade-only prompts partially repaired |
| #506 | Service category ≠ business name | Open — generic trade path (#505 superseded) |
| #322 | Gate stack introduction (130 commits, 182 files) | Claude co-authored; root regression class |
Verification matrix
Filed receipts and row status come from
status.json, updated by
npm run e2e:incident-matrix. Public receipts use hash-bound project refs only — no raw project IDs.
Sealed named-entity verification (Meridian class)
Production E2E on 1 Aug 2026 against
2b818e415a2841d7e9eb390ba9f464722f92e8c7:
HTTP 200 · H1 Meridian Tax Group · 13,684ms · no jargon in first chat reply.
- Receipt:
named_entity_sentence.json - content_sha256:
3daa06f7e116f42346573ed357e2c0f2951bba8732ca4934a1cb6bb5e3c76aba - Evidence lab: agentic-coding-reliability-case-001
| Prompt / probe | Status | Gate | Last run | Summary | Receipt |
|---|---|---|---|---|---|
| Loading matrix… | |||||
Live strip above reads /v1/postmortem/2026-07-app-incident/live-status
(server-side runtime probe + release marker). A single passing row does not close the incident.
Lessons
- Ship app + runtime as a matched pair. Six days of motor-only movement created split-brain.
- Canonical identity ≠ raw prompt ≠ truncated title. Gates must not invent or truncate company identity.
- Health ≠ customer path.
/healthz, unit tests, and “motor succeeded” are insufficient alone. - One client truth. No success UI from stale artifacts; no jargon in front-person chat.
- External verify before “delivered”. Unauthenticated fetch of the public URL is the final gate.
- Recovery = reproducible signed-in E2E matrix. Not commits, not one sanitized harness input.
- Stop paid retries on deterministic metadata defects. Repair contract, don’t regenerate the page.
Evidence index
- Evidence lab — agentic coding reliability case 001
- Deploy Company New workflow
- INC-2026-07-24 UI structural damage
- Commits:
f8701f62(title as brand),850c8030(brief.company attempt),b5d33169(#322) - Motor staging deploy run
30687802615· Company New deploy runs30687758703,30688763618
Dated public record. Status claims apply only to the SHA and time you read /v1/release.json. This page intentionally does not reduce a multi-day, multi-defect incident to a single customer example.