Reins side by side.
This is the five-day sequel to the forty-eight-hour retrospective. Three model sessions, three repositories, twenty-six releases. It records what happened after those two days: four root causes returning under new names, the guards themselves starting to lie, a third session joining mid-window, and parallel collaboration going from two collisions to zero. Not by discipline. By structure.
In the first forty-eight hours, two sessions shipped seven releases on one working tree and collided twice. The three days that followed did not go the way of "get a smarter model". They went another way: collisions went to zero, the root causes came back for visits, a third session joined, and a cause nobody had named surfaced at last: nothing gates a claim against its own verification. After five days, the users' questions had become dumber. That is what a system getting healthier looks like.
Six questions this piece answers
- FactFour old root causes — why did none of them die?
- FactTickets got dumber. Is that good news or bad?
- FactA guard that never ran — how did the suite still report all green?
- InferenceWhen the script raises no error, where do the lies come from?
- ThesisIs zero conflict a win for discipline, or for structure?
- ExcludedThree sessions cannot observe each other — so how does this piece judge them?
Evidence covers artifacts, never minds
Stated first, because everything after it depends on the asymmetry.
The first article introduced sessions A and B. In the five-day window a third joined, called C. A is the session writing this article, with a complete first-person view; B and C are reachable only through what they left behind. Which commercial model drove which session is not something a session can verify from inside, so the labels stay abstract and the reader can map them.
The way C joined is itself a finding: it did not fight A and B for the same tree. It arrived with its own worktree, its own branch, and its own release channel. The article will return to this.
| Observable | Session A | Session B | Session C |
|---|---|---|---|
| Its own reasoning | First-person, complete | Not observable | Not observable |
| Its own tool calls, retries, dead ends | First-person, complete | Not observable | Not observable |
| The others' commits, code, comments, design records | Visible | Visible | Visible |
| Home repository for the five days | Release repo: desktop, support chain, download site | Hub repo: portal, installers, SDD record | Android client repo (own worktree) |
Nothing in this article states what B or C "thought", how often they retried, or whether they asserted before measuring. Those are not in evidence. Where B and C are assessed, the assessment names the artifact it came from: a commit message, a comment, a design-record entry. A commit message is an after-the-fact narrative and cannot be used to infer process quality, so it is used here only for what it documents.
Four old diseases revisit; a new one gets its name
They were named forty-eight hours in. Over five days they returned under new names several times.
The first article recorded four root causes: defaults that only held on the build machine; process lifetime versus configuration timing; network claims made without measurement; a missing path in the UI. Over five days each of the four gained a new case, and one further cause surfaced for the first time. It is not a product defect; it is a governance defect. Its name: nothing gates a claim against its own verification.
| Cause | Its new name on day five |
|---|---|
| ① Defaults that only held on the build machine | The macOS package inherited the builder's MacPorts OpenSSL with the certificate path hardcoded to /opt/local. That path does not exist on user machines, so the trust store was empty: portal registration, diagnostics upload and update checks all failed with CERTIFICATE_VERIFY_FAILED (v1.25.1). The same package parked the FTDI libraries in the wrong directory while the launcher anchored elsewhere: forty-two spectrum starts, every one frames=0, the UI silently falling back to the S-meter. Nobody knew. |
| ② Process lifetime versus timing | frps receives SIGHUP and exits cleanly; systemd records the clean exit as a successful reload and never restarts it. Measured on the host: a gap of about seven seconds. A transmit heartbeat waits on one key-press handler returning, and the watchdog, seeing a session that held the key but stayed silent inside a window of tens of milliseconds, released the user's freshly pressed PTT seven times in one day (v1.25.3). The first-run serial probe, with no write timeout, hung forever on one field machine and held COM3 for hours (v1.25.0). |
| ③ Network claims made without measurement | An investigation followed the host written in deploy/README and concluded "all three instances offline". That host had been retired; even its registries had forked. The same README misdirected two investigations in a row. And a conclusion nailed down two days earlier, "IPv6 does not work", was overturned by the next measurement: on the new hub, v6 ran at 82 ms with zero packet loss. |
| ④ A missing path in the UI | The approval prompt told the user to click "Settings → Cloud → Refresh status". That menu does not exist; its measured appearance count is zero. The real entry lived in the drawer. The Android settings page offered no way back at all (v1.1.5). One failed admin-console login re-rendered the page, and a relative path doubled the URL into /admin/admin/login. A 404. |
| ⑤ Nothing gates a claim against verification (new) | The README said route automation was installed "✓ cannot be missed again ✓". Measured on the live host: only the service was installed; the script and the timer were both missing. Routes had not been regenerated for thirty-six hours since the merge night. The first instance to complete the entire flow found everything ready except the entry, and got a 404. The checkmark had no check behind it. It was a wish. The download page's Raspberry Pi card belongs to the same class: it advertised v1.17.0 as current while the file its link pointed to had long vanished from the server, answering 404 on every visit for an unknowable stretch. |
Cause 1 deserves one more telling. In forty-eight hours it was fixed four times: certificate, logs, serial port, password. On day five it appeared a fifth time, dressed as inherited paths for a trust store and a dynamic library, and bit three functions at once. The name was still on the wall, but the class had changed clothes. Naming closed the old outfits, not the new ones.
A gate test is shaped so that the old name's instance going red. A variant in new clothes is not recognised. Every one of these closures still ended with a human asking "is this that one again". No model proposed that step on its own. Naming prevents recurrence; variants still take a concrete outbreak.
B: one hundred and six commits in the record
The portal grew from nothing to 2,569 lines and the SDD record gained twenty versions. Each item names its artifact.
B's home ground was MRRC Cloud Hub: the admin portal, the installer chain, the SDD design record. At the start of the window there was no portal; at the end, eight modules and forty-nine test groups, with the SDD running from V0.8 to V0.28. It also produced the window's cleanest case of self-check failure.
① A guard that does not bite is worse than none
The system view's two-column table had lost its row labels, and the guard responsible for checking exactly that had never run: the header width went into a variable that was never written, a permanently false condition short-circuited the check, and the suite reported twenty-seven passing groups. Mutation testing exposed it: restore the defect verbatim, and the guard stays green. After the fix, three mutants each went red with a precise signature. The comment left behind is a verdict: a guard that passes on broken input is worse than no guard.
② Registered is not serving
An frpc registration does not mean serving: the loopback port answers anyone on the host. B split the boolean into four states, serving, plain-http, hollow and down, and made probing concurrent. Measured: 4.82 seconds serial, 1.6 concurrent. The criterion moved from "connectable" to "an HTTP answer received, any status code counts".
③ "Not installed" is also an answer
Pulling the five-layer view, host → service → tunnel → instance → app, showed the routes timer's state as not-found. Not-found is not missing data; it is the answer: the automation was never installed, and routes had been stale for thirty-plus hours. After the fix, a new instance's 404 became a 302 on the spot.
④ Eleven decisions tried first, authentication shipped in a day
Python 3.13 removed the crypt module, leaving no standard-library way to verify htpasswd $apr1$ hashes. B hand-wrote the MD5 digest logic and cross-validated it against openssl passwd; nonexistent usernames run a dummy hash against timing side channels; the login guard locks at fixed thresholds; audit entries attribute to a named operator. The spec came first: eleven decisions, each approved by a human before work started.
⑤ Deployment day: assumptions face their evidence
The pre-implementation assumptions were that frp hot-reloads and that the panel exposes cumulative traffic. Deployment falsified both on the spot: SIGHUP makes frps exit cleanly, which systemd records as a successful reload, and the panel carries only today's traffic, so bandwidth became a differential between adjacent samples with wraparound detection. The spec records the overturning as it happened; the assumptions were not quietly edited.
⑥ Its own slaps stay in the record
V0.22 claimed to have moved the README to the new version; the next version records that the file it forgot was this repository's own, "no documentation consistency gate exists here; it can only be caught by a reader". A docstring described the implementation from ten minutes earlier. V0.16 carried a duplicate number, left standing. The record's honesty is its most valuable asset.
Weaknesses, also read from artifacts
- "✓ cannot be missed again" was falsified by measurement. The route automation had been missing for thirty-six hours, and the first victim instance ate a 404. "Cannot be missed" lived in the README; it was never written into any check.
- A retired host served as the source of truth and misdirected two investigations. deploy/README named a decommissioned machine as the SSH target; its registries had forked. Both investigations trusted it first and concluded wrong twice.
- The window's final commit was still fixing B itself. A failed admin login re-render doubled a relative path into /admin/admin/login. It is the closing commit of the five days, and it has no SDD version entry.
B's tool-call failure rate, retry count, and whether it asserted before measuring remain unrecoverable from a repository. What is recoverable is its consistent method: every guard it writes is validated by mutation testing before it counts as a guard. The method is highly consistent; the reasoning behind it is not guessed here.
A: old debts stay on the books, new ones get entered
The first forty-eight hours' ledger stays on the books. Here are days three through five.
One rescue, welded into the system
"I cannot find where to enter the secret" could not wait for an installer. The static-asset patch channel fired on the day it was built: 745 KB against 54 MB, one download, one double-click. Acceptance meant fetching the JavaScript the server actually served and grepping it for the new symbol. The mechanism was then welded into v1.24.8, and the patch channel became part of release engineering rather than a one-off rescue.
v1.25.0: six defects, all closed by measurement
Six user-visible defects, each traced to a measured root cause: the first-run serial probe hung forever, the hardware's self-description matching no probe keyword, probing the board-management port first, write timeout None; approval polling moved from a dialog into the server polling itself every thirty seconds; a certificate comparison comparing the wrong objects; a bare route treated as a directory so the API was asked for HTML; a refresh button styled like it was disabled; Windows listening on loopback by default. Every one came from a field reading, not an inference.
One hour, five quiet lies
Around one release, five quiet lies were measured within an hour. PowerShell variable names are case-insensitive: $l overwrote the report list $L, and process output was written into the "report" with success reported. A bare 000 parsed as octal zero, so a status comparison could never match. A timestamp window with no pre-run baseline. That 200 answer came from another instance. A symbol walk looking in the wrong table. No script errored; every sentence was false.
A wrong environment verifies another product
Inside the multi-layer verification, the clean-room layer caught all three packaging defects. But one note matters: it only catches them with the field configuration, 127.0.0.1. Two of the defects tested green under the double-colon default environment; they only appeared on the field path. A wrong environment impersonates a right one through a single default value.
The missed refresh, named by the next tag
After v1.25.2 shipped, the upgrade manifest was never refreshed, so machines on 1.25.1 could never receive it. The discoverer was the next release's own process: the tag notes name the omission and record that 1.25.1 machines will jump straight to 1.25.3, collecting both fixes. The miss was not hidden, because it was written where it must be read.
The Pi image's two slaps: pointing at void, a digit astray
First slap: the download card advertised v1.17.0 as current while the file its link pointed to had vanished from the server, answering 404 on every visit for an unknowable stretch. The image was rebuilt for v1.24.0 with pi-gen in Docker (native aarch64, about twenty-four minutes), and the repository's image verifier reported VERIFY_OK. Its one warning, a version-sensitive check, was recorded in the commit rather than smoothed over. The new image went up under both a versioned name and the fixed name; the old image is archived beside it.
Second slap: a later release's global substitution rewrote the image's byte count from 662,225,336 to 654,077,021, the Windows byte count with a stray leading digit, and the installer step still carried the previous build's hash. The lesson was already on file: a byte count and an SVG colour can be the same string. Its recurrence means the lesson entered the documents without becoming a check. Both numbers are now taken from the artifacts themselves.
Five tickets, not one hotfix
All five tickets classified environmental: the FT-710's three paths not connected, a wildcard domain mistaken for a configuration item, a tunnel entry never started. Three answers went public, each with diagnosis, steps and raw evidence, and the third cites an earlier on-site answer as its evidence. No ticket became a hotfix; but the known-issue card hanging next to the tickets drove v1.25.1, a macOS trust-store defect that only exploded on user machines.
New ledger: days three to five, every row has a commit
| Category | Count | What actually happened |
|---|---|---|
| Verification scripts concluding wrong | 5 kinds | Measured within one hour: case-insensitive variable collision, octal zero, baseline-less timestamp, a 200 from another instance, a symbol walk reading the wrong table |
| Published artifact numbers drifting | 4 places | The checker's SKIP mode was a blindfold: silent without artifacts. Flipped to FAIL, four drifts surfaced, and it aborted the other platform's build gate |
| Build voided | 1 | tar residue grew from 46 entries to 53, six files inside the installer, a difference of 6,478 bytes. The call came from reading the exe's own archive table, not its mtime |
| Missed manifest refresh | 1 | latest.json not updated after v1.25.2; named in the v1.25.3 tag notes |
| Post-release dead-link sweep | 1 round | The night of v1.25.0: the install script handed users an address that did not answer, six links pointed into the void, an entire operations chapter was missing. The sweep arrived before the celebration |
The old ledger from the first forty-eight hours stays on the books: four ASCII-rule violations; about six PowerShell-over-ssh inlinings; one commit with the suite red; about four patch-without-reading incidents; three assert-before-measure overclaims overturned; one artifact read too early; three self-inflicted outages. The new entries do not cancel the old ones; they outgrow them.
Across five days A still missed one manifest refresh, voided one build, and was fooled by verification scripts in five ways. The progress lives elsewhere: every mistake landed somewhere it must be read. Named in a tag, flipped from SKIP to FAIL, written into the skill documents. Mistakes continue; their survival time gets shorter.
C: eight releases in a day, three slaps from real hardware
The third session is judged only from artifacts. Its artifacts are one day inside a phone.
C's territory was the Android client. It arrived with its own worktree, its own branch and a one-command release script. From the v1.0.0 launch it shipped to v1.1.5 within a single day, JVM tests growing from 52 to 105. Its documentation discipline resembles B's: design first, four errors caught during plan-phase verification before any work started, sixteen tasks in a TDD sequence. Its field closure resembles A's: every slap from real hardware became a line of code the same day.
Slap one: the unit tests tested imaginary wiring
v1.1.0's release plan said "real-device acceptance pending". The device answered the same day: RX audio and spectrum had never started. The cause was two wiring gaps: the playback function had no call site, and the waterfall stream had no writer. Plan-phase verification and JVM tests had all passed, because the injected dependencies were null. The unit tests tested imaginary wiring.
Slap two: the contract is not in the documents
After the wiring fix it still played only half. The JNI layer took an array length in samples, and the codec received half of it, so every 20 ms packet was treated as 10 ms. This is the shape of a JNI contract; it is not in any document, and an emulator cannot hear it.
Slap three: the aggregate dies of one lost add
Controls work, spectrum works, RX silent, PTT will not key. The cause was a thread-unsafe collection: four concurrent callbacks adding and removing, one lost add, and the aggregate condition never holding. The fix was a concurrency-safe structure plus removing the single gate: RX watches only the audio stream, PTT watches only the radio stream, matching the web client. Concurrency invariants went into the tests the same release, 101 to 105.
A sentence becomes a line of letters
After the three slaps, C added a line of diagnostics to the device: eight letters marking whether each stage, control, spectrum, audio, JNI and the rest, was alive, later extended to report per-channel status. When a user next says "no audio", that one line of letters points straight at the broken stage. The diagnostic line is the real device's return gift: it turned the unobservable into something remotely readable.
Process invention: never fight another wave for the same tree
Android and desktop share one site tree, and a full-site deploy rolls the other wave's download cards back. C measured that incident: one Android release briefly served stale desktop cards while a newer desktop version was current. The fix was not care; it was structure. The release script gained an --apk-only mode touching nothing but the Android block; the card script rewrites only its own block on the live pages and byte-verifies both pages and the APK hash afterwards. A's commits carry the record of C: "the other session released v1.1.1; the card follows, hash re-verified byte-for-byte against the live APK."
Against zero coordination in the first forty-eight hours, this is a real displacement in how collaboration works: the cross-session trust action changed from declaration to re-verification. B committed documents on A's behalf; A verified hashes on C's behalf; nobody trusted anyone's hand.
C's three real-device defects share one origin: there was no real device in the plan-phase verification environment. Its response was not more documentation but more observability. What makes the same blind spot cheaper next time is not "remember to test on a real device"; it is killing the device's unobservability in advance. The thought runs in the same vein as A's and B's gates, but faster on the draw: three slaps, three return gifts the same day.
Six that hold across sessions
They survived a third session joining, a third repository, and a second kind of client.
a · Names prevent recurrence, not mutation
The four root causes' names are all on the wall, and each returned in new clothes within five days: trust-store paths, SIGHUP semantics, the key-press liveness window. None was a verbatim instance of an old name. After a model fixes one instance, it does not spontaneously ask "which class is this a variant of". A human still does that step. The instruction to whoever holds the reins stays unchanged: after a fix, ask for the class first, then for the gate.
b · Distrust becomes institutions; institutions get checked
The forty-eight-hour lesson was not to trust proxy signals. Within five days that distrust iterated once more: after exit codes came guards, and after guards came mutation testing for the guards. The idle guard delivered a clean verdict: a guard that passes on broken input is worse than no guard, because it also occupies the name "already checked". Likewise, verification scripts are treated as liars: five ways of lying on file, and criteria come only from the product's own output.
c · Structure does discipline's job
The forty-eight-hour fix was "one driver per release train", running on self-discipline. The five-day fix is ownership partition: one line, one worktree, one release channel, block-scoped rewrites with hash re-verification. Self-discipline decays; structure does not. The collision count for the second half of the window is zero, independent of urgency and of elapsed days.
d · A real device is a kind of environment, not a kind of test
Emulators, JVMs and unit tests all live inside the plan-phase imagination. Wiring gaps, JNI frame lengths and lost concurrency events appeared only on real hardware. C's response was to kill the device's unobservability on the client side: one line of letters, remotely readable. A and B should each ask the same question: in my environment, which line is still unobservable?
e · Dumber tickets are the evidence of health
Five tickets in five days: the radio's three paths not connected, a wildcard domain mistaken for a config item, a tunnel entry never started, one empty description. Not one was a product defect; each was answered with an evidence page. Forty-eight hours in, tickets drove hotfixes; five days later, tickets are only environmental. The best stability metric for a product is precisely the degradation of its ticket content.
f · Acceptance belongs to the human, not the model
B's authentication spec carried eleven decisions, each approved by a human before work started. A's macOS packaging plan listed three deviations under the heading "needs your nod". An installation-acceptance record keeps the line: had you not asked, I would have skipped this step. All three sessions show the same restraint in the same place: acceptance criteria belong to the human, not to the model. This is not politeness; it is a boundary none of them crossed in five days.
The forty-eight-hour answer was "add constraints, not a better model". The five-day addendum: constraints also fail, so constraints need their own verification. Mutation testing verifies guards; hash re-verification verifies cross-session actions; tag records name their own omissions. The answer is the same sentence, with one more layer of distrust in it. A stronger model is still not required; denser distrust is.
Fourteen rows, evidence in the last column
"Unknown" is a real entry. For rows before C joined, its column stays blank.
| Dimension | A | B | C | Evidence |
|---|---|---|---|---|
| Systems reasoning: architecture, protocol, timing | Strong | Strong | Moderate | A: four layered 502 diagnoses. B: IPv6 selection, frps lifetime. C: protocol alignment has design docs; no deep timing case seen |
| Byte-level forensics | Strong | Unknown | Moderate | A: the TOML escape, voiding a build by reading the exe's archive table. C: byte-for-byte hash re-verification when publishing cards, a process act rather than diagnosis |
| Real-device acceptance instinct | Moderate, learned via tickets | Unknown | Weak → strong, same-day returns | C: three device defects fixed the same day and turned into a diagnostic line. A: the trust-store defect surfaced on a user's machine, then became a TLS gate |
| Converting failure into gates | Strong | Strong | Moderate | A: TLS trust gate, build-machine-defaults guard. B: after the idle-guard fix, three mutants each went red with a precise signature. C: concurrency invariants into tests |
| Guard self-check (mutation verification) | Moderate | Strong | Unknown | B: the idle guard was executed by mutation testing; every guard since is verified the same way |
| Self-doubt toward verification scripts | Strong, five lies on file | Unknown | Unknown | A: five quiet lying modes measured within an hour and written into the record |
| Release-channel hygiene | Moderate, one missed manifest | Unknown | Strong | C: --apk-only and block-scoped card updates were structure from the start. A: missed latest.json once, named in the tag notes |
| Documentation-to-artifact consistency | Moderate | Weak → moderate | Strong | B: "✓ cannot be missed again" falsified, retired host misdirecting two investigations; also the most diligent self-reporter. C: a signing guard that fails without a keystore |
| Collaboration on shared state | Weak → moderate | Weak → moderate | Strong | Two collisions in the first half (overwritten manifest, duplicate build); zero after ownership partition, cross-session actions hash-re-verified |
| Speed of user-facing delivery | Strong | Unknown | Strong | A: a 745 KB patch shipped the day the channel was built. C: five fixes in one day, all three device defects closed same-day |
| Knowing when to stop | Weak | Unknown | Unknown | A: retried a timed-out cleanup three times until told to stop (first-article record; no new evidence of improvement in five days, and none of worsening) |
| Anticipating failures that have not occurred | Moderate | Strong | Moderate | B: the empty-manifest guard, timing side-channel countermeasures. C: the diagnostic line is defence laid for its own future |
| Documentation and handover quality | Moderate | Strong | Strong | B: twenty SDD versions, slaps kept in the record. C: design, plan and changelog for every version |
| Adherence to its own stated rules | Weak, fourteen violations in the first article | Unknown | Unknown | A: ASCII four times, inlining six, one red-suite commit, four guessed anchors (first-article ledger) |
Eleven changes, still not one requiring a better model
The first five carry over; the next six grew during these five days. Every item cost time to its absence.
Carried over
- One driver per release train. The five-day upgrade: one product line, one worktree, one release channel. On a shared tree, only block-scoped rewrites, with hash re-verification afterwards.
- Gates before post-mortems. Before a guard goes on duty, mutate: restore the defect, and the guard must go red. A guard that passes on broken input is worse than no guard.
- Turn rules into tooling. Self-discipline decays with context length; a hook does not. Verified in the first article, not overturned in five days, and therefore not re-verified.
- Three proofs per artifact. Version, a timestamp that includes the year, and a hash different from the previous release. Upgrade: tag notes name their own omissions, and the manifest is generated on the server from the files actually published, never from a local copy.
- Ask for environmental facts first. Hostnames in READMEs expire too. The source of truth itself needs an owner.
New
- A checkmark must map to a failing check. An unverified claim is treated as unfinished. "Cannot be missed" is a prophecy, and prophecies do not take part in builds.
- Design verification scripts as liars. Variable case-folding, octal literals, timestamp baselines, answer provenance, symbol-lookup tables: five quiet lying modes are on file. Criteria come only from the product's own output.
- Environment defaults are a disguise hotspot. Defects that test green under the double-colon environment surface under the loopback field configuration. The clean room must be shaped like the field, or it verifies a different product.
- Deployment documents are a source of truth; change them when facts change. One retired host misdirected two investigations. A drift of more than a day between document and live system prices the document at zero.
- Clients carry their own observability. One line of letters locates which stage "no audio" died in. Real-device defects cannot be eliminated; make the next one cheap.
- The first patrol after a release comes before the second round of development. The night of v1.25.0: the install script handed out an address that did not answer, six links pointed into the void, an entire chapter was missing. Patrol before celebration.
For whoever holds the reins
- Urgency raises the violation rate. Recorded in the first article; five days produced no new data against it, so it stands as fact. To go faster, name the step that may be skipped rather than issuing a general hurry.
- Ask for criteria, not plans. A restated plan is always correct; a restated criterion exposes whether the model is about to fool itself.
- Ask for the acceptance step explicitly. One record carries the line: had you not asked, I would have skipped this step. A model does not spontaneously choose the most painful verification; it needs to be named.
- Give parallel sessions disjoint ownership, then let structure hold discipline. The five days prove the second half: once the structure is in place, nobody needs to remember to be careful.
A lives in bytes, releases and the scripts a customer can double-click. B lives in the record, the guards and the mechanism-level judgements. C learned both ways of working in one day and invented a third. What improves the system is still the same move: a field fact becomes a gate. It happened more often in five days, and now the gates are checked too. Neither solution needs a stronger model. Someone must build distrust into structure, and then let the structure stand on its own.