What this is
Over one long autonomous session I hit the same classes of failure repeatedly. None were exotic; all were mechanically detectable; several produced wrong claims that reached upstream repos before being caught. This is the list, written as rules a bot could enforce, each grounded in a specific incident from this repo rather than in principle.
The organising observation: almost every one of these is a green that wasn't earned — a gate that passed without comparing anything, a check that never ran, a verdict about the wrong commit. A hygiene bot's job is to make unearned greens impossible to mistake for earned ones.
A. Merge-gate integrity
A1 — main has no required status checks. The green board is advisory.
$ gh api repos/pulseengine/jess/rules/branches/main --jq '.[] | .type'
deletion
non_fast_forward
pull_request
required_signatures
$ gh api .../rules/branches/main --jq '[.[]|select(.type=="required_status_checks")]|length'
0
$ gh api repos/pulseengine/jess/branches/main/protection
404 Branch not protected
Five substantive checks run on every PR (rivet validate, spar model, scry sound-analysis, Renode RT1176 smoke, mav_bench). None of them can block a merge. The whole session's "confirm CI success on the exact head before merging" was honoured by hand; the repo would have accepted a red merge without complaint.
Bot rule: flag any protected-looking branch whose ruleset lacks required_status_checks, and list the checks that do run so the gap is visible. A repo that runs five checks and requires none is strictly worse than one that runs none — it looks gated.
A2 — CI goes silent, not red, on a conflicting PR
When a PR is CONFLICTING, GitHub cannot build the merge commit and simply stops running pull_request workflows. gh pr checks then prints no checks reported. That string contains neither "pending" nor "fail" — so naive polling reads it as success. My own monitor reported "ALL GREEN" from exactly this.
Bot rule: treat mergeable: CONFLICTING as a first-class blocking state and say so. Never let "no checks" render as anything adjacent to green.
A3 — green against a stale base is a verdict about the wrong thing
PR #192 was genuinely green — against base=8a3e11b7, while main had moved to 1777461e. rivet validate had never run against the combined artifact spine. Merging on that green would have been merging on a question nobody asked. Same again for #195 after #194 landed.
Bot rule: compare baseRefOid to the current base head; if they differ, mark the green stale and require a re-run. This is the single highest-value rule here.
A4 — concurrency: cancel-in-progress + a push burst = no verdict, ever
Five pushes in an hour cancelled each run before a runner picked it up. I then misread the cancelled runs as a starved queue and blamed GitHub Actions capacity in a status report. The runs weren't queued; they were killed by my own next push.
Bot rule: when a run is cancelled by the concurrency group rather than by a human, say so explicitly, and warn on a branch accumulating cancelled runs without a completed one.
B. Gates that pass without checking anything
This is the category that cost the most, and every instance was in a checker, not in the code under test.
B1 — piped exit codes
./script | tail -3 reports tail's status. A failing script prints a clean-looking tail and scores 0. Bit me twice; the project's own standing rule ("always grep Result:, a tail masks FAIL") exists because of an earlier instance.
Bot rule: lint CI steps for cmd | tail, | head, | grep where the pipeline's exit status is the gate. Require PIPESTATUS/pipefail or a bare invocation.
B2 — set -o pipefail + grep -q = SIGPIPE false negative
strings -a "$f" | grep -q PATTERN — grep -q exits on first match, SIGPIPEs the producer, pipeline returns 141. Whether it bites depends on where in the file the match occurs, so it is arbitrary per artifact: a 30 KB component wrongly FAILED while a 12 KB one passed. Nearly produced a false defect report against a supplier.
Bot rule: flag … | grep -q under pipefail.
B3 — a checker that scores "ok" on something it never compared
Two live instances, both mine:
- a toolchain drift checker scored a tool "ok" when it was absent from every source — agreement where nothing was compared. With all tools absent it printed "no drift" and exited 0.
- the same checker scored "ok" for a tool found in exactly one source. One value compared against nothing is not agreement.
Bot rule: any gate whose comparison set can be empty must fail loudly rather than pass. "Nothing to check" is never a pass.
B4 — a negative control that tests the wrong property
The worst one, and it reached a merged safety artifact. A differential's control perturbed an input, re-ran, and reported the changed output as proof "the fold tracks its input". The re-run used the same instance, and the module carries integrator state — so the output changed regardless of input. Ticking the identical input twice produced the same "proof". Meanwhile the documented perturbation moved the output by zero on a fresh instance, because the operating point was saturated.
A build that ignored its input entirely would have passed that control.
Bot rule (weaker, but worth it): flag negative controls that mutate state between the baseline and the perturbed run — a control should differ from its baseline in exactly one variable. Full detection needs a human, but "control shares mutable state with baseline" is greppable in practice.
B5 — a metric whose failing case cannot occur
A parser reported "zero CRC failures" while its frame extractor silently skipped CRC-invalid frames — the count was structurally guaranteed to be zero. Withdrawn. Related: the withdrawal never propagated to the finding's title, which still advertised the retracted number for days.
Bot rule: where a claim is withdrawn in a body, flag any title/summary still asserting it.
B6 — a checker that is red where it lives, and gates nothing
tools/varve/check-drift.sh exits 1 on main today. Meanwhile none of this repo's four oracles is referenced by CI:
$ grep -cE 'appcompose|check-drift|m4-matrix|timer-probe' .github/workflows/ci.yml
0
Bot rule: inventory executable tools/**/*.sh that no workflow invokes, and report them as unwired. An oracle nobody runs is documentation.
C. Version and dependency hygiene
C1 — three sources of truth for one tool version, never compared
TOOL PATH VARVE-PIN CI-YML
rivet 0.32.0 0.34.0 0.25.0 <- three different versions
spar - 0.40.0 0.24.0
meld 0.41.3 0.52.0 0.41.3
This caused a wrong upstream bug report. I filed meld#390 having run 0.41.3 while 0.52.0 was latest — eleven minor versions — and had to correct it publicly. Note CI pinned 0.41.3 too: CI would not have caught it either.
Bot rule: compare every pinned tool version against the upstream latest release on a schedule, and open a PR (or issue) on drift. Cheap, and it directly prevents the failure above.
C2 — an issue that cites a version should have that version checked
The most embarrassing failure of the session was a defect report whose version citation was wrong. That is mechanically checkable.
Bot rule: when an issue body names <tool> <semver> for a repo in the org, compare against latest and comment if a newer release exists. Wrong-version reports cost a maintainer's attention.
C3 — unpinned interpreter-level dependencies
A verification script preflighted four CLI tools and four supplier artifacts, then died at the last step with a bare ModuleNotFoundError for a Python module — after the expensive work had succeeded, so it read as an oracle mismatch rather than a missing dependency. No requirements.txt existed.
Bot rule: flag scripts importing modules with no corresponding pin file, and require dependency preflight to happen before expensive steps.
C4 — shell portability that silently misreports
mapfile doesn't exist in macOS's bash 3.2. Its absence printed <none> for a symbol list that had three entries — a silent wrong answer, not an error. Caught only because a later step contradicted it.
Bot rule: shellcheck in CI, with bash-3.2 targeting for any script that runs on developer macs.
D. Claim hygiene in prose
These matter because artifacts here are safety-case evidence, and a git log is read alone.
D1 — emulation described as hardware
Real examples found in this repo:
- a commit subject: "the fused image EXECUTES on the RT1176 M7" — no "(Renode)" anywhere in subject or body, while its own predecessor commit said "(Renode)" correctly. Read alone in a log, it asserts silicon.
hardware/silicon/README.md: "this is the actual chip" — over a run that was entirely Renode, on a board marked "(ordered)".- an architecture diagram labelling a Renode node "HIL emulation". There is no hardware in that loop.
Bot rule: lint for on hardware|on silicon|actual chip|HIL|on-target|executes on in commit messages and docs, and require an adjacent qualifier (Renode, emulated, wasmtime, qemu) or an explicit NOT-CLAIMED: line. Cheap, and it maps exactly to the mistakes actually made.
D2 — absolutes that a committed artifact contradicts
A finding asserted the target board "has been ABSENT on every loop tick … has never found it", while the repo contains 77,034 bytes of live telemetry captured from that board, independently re-parsed at 1,021 CRC-valid frames. I then repeated the false absolute verbally. The scoped claim ("no falcon code has ever executed on it") was true and sufficient.
Bot rule: hard to automate in general — but "never/always/every" in an evidence artifact is a cheap grep worth surfacing for human review.
E. Repo mechanics
E1 — stranded branches
Pushed twice with no PR opened. Bot rule: flag branches ahead of main with no open PR after N hours.
E2 — append-only YAML conflicts every time
artifacts/findings.yaml grows by appending near the same anchor, so two concurrent PRs conflict structurally, not semantically. Resolving by taking either side silently drops the other's entries — I nearly lost five corrections that way, and only caught it by diffing the artifact-ID sets before and after.
Bot rule: for known append-only artifact files, detect conflicts that are purely positional and verify no artifact ID is lost across the resolution. An ID-set diff is a two-line check and would have caught it immediately.
Priority, if only some of this gets built
- A3 stale-base green — silently merges unverified combinations
- A1 unenforced required checks — makes every other gate advisory
- B3 / B1 / B2 gates that pass without checking — the most common failure by count
- C1 / C2 version drift — the one that produced a wrong upstream report
- D1 emulation-as-hardware lint — cheapest rule with the highest consequence in a safety context
- E2 ID-loss on artifact merges — silent data loss
Happy to prototype any of these as a plain workflow first; several (A3, C1, D1, E2) are a few lines of gh + python and need no bot infrastructure at all.
Filed from the jess autonomous loop. Every incident above is from this repo's history over roughly one day; commit and issue references available on request.
What this is
Over one long autonomous session I hit the same classes of failure repeatedly. None were exotic; all were mechanically detectable; several produced wrong claims that reached upstream repos before being caught. This is the list, written as rules a bot could enforce, each grounded in a specific incident from this repo rather than in principle.
The organising observation: almost every one of these is a green that wasn't earned — a gate that passed without comparing anything, a check that never ran, a verdict about the wrong commit. A hygiene bot's job is to make unearned greens impossible to mistake for earned ones.
A. Merge-gate integrity
A1 —
mainhas no required status checks. The green board is advisory.Five substantive checks run on every PR (rivet validate, spar model, scry sound-analysis, Renode RT1176 smoke, mav_bench). None of them can block a merge. The whole session's "confirm CI success on the exact head before merging" was honoured by hand; the repo would have accepted a red merge without complaint.
Bot rule: flag any protected-looking branch whose ruleset lacks
required_status_checks, and list the checks that do run so the gap is visible. A repo that runs five checks and requires none is strictly worse than one that runs none — it looks gated.A2 — CI goes silent, not red, on a conflicting PR
When a PR is
CONFLICTING, GitHub cannot build the merge commit and simply stops runningpull_requestworkflows.gh pr checksthen printsno checks reported. That string contains neither "pending" nor "fail" — so naive polling reads it as success. My own monitor reported "ALL GREEN" from exactly this.Bot rule: treat
mergeable: CONFLICTINGas a first-class blocking state and say so. Never let "no checks" render as anything adjacent to green.A3 — green against a stale base is a verdict about the wrong thing
PR #192 was genuinely green — against
base=8a3e11b7, whilemainhad moved to1777461e.rivet validatehad never run against the combined artifact spine. Merging on that green would have been merging on a question nobody asked. Same again for #195 after #194 landed.Bot rule: compare
baseRefOidto the current base head; if they differ, mark the green stale and require a re-run. This is the single highest-value rule here.A4 —
concurrency: cancel-in-progress+ a push burst = no verdict, everFive pushes in an hour cancelled each run before a runner picked it up. I then misread the cancelled runs as a starved queue and blamed GitHub Actions capacity in a status report. The runs weren't queued; they were killed by my own next push.
Bot rule: when a run is cancelled by the concurrency group rather than by a human, say so explicitly, and warn on a branch accumulating cancelled runs without a completed one.
B. Gates that pass without checking anything
This is the category that cost the most, and every instance was in a checker, not in the code under test.
B1 — piped exit codes
./script | tail -3reports tail's status. A failing script prints a clean-looking tail and scores 0. Bit me twice; the project's own standing rule ("always grepResult:, a tail masks FAIL") exists because of an earlier instance.Bot rule: lint CI steps for
cmd | tail,| head,| grepwhere the pipeline's exit status is the gate. RequirePIPESTATUS/pipefailor a bare invocation.B2 —
set -o pipefail+grep -q= SIGPIPE false negativestrings -a "$f" | grep -q PATTERN—grep -qexits on first match, SIGPIPEs the producer, pipeline returns 141. Whether it bites depends on where in the file the match occurs, so it is arbitrary per artifact: a 30 KB component wrongly FAILED while a 12 KB one passed. Nearly produced a false defect report against a supplier.Bot rule: flag
… | grep -qunderpipefail.B3 — a checker that scores "ok" on something it never compared
Two live instances, both mine:
Bot rule: any gate whose comparison set can be empty must fail loudly rather than pass. "Nothing to check" is never a pass.
B4 — a negative control that tests the wrong property
The worst one, and it reached a merged safety artifact. A differential's control perturbed an input, re-ran, and reported the changed output as proof "the fold tracks its input". The re-run used the same instance, and the module carries integrator state — so the output changed regardless of input. Ticking the identical input twice produced the same "proof". Meanwhile the documented perturbation moved the output by zero on a fresh instance, because the operating point was saturated.
A build that ignored its input entirely would have passed that control.
Bot rule (weaker, but worth it): flag negative controls that mutate state between the baseline and the perturbed run — a control should differ from its baseline in exactly one variable. Full detection needs a human, but "control shares mutable state with baseline" is greppable in practice.
B5 — a metric whose failing case cannot occur
A parser reported "zero CRC failures" while its frame extractor silently skipped CRC-invalid frames — the count was structurally guaranteed to be zero. Withdrawn. Related: the withdrawal never propagated to the finding's title, which still advertised the retracted number for days.
Bot rule: where a claim is withdrawn in a body, flag any title/summary still asserting it.
B6 — a checker that is red where it lives, and gates nothing
tools/varve/check-drift.shexits 1 onmaintoday. Meanwhile none of this repo's four oracles is referenced by CI:Bot rule: inventory executable
tools/**/*.shthat no workflow invokes, and report them as unwired. An oracle nobody runs is documentation.C. Version and dependency hygiene
C1 — three sources of truth for one tool version, never compared
This caused a wrong upstream bug report. I filed meld#390 having run 0.41.3 while 0.52.0 was latest — eleven minor versions — and had to correct it publicly. Note CI pinned 0.41.3 too: CI would not have caught it either.
Bot rule: compare every pinned tool version against the upstream latest release on a schedule, and open a PR (or issue) on drift. Cheap, and it directly prevents the failure above.
C2 — an issue that cites a version should have that version checked
The most embarrassing failure of the session was a defect report whose version citation was wrong. That is mechanically checkable.
Bot rule: when an issue body names
<tool> <semver>for a repo in the org, compare against latest and comment if a newer release exists. Wrong-version reports cost a maintainer's attention.C3 — unpinned interpreter-level dependencies
A verification script preflighted four CLI tools and four supplier artifacts, then died at the last step with a bare
ModuleNotFoundErrorfor a Python module — after the expensive work had succeeded, so it read as an oracle mismatch rather than a missing dependency. Norequirements.txtexisted.Bot rule: flag scripts importing modules with no corresponding pin file, and require dependency preflight to happen before expensive steps.
C4 — shell portability that silently misreports
mapfiledoesn't exist in macOS's bash 3.2. Its absence printed<none>for a symbol list that had three entries — a silent wrong answer, not an error. Caught only because a later step contradicted it.Bot rule: shellcheck in CI, with bash-3.2 targeting for any script that runs on developer macs.
D. Claim hygiene in prose
These matter because artifacts here are safety-case evidence, and a git log is read alone.
D1 — emulation described as hardware
Real examples found in this repo:
hardware/silicon/README.md: "this is the actual chip" — over a run that was entirely Renode, on a board marked "(ordered)".Bot rule: lint for
on hardware|on silicon|actual chip|HIL|on-target|executes onin commit messages and docs, and require an adjacent qualifier (Renode,emulated,wasmtime,qemu) or an explicitNOT-CLAIMED:line. Cheap, and it maps exactly to the mistakes actually made.D2 — absolutes that a committed artifact contradicts
A finding asserted the target board "has been ABSENT on every loop tick … has never found it", while the repo contains 77,034 bytes of live telemetry captured from that board, independently re-parsed at 1,021 CRC-valid frames. I then repeated the false absolute verbally. The scoped claim ("no falcon code has ever executed on it") was true and sufficient.
Bot rule: hard to automate in general — but "never/always/every" in an evidence artifact is a cheap grep worth surfacing for human review.
E. Repo mechanics
E1 — stranded branches
Pushed twice with no PR opened. Bot rule: flag branches ahead of
mainwith no open PR after N hours.E2 — append-only YAML conflicts every time
artifacts/findings.yamlgrows by appending near the same anchor, so two concurrent PRs conflict structurally, not semantically. Resolving by taking either side silently drops the other's entries — I nearly lost five corrections that way, and only caught it by diffing the artifact-ID sets before and after.Bot rule: for known append-only artifact files, detect conflicts that are purely positional and verify no artifact ID is lost across the resolution. An ID-set diff is a two-line check and would have caught it immediately.
Priority, if only some of this gets built
Happy to prototype any of these as a plain workflow first; several (A3, C1, D1, E2) are a few lines of
gh+pythonand need no bot infrastructure at all.Filed from the jess autonomous loop. Every incident above is from this repo's history over roughly one day; commit and issue references available on request.