feat(hardware): diff the twin against a real desk board - #41
Merged
Conversation
Same firmware artifact, two independent targets: the LabWired digital twin and a physical desk board. `hardware diff` compares the two authenticated evidence bundles and publishes the disagreement as a first-class result. Each side is summarized from its own bundle only, so there is no code path that upgrades a hardware green to a twin green or the reverse. A twin bundle that records `hardware_observed` is rejected as invalid. Both sides must bind to the same artifact digest. A missing probe or a missing desk bundle yields `desk-unavailable` (exit 4), never a silent pass. Verdicts bind to exit codes: agree 0, invalid 2, disagree 3, desk-unavailable 4, twin-unavailable 5. A failed diff invocation returns 2 so it can never borrow the disagree code. Tests run entirely from recorded evidence bundles under fixtures/twin-desk-diff, so the differential is provable with no board attached.
check-public-package.sh forbids the literal evidence statuses outside docs/VERIFY.md and config/AGENTS.md, and the diff documentation had put model_verified and hardware_observed straight into USAGE.md. USAGE.md now describes the two sides as disjoint model and physical-evidence grades and points at VERIFY.md; VERIFY.md gains the hardware diff section with the exact grade names, the refusal rules, and the verdict-to-exit-code binding.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Same firmware artifact, two independent targets — the LabWired digital twin and a physical desk board — with the disagreement published as a first-class result. We are the only party holding both sides: competitors have silicon but no model, pure simulators have no silicon.
What it does
hardware diffcompares two authenticated evidence bundles and emits a structured verdict. Verdicts bind to exit codes: agree 0, invalid 2, disagree 3, desk-unavailable 4, twin-unavailable 5. A failed invocation returns 2 explicitly, so it can never borrow 3.Semantics enforced structurally, not by check
Each side is summarized from its own bundle only — no expression in the code reads the other side's level, so upgrading a hardware green to a twin green (or the reverse) is impossible rather than merely guarded. A twin bundle recording
hardware_observedisinvalid, not accepted. A desk record carrying a model level is inconclusive desk evidence, never a desk pass.compiledon the twin side is inconclusive, not behavior evidence. Both sides must bind to the same artifact digest; a mismatch isinvalid, never smoothed into agreement.agreerequires at least one behavior both sides actually decided.Files
lib/hardware/differential.mjs— the core (324 lines)scripts/hardware-runner.mjs— newdiffverbtests/hardware-differential.test.mjs+tests/helpers/make-twin-desk-fixtures.mjsfixtures/twin-desk-diff/— 6 recorded bundles + receiptsdocs/USAGE.md,skills/desk-hw/SKILL.md, gate lines intests/hardware-public-docs.shFixtures are genuine bundles produced by
createEvidenceBundle, not hand-written JSON, so they pass through the same authentication path a live run does.Verification
hardware-differential18 pass / 0 fail / 0 skipped;hardware-*293 pass;test:node18-min,hardware-cli.sh,hardware-public-docs.sh,hardware-release-contract.sh,skills-verify-all.shall exit 0. Live shell path:disagree→ exit 3, no desk args → exit 4, no--artifact→ exit 2.Negative controls, each with a non-empty diff asserted first, restored path-scoped: making a real disagreement report agreement → 2 named tests red; making a missing board a pass → 5 named tests red.
What this does NOT prove
Blunt version: no live desk bundle has ever been fed to this. Every desk fixture was minted through
createEvidenceBundle, and thedisagree/deskcase is a hand-authored twin-green/desk-red. The comparison is sound; neither side's capture is proven end to end against silicon. This is a working instrument with no measurement yet.Also open: twin and desk profiles are separate files, so nothing forces the two lanes to name the same behavior IDs — a real run can produce zero paired behaviors. That surfaces as
invalid / no_comparable_behaviorrather than a false pass, but one profile emitting both lanes would be better. Artifact equality is by digest only; that the twin loaded the image as faithfully as the flasher wrote it is not checked.Not reused
bench/twin2silicon-deepseek(merged as #40) compares AI runtimes against one HIL oracle in Python, with the board as referee. Different axis. Reusingrun_hil.pywould have forked the evidence model, so this builds onlib/hardware/{runner,evidence,locks}.mjsinstead.