v2.9.0 feat(eng): gate deploys behind a ceremony, add /eng:deploy - #64
Conversation
The command-position matcher shared by the ship, merge, and qa-plan gates required a non-empty assignment value: ([A-Za-z_][A-Za-z0-9_]*=[^[:space:]]+[[:space:]]+)* so `FOO= gh pr merge` matched neither branch. Not the env-prefixed one (the value is empty), and not the bare one (no shell separator sits before `gh`). The command sailed straight through every gate. Found while building the deploy gate, which copied the same prefix; grepping the shape turned up four more sites. The quantifier is now `*` in all of them, with a regression test on pr-merge-gate, the highest-stakes site. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every PreToolUse guard on this machine matched `gh pr merge` or `gh pr create`.
None matched a deploy. So the ~/dev rule "/land-and-deploy is the only way a PR
reaches main AND production" was one sentence covering two things, and only the
merge half was enforced.
On 2026-07-24 a hand-rolled ssh deploy to the Mac mini skipped the
upgrade-marker stamp, tripped nanoclaw's version tripwire, and the host
crash-looped behind a 900s circuit breaker for 16h46m across 72 failed starts.
Nothing was bypassed, because nothing was in the path. The agent was following
the repo's own CLAUDE.md, which documented the bare sequence.
deploy-gate.sh is sentinel and gate in one script, wired to three events like
land-deploy-sentinel.sh. It arms on /land-and-deploy or /eng:deploy and blocks
deploy-shaped Bash otherwise, in two tiers:
tier 1 the entrypoint declared in deploy.json (plus deploy-*.sh)
tier 2 ssh to a marker-listed host carrying git pull / build / kickstart
/ systemctl restart, which is the shape the incident used
Read-only forms stay allowed unconditionally: check, --status, --dry-run, and
ssh carrying only read verbs. A retry follows a failure, so diagnosing that
failure must never require a ceremony.
It arms its OWN kind ("deploy"), never land-deploy-sentinel's "land", so a
deploy-only ceremony can never mint merge clearance. Tested directly.
/eng:deploy exists because gstack has no /deploy and /land-and-deploy hard-stops
once a PR is merged. Retry, recovery, and --rebuild-base reruns are all real and
all PR-less; without a second ceremony the gate's break-glass override would
have become the routine deploy path, which is the ungated state with extra
typing. It refuses when the branch has an open PR, so it cannot become a
merge-gate bypass, and it ends in `devops check`.
Opt-in per repo via .deploy-gate.json; fails open everywhere else.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
eng is a feature bump (deploy gate + /eng:deploy). qa is a patch: its qa-plan-pr-gate carried the same empty-env-prefix bypass. Both bumps are load-bearing, not bookkeeping. bin/install COPIES each plugin into ~/.claude/plugins/cache/<marketplace>/<name>/<version>/, and `claude plugin update` no-ops while the version is unchanged, so without the bump the installed copy stays on the old version and nothing takes effect. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
📝 WalkthroughWalkthroughAdds deploy-gate enforcement, the ChangesDeploy ceremony enforcement
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant Operator
participant HookWiring
participant DeployGate
participant Repository
participant DeploySkill
Operator->>HookWiring: Invoke Skill or Bash tool
HookWiring->>DeployGate: Run deploy gate hook
DeployGate->>Repository: Inspect deploy markers
DeployGate-->>Operator: Allow or block deploy command
Operator->>DeploySkill: Start sanctioned deployment
DeploySkill->>Repository: Validate deploy contract and main state
DeploySkill->>Repository: Execute declared deploy command
DeploySkill-->>Operator: Report deployment and verification
Possibly related PRs
Suggested labels: Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Comment |
Both found by the pre-merge review, both verified against the real mutwo checkout before and after. 1. The read-only escape tested for the bare word `check` ANYWHERE in the command, so `scripts/deploy.sh && devops check` read as read-only and deployed straight past the gate. That is a wholly natural thing to type. The escape is now anchored to the matched command's own argument list and may not cross a shell separator or a comment. 2. The trailing boundary accepted only whitespace or end-of-string, so `scripts/deploy.sh;`, `scripts/deploy.sh&&...`, and `(scripts/deploy.sh)` were not recognized as deploy commands at all. It now accepts a shell separator too. Seven regressions cover the bypasses, plus four asserting that degraded config still gates (a malformed marker, a missing deploy.json, a custom entrypoint) and that the allow path writes nothing to stdout. That last one matters because this hook fires on every Bash call in every ~/dev repo, so a single stray byte would be read as a hook decision fleet-wide. 38 tests, all green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The assertion piped the hook's stdout through `od -c`. GNU od prints an offset line (`0000000`) for empty input while BSD od prints nothing, so the test passed on macOS and failed on the Linux runner. It was the test that was wrong, not the gate. Now asserts on file size instead, which is byte-exact and portable. That also fixes a second latent weakness: bats strips trailing newlines from $output, so a lone stray "\n" would have read as empty either way. Verified with a negative control (an injected `echo STRAY` is detected), so the test can actually fail. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@eng/hooks/scripts/deploy-gate.sh`:
- Around line 205-210: Update the DEPLOY_GATE_OVERRIDE check in the break-glass
block to recognize the assignment only at command position, using the same
^|[;&|(] boundary semantics as T1_RE and the ssh-prefix regex. Preserve support
for quoted and unquoted non-empty values while preventing matches embedded in
commit messages or other argument values.
- Around line 186-201: Update MUTATING_RE to require command boundaries after
each mutating verb, preventing matches such as “builder” or “pull-request”. In
the hand-rolled SSH branch, parse and validate the command payload associated
with the matched ssh invocation, then apply the host and mutating-command checks
only to that payload rather than the entire CMD, so locally chained commands
cannot trigger the gate.
In `@eng/hooks/tests/pr-merge-gate.bats`:
- Around line 109-119: Extend the regression coverage for empty environment
assignments beyond pr-merge-gate.sh by adding equivalent blocked-command cases
for ship-pr-gate.sh, ship-watch-nudge-lib.sh, and qa-plan-pr-gate.sh. Reuse the
existing test setup and assertions, ensuring each matcher consumer rejects
commands such as “FOO= gh …” and remains protected against copy drift.
In `@eng/skills/deploy/SKILL.md`:
- Around line 84-89: Update the deployment status check in the `wtr status`
command flow to stop suppressing failures: capture the command’s error, report
the inability to read live host status, and terminate before continuing to the
`git log` or deployment steps. Preserve the existing absolute `wtr` path and
status query arguments.
- Around line 104-109: Update the deployment command block to capture the
skill’s supported flags in an argv array, construct the declared command from
deploy.json as an executable plus its arguments rather than quoting the entire
string, and invoke it with the forwarded-flags array so flags execute correctly.
Preserve the default scripts/deploy.sh fallback and the existing --rebuild-base,
--rederive-all, --recycle, and --force options.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: b10e366c-d480-4dfa-b45a-9432b10b1422
📒 Files selected for processing (13)
eng/.claude-plugin/plugin.jsoneng/hooks/hooks.jsoneng/hooks/scripts/deploy-gate.sheng/hooks/scripts/pr-merge-gate.sheng/hooks/scripts/ship-pr-gate.sheng/hooks/scripts/ship-watch-nudge-lib.sheng/hooks/tests/deploy-gate.batseng/hooks/tests/hooks-json.batseng/hooks/tests/pr-merge-gate.batseng/skills/deploy/SKILL.mdqa/.claude-plugin/plugin.jsonqa/hooks/scripts/qa-plan-pr-gate.shspec/plans/deploy-gate-and-eng-deploy-ceremony.md
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@eng/hooks/scripts/deploy-gate.sh`:
- Around line 195-201: Update the deploy-command detection around the CMD check
in eng/hooks/scripts/deploy-gate.sh: split the command chain on shell separators
and require every deploy invocation to include a read-only flag, so a later
--force invocation cannot be masked by an earlier --dry-run. Add a regression
covering scripts/deploy.sh --dry-run && scripts/deploy.sh --force in
eng/hooks/tests/deploy-gate.bats, with the expected gate behavior.
In `@eng/hooks/tests/deploy-gate.bats`:
- Around line 124-145: Add a regression test in the deploy-gate test suite
covering two real entrypoint invocations chained with &&, where the first uses
--dry-run and the second uses --force. Use the existing opt_in, deploy_json,
run_gate, and assert_block helpers, and verify the gate blocks the chained
command rather than treating it as read-only.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: d033d523-465e-4044-a1bc-7c4444d49409
📒 Files selected for processing (2)
eng/hooks/scripts/deploy-gate.sheng/hooks/tests/deploy-gate.bats
Every one was verified failing against the real mutwo checkout before the fix and passing after. Two of the three arrived marked "✅ Addressed in commits 0b92118 to 2cddecc" and were still fully live: CodeRabbit resolved them on commit activity, not on a fix. The behavior is what these tests assert, not the label. 1. CRITICAL, full gate defeat. The break-glass override was a bare substring search, so the literal text DEPLOY_GATE_OVERRIDE=x anywhere in the command authorized a bypass with no variable ever being set: ssh mutwos-mac-mini "git commit -m 'DEPLOY_GATE_OVERRIDE=oops' \ && git pull && pnpm run build" -> ALLOW That is the exact hand-rolled shape the gate exists to block. Now anchored at command position like every other matcher here. 2. The entrypoint matcher judged the whole command at once, so a read-only flag on the FIRST invocation excused a second, real one: scripts/deploy.sh --dry-run && scripts/deploy.sh --force -> ALLOW Matching is now per segment, each invocation judged on its own arguments. 3. Tier 2 over-matched. The mutating verbs had no trailing boundary, and the ssh, host, and verb checks were three independent greps over the whole command: ssh host 'pnpm run builder' -> BLOCK (not a deploy) ssh host 'tail log' ; git pull -> BLOCK (the pull is local) Verbs are bounded, and the host and verb are now read from the ssh invocation's own target and quoted payload. The segment splitter is quote-aware on purpose. A plain sed split also cuts at separators INSIDE a quoted argument, which shreds the payload tier 2 must read: the && in `ssh host 'cd ~/nanoclaw && git pull && pnpm run build'` belongs to the REMOTE command. Splitting there dropped the mutating verb and let the incident command through, which the first pass at this fix did before the tests caught it. Also from the review: - /eng:deploy no longer suppresses a failed `wtr status`. Deploying without knowing what the host runs defeats the point of the step, so a failed query now stops. - /eng:deploy builds an argv array instead of interpolating a `<forwarded flags>` placeholder, which is not executable shell, and validates forwarded flags against the four it declares. - The empty-env-prefix regression now covers ship-pr-gate and qa-plan-pr-gate too, so copy drift cannot reopen it in one consumer only. 45 deploy-gate tests, all siblings green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Addressed all seven CodeRabbit threads in Two arrived marked "✅ Addressed in commits 0b92118 to 2cddecc" and were still fully live. Those commits fixed different bugs I had found independently; CodeRabbit resolved these on commit activity, not on a fix. Worth flagging because the resolved flag read as done when it was not.
The override one is the serious one: it defeated the gate entirely via text embeddable in a commit message, on exactly the hand-rolled shape this exists to stop. Fixes: the override is anchored at command position like every other matcher; entrypoint matching is per segment so each invocation is judged on its own arguments; mutating verbs carry trailing boundaries; and the host and verb are read from the ssh invocation's own target token and quoted payload rather than three independent greps over the whole command. One thing your suggested fix would have missed. Scoping via Also done: 45 deploy-gate tests (was 38), all siblings green. |
The prose said a failed host-status query is a STOP, but the code block only printed the error and fell through to the deploy. A SKILL.md snippet is what gets copied, so the two must not disagree. It now exits non-zero, sends the diagnosis to stderr, and captures rc before anything else can clobber $?. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Merged as Row 1: the versioned artifact exists The version bump was load-bearing exactly as the layer-walk predicted: Row 2: incident payloads driven through the installed copy, against the real Row 2 covers all four bypasses found during review (two by me, two by CodeRabbit), verified through the production artifact rather than the repo copy. One dimension deliberately not claimed. Whether the harness fires the hook needs a session started after this install, since plugin hooks load at session start and I cannot restart the session I am running in. Row 2 proves the installed file decides correctly; the registration that invokes it is the same Row 3 (a real QA_STATUS: prod_verified |
|
Production row 3 verified: The ceremony's guards all behaved as designed on a live repo: Step 3 is worth calling out: those 5 untracked files are exactly the case that made me change the guard from "any dirty tree" to "tracked modifications only". As originally written, The deploy, with the line that matters: That stamp is the single step the 2026-07-24 hand-roll skipped, and skipping it is what crash-looped the host for 16h46m. Post-deploy liveness, checked the way the incident taught: Marker and code agree, so the tripwire is satisfied rather than merely quiet. QA_STATUS: prod_verified |
1 participant
Summary
Deploys are now gated behind a ceremony, the way merges already are, and a new
/eng:deploycovers the PR-less deploys/land-and-deploystructurally cannot.The gap. Every PreToolUse guard on this machine matched
gh pr mergeorgh pr create. None matched a deploy. A grep across*/hooks/scripts/fordeploy.sh|kickstart|deploy-mini|launchctl|sshreturned zero files. So the~/devrule "/land-and-deploy is the only way a PR reaches main AND production" was one sentence covering two things, and only the merge half was backed by anything.What it cost. On 2026-07-24 a hand-rolled ssh deploy to the Mac mini skipped the upgrade-marker stamp, tripped nanoclaw's version tripwire, and the host crash-looped behind a 900s circuit breaker for 16h46m across 72 failed starts. Nothing was bypassed, because nothing was in the path. The agent was following the repo's own CLAUDE.md, which documented the bare sequence.
What is here
eng/hooks/scripts/deploy-gate.shis sentinel and gate in one script, wired to three events exactly likeland-deploy-sentinel.sh. It arms on/land-and-deployor/eng:deployand blocks deploy-shaped Bash otherwise, in two tiers:deploy.json, plus thedeploy-*.shfamilysshto a marker-listed host carryinggit pull/ `pnpmRead-only forms stay allowed unconditionally:
check,--status,--dry-run, and ssh carrying only read verbs likelaunchctl listortail. A retry follows a failure, so diagnosing that failure must never require a ceremony.It arms its own kind (
"deploy"), neverland-deploy-sentinel's"land", so a deploy-only ceremony can never mint merge clearance. There is a direct test for that.eng/skills/deploy/SKILL.md(/eng:deploy) exists because gstack has no/deployand/land-and-deployhard-stops once a PR is merged ("nothing to deploy, run /canary"). Retry after a failed deploy, recovery after a host wedge, and--rebuild-basereruns are all real and all PR-less. Without a second ceremony the gate's break-glass override would have become the routine deploy path, which is the ungated state with extra typing. It refuses when the branch has an open PR, so it cannot become a merge-gate bypass, and it ends indevops check.Opt-in per repo via
.deploy-gate.json; fails open everywhere else. Currently armed for~/dev/tooling/mutwoonly.Also in here: a bypass that affected the existing gates
The command-position matcher shared by the ship, merge, and qa-plan gates required a non-empty assignment value, so
FOO= gh pr mergematched neither the env-prefixed branch (empty value) nor the bare one (no shell separator beforegh), and sailed through every gate. Found while building the deploy gate, which had copied the same prefix; grepping the shape turned up four more sites. Fixed in all five, with a regression test onpr-merge-gate.This is why the
qaplugin also gets a patch bump.Version bumps are load-bearing
bin/installCOPIES each plugin into~/.claude/plugins/cache/<marketplace>/<name>/<version>/, andclaude plugin updateno-ops while the version is unchanged. Without the bump the installed copy stays on the old version and nothing takes effect. eng 2.8.1 to 2.9.0, qa 3.6.0 to 3.6.1.Test plan
52 tests, zero failures:
Plus the gate driven against the real
~/dev/tooling/mutwocheckout:Full suites run in CI (this machine cannot run them).
Known limits, stated plainly
~/devkeeps the window alive (1800s idle budget). Sliding is required because only Skill and prompt events arm and a real ceremony outlives a fixed window. This is an accident-guard, matchingpr-merge-gate.sh's own framing, not a sandbox.scripts/checks/nanoclaw.shalready detects the exact failure but runs only post-deploy. Wiring it into the mini's 10-minute health probe is the complementary follow-up, tracked separately.Follow-ups not in this PR
~/dev/gated-repos.jsonregistry entrya2fb54c), prod-verifiedCLAUDE.mdpointer at/eng:deployfor the retry case (separate repo; the gate's own block message already names it)mini-health.shQA
📄 Plan view: https://claude.ai/code/artifact/85dad762-f13f-4af6-be7f-41d1083d1d1e
QA driver: Claude (the building agent, this session) - both phases are deterministic hook-payload, bats, and deploy-kit work this session can drive and record now.
Standard (all green): unit tests · lint/types · CI ·
/eng:cr🖥️ Development
Before this can merge, we replay the exact commands from the outage at the gate as fake tool calls on a throwaway repo, and check it blocks those while still waving through the harmless ones.
bats eng/hooks/tests/deploy-gate.batsbash -nplus a no-marker payload through the real hookset -u; an unmarked repo stays silent and exits 0pr-merge-gate.shagh pr mergepayload with only a deploy arm presentbats eng/hooks/tests/hooks-json.bats/eng:deploy's guards on a scratch repo: dirty tree, non-main branch, open unmerged PR🚀 Production
Once it is merged and the plugin is reinstalled, we point the exact commands from the outage at the installed copy Claude Code actually runs, then use the new ceremony to ship the deploy the mini has been waiting on.
bin/gstack-extensions-upgrade, then list the plugin cache.../eng/2.9.0/hooks/scripts/deploy-gate.shpresent and executable~/dev/tooling/mutwo/eng:deployin~/dev/tooling/mutwodevops checkpasses,launchctl list | grep com.nanoclawshows a live pid with lastexit=0Not self-verifiable in this session: whether the harness fires the hook needs a session started after the install, since plugin hooks load at session start. Production row 2 proves the installed file decides correctly; the registration that invokes it is the same
hooks.jsonwiring the four existing eng gates run on, asserted byhooks-json.bats, and it is confirmed the first time any later session hits a deploy command in mutwo.Production artifacts:
~/.claude/plugins/cache/gstack-extensions/eng/2.9.0/hooks/scripts/deploy-gate.shon the laptop. This is the file Claude Code executes as a PreToolUse hook.~/.claude/plugins/cache/gstack-extensions/eng/2.9.0/skills/deploy/SKILL.md, the installed/eng:deploy.bin/installcopies, then the version-keyed cache, then loaded at session start. Production executes the cache, never the repo.claude plugin updateno-ops on an unchanged version, so theplugin.jsonbump AND a session restart are both required.mutwos-mac-mini.Definition of Done:
/eng:cr) + CodeRabbit addressed~/dev/gated-repos.jsonupdated so the marker survives a fresh cloneQA posture: Pre-merge, state
QA_STATUS: dev_verifiedplusEVIDENCE:once every Dev QA tick and every Definition-of-Done bullet is checked. Post-deploy, stateQA_STATUS: prod_verifiedplusEVIDENCE:once the Production rows are verified live.Summary by CodeRabbit
New Features
/eng:deploydeployment ceremony and an opt-in deploy gate for repositories using deployment gating markers.eng/skills/deployskill documentation for deploying the current main branch safely.Bug Fixes
gh pr create/gh pr mergecommands when preceded by empty environment-variable assignments.Tests