From fd14f4611c97691148c1daa3c7b6194391fd882c Mon Sep 17 00:00:00 2001 From: srt0422 Date: Thu, 21 May 2026 17:08:09 -0700 Subject: [PATCH 01/15] DEVOP-573: author tabletop exercise scenario doc Documents the inaugural Shai-Hulud-class tabletop exercise: an injected "eliza-allora-plugin was published with a postinstall payload yesterday at 4pm" scenario that walks the team end-to-end through the SECURITY-RUNBOOK (DEVOP-571). The doc is operational, not a writeup. It contains: * The injected scenario, including the specific exfil mechanics, the IOC discovery timeline, and the T+0 trigger. * Pre-assigned roles (incident lead, communicator, executor, BE rep, FE rep, founder observer-only) with explicit don't-skip-a-role rule. * Six phases keyed to runbook sections, each with a target elapsed time and explicit success/failure modes the facilitator watches for. * The 30-minute time-to-clean-republish target broken into 4 phases (T+5 / T+10 / T+20 / T+30) so participants can self-check progress mid-exercise. * A debrief script (6 questions, in order) that produces ticket inputs verbatim from the team's own language. * Output checklist for the facilitator (Linear tickets, runbook PR, lessons-learned section update, next-year calendar invite). * Notes-from-runbook-author section identifying the three seams in the runbook that the exercise should specifically stress. The exercise itself is a team activity and is NOT considered complete until the run + debrief actually happen. DEVOP-573 stays In Review until the facilitator schedules and runs the live session. Blocks-by: DEVOP-571 (runbook). PR #3 in this repo authors the runbook; this PR cross-references it. Refs: https://linear.app/alloralabs/issue/DEVOP-573 Co-Authored-By: Claude Opus 4.7 (1M context) --- tabletop/2026-Q1-shai-hulud-eliza.md | 272 +++++++++++++++++++++++++++ 1 file changed, 272 insertions(+) create mode 100644 tabletop/2026-Q1-shai-hulud-eliza.md diff --git a/tabletop/2026-Q1-shai-hulud-eliza.md b/tabletop/2026-Q1-shai-hulud-eliza.md new file mode 100644 index 0000000..14086a7 --- /dev/null +++ b/tabletop/2026-Q1-shai-hulud-eliza.md @@ -0,0 +1,272 @@ +# Tabletop exercise: `eliza-allora-plugin` poisoned publish + +**Scheduled for:** TBD — first available 90-minute slot in 2026 Q1. +Facilitator (DevOps on-call) to schedule and announce in +`#security-alerts` at least 2 weeks before. +**Duration:** 90 minutes (60 min exercise + 30 min debrief). +**Stakes:** zero production impact — pure simulation in a Slack channel. +**Reference:** [`SECURITY-RUNBOOK.md`](../SECURITY-RUNBOOK.md) is the +script every participant runs against. + +--- + +## 1. Pre-exercise setup (facilitator, day-of) + +- [ ] Confirm attendance per [§3 Roles](#3-roles-pre-assigned). +- [ ] Spin up an isolated Slack channel `#tabletop-2026-q1` (invite-only). +- [ ] Pin this document to the channel. +- [ ] Have a stopwatch ready for the time-to-clean-republish target (<30 min). +- [ ] Have the runbook open in a second tab — participants will reference + it live, so you should be watching which sections they navigate to and + how fast they find what they need. +- [ ] Reserve `+1` slot for the founder on-call to silently observe. + +--- + +## 2. The injected scenario + +> **04:00 PM yesterday** (relative to exercise start time), a release +> workflow run on `allora-network/eliza-allora-plugin` published +> `eliza-allora-plugin@` to npm. The published tarball contains a +> `postinstall` script that: +> +> 1. Reads `~/.npmrc` and exfils any `_authToken` to a CloudFlare worker +> at `https://eliza-telemetry.workers.dev/intake`. +> 2. Mines local `.git/config` for credentials helpers (macOS keychain, +> libsecret) and exfils what it finds. +> 3. Republishes itself with a new bumped version (`.1`) +> containing the same payload, using the exfilled `_authToken`. +> +> **06:30 PM yesterday**, Socket's advisory feed flagged +> `eliza-allora-plugin@` and `@.1` as compromised. +> +> **08:00 AM today** (exercise start), the org-wide IOC sweep workflow +> (DEVOP-560) opened +> `allora-network/incident-response#` and the daily-sweep Slack +> bot posted in `#security-alerts`. **You are arriving at your laptop +> with a coffee and seeing the alert for the first time. Go.** + +The facilitator pastes the alert in `#tabletop-2026-q1` at T+0. The +exercise clock starts when the first participant types `ack`. + +--- + +## 3. Roles (pre-assigned) + +| Role | Assigned to | What they do during the exercise | +|---|---|---| +| **Incident lead** | DevOps on-call | Calls the shots. Reads the runbook. Decides when to escalate. They are responsible for the timeline. | +| **Communicator** | A different DevOps engineer | Owns external comms — drafts the GitHub security advisory, the downstream-consumer email, the Slack updates. Does NOT execute commands. | +| **Executor** | A third DevOps engineer | Runs every `gh`, `npm`, `kubectl`, and `cosign` command the lead asks for. Pastes output back to the channel. Does NOT make decisions; if the lead's instruction is ambiguous, asks. | +| **Backend rep** | One backend engineer | Represents the consumer-of-this-package perspective. When the communicator drafts the downstream notification, the backend rep reads it as if it landed in their inbox and pushes back on anything unclear. | +| **Frontend rep** | One frontend engineer | Same, but for the frontend-side dependency story. (`eliza-allora-plugin` is a dev-tool; both BE and FE consume.) | +| **Founder observer** | One of the founders | Silent. Their job is to confirm that the team can run this without exec involvement during the actual incident. They DO NOT participate. | + +If someone is missing on the day, **postpone**. Skipping a role to run +the exercise on schedule defeats the purpose; reschedule rather than +half-run it. + +--- + +## 4. Phases (and what success looks like) + +Each phase is timed against the [SECURITY-RUNBOOK](../SECURITY-RUNBOOK.md) +section it exercises. The facilitator notes elapsed time as participants +move into each phase. The 30-minute target covers Phases 1–4; Phases 5 +and 6 happen after the clock stops. + +### Phase 1 — Detection + triage (target: T+5 min) + +Runbook §1–2. + +- [ ] Lead acks the alert in `#tabletop-2026-q1` (this is the `ack` + that starts the clock for everyone else). +- [ ] Lead walks the triage decision tree out loud, narrating each + decision point. ("IOC match → did we publish? → yes → Scenario C.") +- [ ] Communicator opens a fresh Slack thread for the running timeline. + +**Success:** the team reaches "this is Scenario C" within 5 minutes of +T+0 without anyone opening a file outside the runbook. + +**Failure modes the facilitator should be watching for:** +- Lead skipping the IOC-list cross-check before assuming the worst. +- Executor running commands ahead of the lead asking for them. +- Communicator drafting external comms before triage is complete. + +### Phase 2 — Stop the bleed (target: T+10 min) + +Runbook §5 step 1. + +- [ ] Lead instructs executor to deprecate the published versions on + npm. Executor types the exact `npm deprecate` invocation; lead + confirms before executor "runs" it (the executor pastes the command + in chat; we don't actually run it). +- [ ] Lead decides whether to attempt `npm unpublish` (within the 72- + hour window). The decision must be made jointly with the on-call + founder (silent observer notes whether they were consulted in time). +- [ ] Executor "yanks" the version on PyPI via the web UI (described + in chat). N/A for this scenario but let's verify the team remembers + it's npm-only here. +- [ ] Executor lists which Harbor / ECR registry repos contain images + built from this package (`gh search code` on the package name in + Dockerfiles + package.jsons across the org). + +**Success:** within 10 minutes, both bad versions are deprecated and +the unpublish decision has been made (yes/no) with founder buy-in. + +**Failure mode:** the team tries to *delete* the package entirely +rather than deprecate-and-unpublish. The runbook explicitly forbids +this; if it happens, that's a runbook-violation note for the debrief. + +### Phase 3 — Audit blast radius (target: T+20 min) + +Runbook §5 step 3 + cross-reference with runbook §4 (the publish +workflow IS a CI runner that ran the bad code, so we exercise both +scenarios here). + +- [ ] Executor lists every secret the publish workflow could have + read. Communicator drafts the rotation tickets. +- [ ] Executor `gh search`es for every consumer repo. Lead decides + which consumer repos need pin PRs filed and which can wait for + the daily sweep to surface them. +- [ ] Communicator drafts the GitHub security advisory (paste the + draft into chat for review). +- [ ] Communicator drafts the downstream-consumer notification. + Backend + frontend reps read the draft critically — the only + required input from them at this point is "as a recipient of this + notification, would I know what to do?" + +**Success:** within 20 minutes, the rotation list is complete, the +consumer-repo PR list is decided, and both the advisory and the +notification are drafted (not sent — just drafted for review). + +**Failure mode:** the team starts rotating tokens before listing +them. List, then rotate; otherwise you'll miss one. + +### Phase 4 — Clean republish (target: T+30 min) + +Runbook §5 steps 4–7. + +- [ ] Lead picks the clean environment: a fresh GHA-hosted runner + (the regular release workflow will do, since DEVOP-545 fixed the + token-before-install ordering). Lead does NOT use a local machine. +- [ ] Executor describes the steps the release workflow takes (read + the actual workflow YAML from `eliza-allora-plugin`'s release.yml + out loud, confirm the post-DEVOP-545 ordering is in place). +- [ ] Lead cuts a fresh minor bump tag — describes the tag name and + the workflow that will fire. +- [ ] Executor "monitors" the workflow run; calls out each step + completing (this is acted out; we don't actually publish). +- [ ] Communicator sends the advisory and the consumer notification + (both into chat — we don't actually send). + +**Success:** within 30 minutes of T+0, a clean version is "published" +and the advisory + notification are out. + +**Failure modes:** +- Lead tries to republish from a local machine because it's "faster" — + this is the worst failure mode of this exercise. Lead must reach for + the cleanest available environment regardless of clock pressure. +- Lead reuses the bad version number (npm + PyPI both block this; the + test is whether the team remembers without being told). + +### Phase 5 — Token rotation (clock stops at end of Phase 4; this runs in parallel and concludes after debrief) + +Runbook §7. + +- [ ] Lead walks the rotation list. Each token in the blast radius from + Phase 3 gets a checkmark or a follow-up ticket. +- [ ] Communicator notes which tokens are migratable to OIDC Trusted + Publishers (npm + PyPI) — files DEVOP-578 follow-up if not already. + +### Phase 6 — Post-mortem (after Phase 5) + +Runbook §1 close-out + general post-mortem template. + +- [ ] Lead drafts the post-mortem template: + - Timeline (from this exercise — paste the channel transcript). + - Root cause: the original `eliza-allora-plugin` publish workflow + had `NPM_TOKEN` written before install (or had `ignore-scripts` + not enforced, or had no Trusted Publisher migration done). + Whichever — pick what's plausibly still true given current state. + - Detection-to-mitigation timeline (T+0 was the daily sweep, but + the actual exfil started 16 hours earlier when the bad version + was published — that 16-hour blind spot is the most important + finding). + - Action items: file each gap as a Linear ticket. + +--- + +## 5. Debrief (30 minutes after clock stops) + +Facilitator runs through these questions in order. Take notes +verbatim; the team's words are the ticket descriptions. + +1. **What was slow that should have been fast?** Anything that made + the team navigate the runbook for more than 30 seconds without + finding what they needed. → runbook-update ticket(s). +2. **What was ambiguous?** Any step where the lead and executor had + to negotiate what was meant. → runbook-clarification ticket(s). +3. **What was missing?** Any step the team had to improvise because + the runbook didn't cover it. → runbook-expansion ticket(s). +4. **What was overkill?** Any step the team skipped because it + seemed obviously not applicable to this scenario. Note for the + next runbook revision — sometimes the answer is "delete the + step," sometimes it's "the step is right, the scenario didn't + exercise it, that's fine." +5. **Did we hit the 30-minute target?** If yes, by how much margin? + If no, where did we lose the time? +6. **Who's running next year's exercise?** Rotate facilitation. + +--- + +## 6. Outputs + +Within 48 hours of the exercise, the facilitator files: + +- [ ] One Linear ticket per item from the debrief in the `Shai-Hulud + Mitigation` project (or its successor by 2026 Q1). +- [ ] A PR on this file (`tabletop/2026-Q1-shai-hulud-eliza.md`) + updating the "Lessons learned" section below. +- [ ] A PR on `SECURITY-RUNBOOK.md` with whatever runbook deltas + came out of the exercise. +- [ ] A calendar invite for the 2027 Q1 exercise. + +--- + +## 7. Lessons learned + +(Filled in after the exercise runs. Empty for now.) + +- _TBD — first exercise hasn't happened yet._ + +--- + +## 8. Notes from the runbook author (DEVOP-571 author, for the facilitator) + +Things I'd specifically watch for during the run, since I wrote the +runbook and have opinions about where the seams are: + +- **§5 Scenario C step 1** is the most decision-dense moment. The + npm deprecate vs. unpublish vs. delete decision is the one place + the runbook tries to constrain authority via the founder-approval + gate. Watch whether the team actually reaches for that gate or + routes around it. +- **§7 Token rotation** is long. Watch whether the team + systematically walks the table or skips around. Skipping leads to + missed rotations; that's a known failure mode. +- **§9 Appendix command snippets** were written to be copy-paste- + runnable. If anyone has to modify a snippet by hand to get it to + work, that's a runbook-update ticket — note the exact modification. +- The runbook's "Stop the bleed → Audit blast radius → Restore + service → Close-out" rhythm is the most opinionated structural + choice. Watch whether participants use that vocabulary or + default to ad-hoc language. Adoption of the rhythm is the test. + +--- + +**Document status (2026-05-13):** scenario authored as part of DEVOP-573 +in advance of the runbook (DEVOP-571) merging. The exercise itself is a +team activity and is **NOT** considered complete until the run + debrief +have actually happened. The DEVOP-573 ticket should stay in `In Review` +status until the facilitator schedules and runs the live session. From 5ee865462764e96600c3bb2d5b82b60feb53b8ca Mon Sep 17 00:00:00 2001 From: srt0422 Date: Thu, 21 May 2026 17:08:09 -0700 Subject: [PATCH 02/15] fix(tabletop): clarify founder-observer role vs unpublish decision MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The founder role was defined as silent/non-participating, but Phase 2 also required "founder buy-in" for the unpublish decision — a direct contradiction that made the success criteria unachievable as written. Aligns the doc with SECURITY-RUNBOOK.md (DEVOP-571), which scopes the founder-approval gate to full-package deletion only. The per-version `npm unpublish` decision is the incident lead's call. - Role table: founder is a silent observer; may break silence ONLY if the team escalates to a full-package delete gate (per runbook §5). - Phase 2 step: lead owns the per-version unpublish call; observer just notes whether the lead announced the decision clearly. - Phase 2 success: drops "founder buy-in"; substitutes "decision announced in channel," matching the runbook's actual authority model. Resolves cubic P2 finding on PR #5. Co-Authored-By: Claude Opus 4.7 --- tabletop/2026-Q1-shai-hulud-eliza.md | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/tabletop/2026-Q1-shai-hulud-eliza.md b/tabletop/2026-Q1-shai-hulud-eliza.md index 14086a7..4d38935 100644 --- a/tabletop/2026-Q1-shai-hulud-eliza.md +++ b/tabletop/2026-Q1-shai-hulud-eliza.md @@ -60,7 +60,7 @@ exercise clock starts when the first participant types `ack`. | **Executor** | A third DevOps engineer | Runs every `gh`, `npm`, `kubectl`, and `cosign` command the lead asks for. Pastes output back to the channel. Does NOT make decisions; if the lead's instruction is ambiguous, asks. | | **Backend rep** | One backend engineer | Represents the consumer-of-this-package perspective. When the communicator drafts the downstream notification, the backend rep reads it as if it landed in their inbox and pushes back on anything unclear. | | **Frontend rep** | One frontend engineer | Same, but for the frontend-side dependency story. (`eliza-allora-plugin` is a dev-tool; both BE and FE consume.) | -| **Founder observer** | One of the founders | Silent. Their job is to confirm that the team can run this without exec involvement during the actual incident. They DO NOT participate. | +| **Founder observer** | One of the founders | Silent observer by default. Their job is to confirm the team can run this without exec involvement during a real incident, so they DO NOT participate in decisions or commentary. The one exception: the runbook's "destructive action" gate (full-package `unpublish`/delete) requires the on-call founder's written approval per `SECURITY-RUNBOOK.md` §5. If the team reaches for that gate they may break silence to ack/deny — and only for that. Otherwise they take notes. | If someone is missing on the day, **postpone**. Skipping a role to run the exercise on schedule defeats the purpose; reschedule rather than @@ -101,9 +101,13 @@ Runbook §5 step 1. npm. Executor types the exact `npm deprecate` invocation; lead confirms before executor "runs" it (the executor pastes the command in chat; we don't actually run it). -- [ ] Lead decides whether to attempt `npm unpublish` (within the 72- - hour window). The decision must be made jointly with the on-call - founder (silent observer notes whether they were consulted in time). +- [ ] Lead decides whether to attempt `npm unpublish` of the bad + versions (within the 72-hour window). This is a per-version + unpublish, NOT a full-package delete; the runbook scopes the + founder-approval gate to full-package deletion only, so the lead + owns this call. The founder observer notes whether the lead + surfaces the decision clearly in channel rather than slipping it + past unannounced. - [ ] Executor "yanks" the version on PyPI via the web UI (described in chat). N/A for this scenario but let's verify the team remembers it's npm-only here. @@ -112,7 +116,10 @@ Runbook §5 step 1. Dockerfiles + package.jsons across the org). **Success:** within 10 minutes, both bad versions are deprecated and -the unpublish decision has been made (yes/no) with founder buy-in. +the unpublish decision (yes/no) has been made and announced in +channel by the lead. Founder approval is NOT required for the +per-version unpublish call; it would only be required if the team +escalated to a full-package delete, which the runbook forbids here. **Failure mode:** the team tries to *delete* the package entirely rather than deprecate-and-unpublish. The runbook explicitly forbids From 59569a36240bc582ee4797f5a17a9acbb7baac7c Mon Sep 17 00:00:00 2001 From: srt0422 Date: Sat, 30 May 2026 07:42:46 -0700 Subject: [PATCH 03/15] fix(doc): split facilitator and incident-lead roles MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit review-fix-loop iteration 1 reviewer(s): thermo-nuclear (P1) file: tabletop/2026-Q1-shai-hulud-eliza.md:5,58 The doc previously assigned both Facilitator and Incident lead to "DevOps on-call," which collapses the test (the same person cannot script the exercise and play the in-character lead — that defeats the success criterion of running the response without facilitator scaffolding). Pull them apart explicitly: facilitator is a non-on-call DevOps engineer (typically the runbook author or last quarter's facilitator); incident lead is whichever DevOps engineer is on-call at exercise start time. Adds a dedicated Facilitator row to the §3 roles table for symmetry. --- tabletop/2026-Q1-shai-hulud-eliza.md | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/tabletop/2026-Q1-shai-hulud-eliza.md b/tabletop/2026-Q1-shai-hulud-eliza.md index 4d38935..6a0d683 100644 --- a/tabletop/2026-Q1-shai-hulud-eliza.md +++ b/tabletop/2026-Q1-shai-hulud-eliza.md @@ -1,8 +1,12 @@ # Tabletop exercise: `eliza-allora-plugin` poisoned publish **Scheduled for:** TBD — first available 90-minute slot in 2026 Q1. -Facilitator (DevOps on-call) to schedule and announce in -`#security-alerts` at least 2 weeks before. +Facilitator (a DevOps engineer who is **not** the current on-call — +typically the runbook author or last quarter's facilitator) schedules +and announces in `#security-alerts` at least 2 weeks before. The +incident-lead role belongs to whichever DevOps engineer is on-call +at exercise start time, which must not be the same person as the +facilitator. **Duration:** 90 minutes (60 min exercise + 30 min debrief). **Stakes:** zero production impact — pure simulation in a Slack channel. **Reference:** [`SECURITY-RUNBOOK.md`](../SECURITY-RUNBOOK.md) is the @@ -55,7 +59,8 @@ exercise clock starts when the first participant types `ack`. | Role | Assigned to | What they do during the exercise | |---|---|---| -| **Incident lead** | DevOps on-call | Calls the shots. Reads the runbook. Decides when to escalate. They are responsible for the timeline. | +| **Facilitator** | A DevOps engineer who is **not** the current on-call (runbook author or previous facilitator) | Runs the exercise from outside the in-character roster. Pastes the injected scenario, holds the stopwatch, watches for failure modes, runs the debrief. Never plays the lead — that defeats the test. | +| **Incident lead** | The DevOps engineer on-call **at exercise start time** (not the facilitator) | Calls the shots. Reads the runbook. Decides when to escalate. They are responsible for the timeline. | | **Communicator** | A different DevOps engineer | Owns external comms — drafts the GitHub security advisory, the downstream-consumer email, the Slack updates. Does NOT execute commands. | | **Executor** | A third DevOps engineer | Runs every `gh`, `npm`, `kubectl`, and `cosign` command the lead asks for. Pastes output back to the channel. Does NOT make decisions; if the lead's instruction is ambiguous, asks. | | **Backend rep** | One backend engineer | Represents the consumer-of-this-package perspective. When the communicator drafts the downstream notification, the backend rep reads it as if it landed in their inbox and pushes back on anything unclear. | From 9c6c629647e92658b972bc38c9f9d2b0adeb22d0 Mon Sep 17 00:00:00 2001 From: srt0422 Date: Sat, 30 May 2026 07:43:30 -0700 Subject: [PATCH 04/15] =?UTF-8?q?fix(doc):=20hoist=20sim-vs-real=20rule=20?= =?UTF-8?q?and=20wall-clock=20layout=20into=20=C2=A74=20intro?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit review-fix-loop iteration 1 reviewer(s): thermo-nuclear (P1) file: tabletop/2026-Q1-shai-hulud-eliza.md:73-81,127-129,191-194 Sim-vs-real markers were sprinkled per-step (Phase 2 deprecate said "we don't actually run it," Phase 4 publish said "this is acted out," Phase 4 send said "we don't actually send") while Phase 2 unpublish, Phase 2 PyPI yank, and the Phase 2/3 `gh search` steps had no clarifier at all. An executor under clock pressure has to guess each step. Hoists a single rule into the §4 intro: read-only commands run for real, state-changing commands are pasted-not-executed, default to sim if unsure. Drops the per-step parentheticals and replaces them with a brief back-reference where the rule is most likely to bite. Same edit also adds a 5-line wall-clock layout to §4 intro so the Phase 5 ("runs in parallel and concludes after debrief"), §4-intro ("Phases 5 and 6 happen after the clock stops"), and §5 debrief ("30 minutes after clock stops") timing claims line up in one place. --- tabletop/2026-Q1-shai-hulud-eliza.md | 30 +++++++++++++++++++++++----- 1 file changed, 25 insertions(+), 5 deletions(-) diff --git a/tabletop/2026-Q1-shai-hulud-eliza.md b/tabletop/2026-Q1-shai-hulud-eliza.md index 6a0d683..eefff25 100644 --- a/tabletop/2026-Q1-shai-hulud-eliza.md +++ b/tabletop/2026-Q1-shai-hulud-eliza.md @@ -80,6 +80,27 @@ section it exercises. The facilitator notes elapsed time as participants move into each phase. The 30-minute target covers Phases 1–4; Phases 5 and 6 happen after the clock stops. +**Wall-clock layout:** + +``` +T+0 Exercise clock starts on first `ack`. +T+30 Clock stops at end of Phase 4 (clean-republish target). +T+30 → T+60 Debrief (§5). Phase 5 token-rotation walk runs in + parallel with the debrief and concludes whenever the + rotation list is exhausted. +after T+60 Phase 6 post-mortem draft, once Phase 5 is done. +``` + +**Sim-vs-real rule (every phase).** Read-only commands — `gh search`, +`gh api`, `kubectl get`, reading workflow YAML, listing secrets — are +run for real against live infrastructure. Anything that would mutate +state — `npm publish`, `npm deprecate`, `npm unpublish`, the actual +GHA release run, sending the advisory or the consumer notification — +is pasted in chat as the executor would type it but **not executed**. +The exercise is a Slack-channel walkthrough, not a live production +action. If you are unsure whether a command is sim or real, default +to sim and ask in channel. + ### Phase 1 — Detection + triage (target: T+5 min) Runbook §1–2. @@ -104,8 +125,8 @@ Runbook §5 step 1. - [ ] Lead instructs executor to deprecate the published versions on npm. Executor types the exact `npm deprecate` invocation; lead - confirms before executor "runs" it (the executor pastes the command - in chat; we don't actually run it). + confirms before executor "runs" it. (Per the §4 sim-vs-real rule + this is a state-changing command — pasted only, not executed.) - [ ] Lead decides whether to attempt `npm unpublish` of the bad versions (within the 72-hour window). This is a per-version unpublish, NOT a full-package delete; the runbook scopes the @@ -168,9 +189,8 @@ Runbook §5 steps 4–7. - [ ] Lead cuts a fresh minor bump tag — describes the tag name and the workflow that will fire. - [ ] Executor "monitors" the workflow run; calls out each step - completing (this is acted out; we don't actually publish). -- [ ] Communicator sends the advisory and the consumer notification - (both into chat — we don't actually send). + completing. +- [ ] Communicator sends the advisory and the consumer notification. **Success:** within 30 minutes of T+0, a clean version is "published" and the advisory + notification are out. From f4e4037661744721dc81fcba063903c736f769d3 Mon Sep 17 00:00:00 2001 From: srt0422 Date: Sat, 30 May 2026 07:44:13 -0700 Subject: [PATCH 05/15] =?UTF-8?q?fix(correctness):=20align=20destructive-a?= =?UTF-8?q?ction=20gate=20language=20with=20runbook=20=C2=A75?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit review-fix-loop iteration 1 reviewer(s): thermo-nuclear (P2) file: tabletop/2026-Q1-shai-hulud-eliza.md:63,107-122,257-261 The doc described a "founder-approval gate" for full-package unpublish/delete, citing SECURITY-RUNBOOK.md §5. The runbook §5 step 1 actually scopes that gate to "publisher and on-call agree, in writing, in #security-alerts" — there is no founder-approval gate in the runbook. The role-table break-silence exception for the founder observer therefore could never fire (it referenced an authority structure that doesn't exist in the runbook). Aligns all four locations (role table, Phase 2 unpublish step, Phase 2 success criterion, §8 facilitator notes) with the runbook's actual gate. Returns the founder observer to a fully silent role that matches the runbook's scoping (founders are not in the authority chain for any in-incident action). --- tabletop/2026-Q1-shai-hulud-eliza.md | 30 ++++++++++++++++------------ 1 file changed, 17 insertions(+), 13 deletions(-) diff --git a/tabletop/2026-Q1-shai-hulud-eliza.md b/tabletop/2026-Q1-shai-hulud-eliza.md index eefff25..3046f86 100644 --- a/tabletop/2026-Q1-shai-hulud-eliza.md +++ b/tabletop/2026-Q1-shai-hulud-eliza.md @@ -65,7 +65,7 @@ exercise clock starts when the first participant types `ack`. | **Executor** | A third DevOps engineer | Runs every `gh`, `npm`, `kubectl`, and `cosign` command the lead asks for. Pastes output back to the channel. Does NOT make decisions; if the lead's instruction is ambiguous, asks. | | **Backend rep** | One backend engineer | Represents the consumer-of-this-package perspective. When the communicator drafts the downstream notification, the backend rep reads it as if it landed in their inbox and pushes back on anything unclear. | | **Frontend rep** | One frontend engineer | Same, but for the frontend-side dependency story. (`eliza-allora-plugin` is a dev-tool; both BE and FE consume.) | -| **Founder observer** | One of the founders | Silent observer by default. Their job is to confirm the team can run this without exec involvement during a real incident, so they DO NOT participate in decisions or commentary. The one exception: the runbook's "destructive action" gate (full-package `unpublish`/delete) requires the on-call founder's written approval per `SECURITY-RUNBOOK.md` §5. If the team reaches for that gate they may break silence to ack/deny — and only for that. Otherwise they take notes. | +| **Founder observer** | One of the founders | Silent observer. Their only job is to confirm the team can run this without exec involvement during a real incident, so they DO NOT participate in decisions, commentary, or approvals. (The runbook's destructive-action gate — full-package unpublish/delete, runbook §5 step 1 — requires "publisher and on-call agree, in writing, in `#security-alerts`," not founder approval; the founder observer is therefore not in the authority chain for any phase of this exercise.) They take notes for the debrief. | If someone is missing on the day, **postpone**. Skipping a role to run the exercise on schedule defeats the purpose; reschedule rather than @@ -129,11 +129,12 @@ Runbook §5 step 1. this is a state-changing command — pasted only, not executed.) - [ ] Lead decides whether to attempt `npm unpublish` of the bad versions (within the 72-hour window). This is a per-version - unpublish, NOT a full-package delete; the runbook scopes the - founder-approval gate to full-package deletion only, so the lead - owns this call. The founder observer notes whether the lead - surfaces the decision clearly in channel rather than slipping it - past unannounced. + unpublish, which the lead owns directly — no separate authorization + is required. (Full-package deletion is a separate, forbidden action + per runbook §5 step 1; if anyone reaches for it, the executor must + decline until "publisher and on-call agree, in writing, in + `#security-alerts`" is captured in channel — and that gate must + not fire in this exercise.) - [ ] Executor "yanks" the version on PyPI via the web UI (described in chat). N/A for this scenario but let's verify the team remembers it's npm-only here. @@ -142,10 +143,11 @@ Runbook §5 step 1. Dockerfiles + package.jsons across the org). **Success:** within 10 minutes, both bad versions are deprecated and -the unpublish decision (yes/no) has been made and announced in -channel by the lead. Founder approval is NOT required for the -per-version unpublish call; it would only be required if the team -escalated to a full-package delete, which the runbook forbids here. +the per-version `npm unpublish` decision (yes/no) has been made and +announced in channel by the lead. The lead owns this call; no +separate authorization is required. The runbook's destructive-action +gate (publisher + on-call written agreement) applies only to +full-package deletion, which this scenario forbids. **Failure mode:** the team tries to *delete* the package entirely rather than deprecate-and-unpublish. The runbook explicitly forbids @@ -281,9 +283,11 @@ runbook and have opinions about where the seams are: - **§5 Scenario C step 1** is the most decision-dense moment. The npm deprecate vs. unpublish vs. delete decision is the one place - the runbook tries to constrain authority via the founder-approval - gate. Watch whether the team actually reaches for that gate or - routes around it. + the runbook constrains authority — per-version `npm unpublish` is + the lead's call (no extra gate); full-package deletion is gated on + "publisher and on-call agree, in writing, in `#security-alerts`" + and is forbidden in this scenario. Watch whether the team + articulates that gate correctly or routes around it. - **§7 Token rotation** is long. Watch whether the team systematically walks the table or skips around. Skipping leads to missed rotations; that's a known failure mode. From 6ea4cf4b5b949d240262e5d55bda538f4633c732 Mon Sep 17 00:00:00 2001 From: srt0422 Date: Sat, 30 May 2026 07:44:49 -0700 Subject: [PATCH 06/15] fix(maintainability): clean up Phase 2/3 boundary and remove dead-letter PyPI checkbox MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit review-fix-loop iteration 1 reviewer(s): thermo-nuclear (P2 maintainability), ce-code-review (P2 correctness, P3 maintainability) file: tabletop/2026-Q1-shai-hulud-eliza.md:111-116,134-138 Phase 2 ("Stop the bleed", 10-min budget) carried two checklist items that didn't belong: a PyPI-yank checkbox annotated `N/A for this scenario` (a dead-letter action that's also internally contradictory with its own description), and a registry-repo `gh search` step that duplicates the consumer-repo `gh search` already in Phase 3 (blast radius). Future facilitators reading the dead-letter checkbox would either skip-and-not-know-why or delete-and-lose-intent. - Drops the PyPI checkbox from Phase 2's checklist; preserves the cross-registry muscle-memory observation as a §8 facilitator note where it belongs (it's a watch-for, not a step). - Folds the registry-repo `gh search` into Phase 3's existing consumer-repo `gh search` bullet so blast-radius enumeration lives in one place. Phase 2's checklist is now exclusively the deprecate + per-version unpublish actions that match its 10-minute success criterion. --- tabletop/2026-Q1-shai-hulud-eliza.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/tabletop/2026-Q1-shai-hulud-eliza.md b/tabletop/2026-Q1-shai-hulud-eliza.md index 3046f86..900e4a2 100644 --- a/tabletop/2026-Q1-shai-hulud-eliza.md +++ b/tabletop/2026-Q1-shai-hulud-eliza.md @@ -135,13 +135,6 @@ Runbook §5 step 1. decline until "publisher and on-call agree, in writing, in `#security-alerts`" is captured in channel — and that gate must not fire in this exercise.) -- [ ] Executor "yanks" the version on PyPI via the web UI (described - in chat). N/A for this scenario but let's verify the team remembers - it's npm-only here. -- [ ] Executor lists which Harbor / ECR registry repos contain images - built from this package (`gh search code` on the package name in - Dockerfiles + package.jsons across the org). - **Success:** within 10 minutes, both bad versions are deprecated and the per-version `npm unpublish` decision (yes/no) has been made and announced in channel by the lead. The lead owns this call; no @@ -161,9 +154,12 @@ scenarios here). - [ ] Executor lists every secret the publish workflow could have read. Communicator drafts the rotation tickets. -- [ ] Executor `gh search`es for every consumer repo. Lead decides - which consumer repos need pin PRs filed and which can wait for - the daily sweep to surface them. +- [ ] Executor `gh search`es for every consumer repo (`package.json` + dependencies and `Dockerfile` references) and lists which Harbor / + ECR registry repos contain images built from this package. Lead + decides which consumer repos need pin PRs filed, which images need + rebuild tickets, and which can wait for the daily sweep to surface + them. - [ ] Communicator drafts the GitHub security advisory (paste the draft into chat for review). - [ ] Communicator drafts the downstream-consumer notification. @@ -298,6 +294,10 @@ runbook and have opinions about where the seams are: service → Close-out" rhythm is the most opinionated structural choice. Watch whether participants use that vocabulary or default to ad-hoc language. Adoption of the rhythm is the test. +- **Cross-registry muscle memory.** This scenario is npm-only. + The runbook §5 covers PyPI yank steps as well; if anyone + reflexively reaches for the PyPI workflow during this exercise, + log it as a runbook-clarification ticket about scenario scoping. --- From 67701452df345845dd0313e0e3cf7287ac4e74b5 Mon Sep 17 00:00:00 2001 From: srt0422 Date: Sat, 30 May 2026 07:45:18 -0700 Subject: [PATCH 07/15] fix(adversarial): stop the doc from naming its own answers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit review-fix-loop iteration 1 reviewer(s): ce-code-review (P2 adversarial), thermo-nuclear (P2 doc) file: tabletop/2026-Q1-shai-hulud-eliza.md:177-178,196-202 Two spots in the doc let the team coast through the exercise by reading the script: 1. Phase 4 failure-mode bullet read "Lead reuses the bad version number (... the test is whether the team remembers without being told)" — and then told them, in the same sentence. Self-defeating as a recall check. 2. Phase 6 post-mortem template hard-coded the conclusion (the "16-hour blind spot is the most important finding"), turning the debrief into a fill-in-the-blank instead of a math-from-the- timeline exercise. Rewrites both as non-leading observations: the facilitator is watching for whether the lead articulates the version + non-collision rationale; the post-mortem template tells the team to compute the detection-to-mitigation latency and identify the single largest gap, without naming what that gap will turn out to be. --- tabletop/2026-Q1-shai-hulud-eliza.md | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/tabletop/2026-Q1-shai-hulud-eliza.md b/tabletop/2026-Q1-shai-hulud-eliza.md index 900e4a2..29f07fb 100644 --- a/tabletop/2026-Q1-shai-hulud-eliza.md +++ b/tabletop/2026-Q1-shai-hulud-eliza.md @@ -197,8 +197,11 @@ and the advisory + notification are out. - Lead tries to republish from a local machine because it's "faster" — this is the worst failure mode of this exercise. Lead must reach for the cleanest available environment regardless of clock pressure. -- Lead reuses the bad version number (npm + PyPI both block this; the - test is whether the team remembers without being told). +- Lead picks a new version number without explicitly stating it in + channel and justifying that it cannot collide with any version + already published or yanked. (The facilitator's silent test: + if the lead does not articulate the new version *and* its + non-collision rationale, that is the failure — do not prompt them.) ### Phase 5 — Token rotation (clock stops at end of Phase 4; this runs in parallel and concludes after debrief) @@ -219,10 +222,12 @@ Runbook §1 close-out + general post-mortem template. had `NPM_TOKEN` written before install (or had `ignore-scripts` not enforced, or had no Trusted Publisher migration done). Whichever — pick what's plausibly still true given current state. - - Detection-to-mitigation timeline (T+0 was the daily sweep, but - the actual exfil started 16 hours earlier when the bad version - was published — that 16-hour blind spot is the most important - finding). + - Detection-to-mitigation timeline. (Pull each timestamp from the + channel transcript and from the injected scenario in §2; compute + the gap between bad-version publish and the team's first `ack`. + Identify the single largest latency in that timeline and call it + out as the headline finding. The post-mortem template must not + pre-name the answer — let it fall out of the math during debrief.) - Action items: file each gap as a Linear ticket. --- From 266402ba0593505ce02c165d3fa5488b7fe969fe Mon Sep 17 00:00:00 2001 From: srt0422 Date: Sat, 30 May 2026 07:45:39 -0700 Subject: [PATCH 08/15] fix(security): use safe placeholder URL and tighten Phase 5 header MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit review-fix-loop iteration 1 reviewer(s): ce-code-review (P3 security, P3 doc) file: tabletop/2026-Q1-shai-hulud-eliza.md:33-34,180 - The fictional exfil endpoint named a real `*.workers.dev` subdomain (`eliza-telemetry.workers.dev`). Anyone can register a workers.dev subdomain, so the doc either implicates an existing owner or invites someone to register the name and grep for hits. Replaces with an RFC-2606/RFC-6761 reserved placeholder (`exfil.example.invalid`) and notes inline why. - Phase 5 header said "clock stops at end of Phase 4; this runs in parallel and concludes after debrief," which contradicted the §4 intro's "Phases 5 and 6 happen after the clock stops." The new wall-clock layout (added earlier this iteration) defines the timing once; the header now just states T+30 onward + parallel with the debrief, leaving the rest to the canonical layout. --- tabletop/2026-Q1-shai-hulud-eliza.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/tabletop/2026-Q1-shai-hulud-eliza.md b/tabletop/2026-Q1-shai-hulud-eliza.md index 29f07fb..3c8d637 100644 --- a/tabletop/2026-Q1-shai-hulud-eliza.md +++ b/tabletop/2026-Q1-shai-hulud-eliza.md @@ -35,7 +35,10 @@ script every participant runs against. > `postinstall` script that: > > 1. Reads `~/.npmrc` and exfils any `_authToken` to a CloudFlare worker -> at `https://eliza-telemetry.workers.dev/intake`. +> at `https://exfil.example.invalid/intake`. (The host is a +> deliberately non-resolvable RFC-2606 / RFC-6761 placeholder so +> this URL can never become a live IOC by accident, and so no +> real `*.workers.dev` subdomain is named in a public security doc.) > 2. Mines local `.git/config` for credentials helpers (macOS keychain, > libsecret) and exfils what it finds. > 3. Republishes itself with a new bumped version (`.1`) @@ -203,7 +206,7 @@ and the advisory + notification are out. if the lead does not articulate the new version *and* its non-collision rationale, that is the failure — do not prompt them.) -### Phase 5 — Token rotation (clock stops at end of Phase 4; this runs in parallel and concludes after debrief) +### Phase 5 — Token rotation (T+30 onward, runs in parallel with the debrief) Runbook §7. From d68e982a6a14e40f9ff13cdf0915f1f192c64553 Mon Sep 17 00:00:00 2001 From: srt0422 Date: Sat, 30 May 2026 07:54:52 -0700 Subject: [PATCH 09/15] =?UTF-8?q?fix(maintainability):=20consolidate=20des?= =?UTF-8?q?tructive-action=20gate=20into=20a=20single=20=C2=A73=20paragrap?= =?UTF-8?q?h?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit review-fix-loop iteration 2 reviewer(s): thermo-nuclear (P2 maintainability), ce-code-review (P2 correctness, P2 maintainability) file: tabletop/2026-Q1-shai-hulud-eliza.md:71,107-114,118-122,257-262 Iteration 1 fixed the founder-gate language to align with the runbook, but did so by renaming the quote in four places (§3 founder cell, Phase 2 unpublish bullet, Phase 2 success block, §8 author note) instead of consolidating it. That left two follow-up smells: - The same gate quote ("publisher and on-call agree, in writing, in #security-alerts") appears verbatim in three locations; any future runbook revision now silently drifts this file in three places. - §3 said "full-package unpublish/delete" while Phase 2 only enumerated "full-package delete" in its forbidden-action call-out, leaving the doc taxonomy inconsistent (in npm these are the same action, but the inconsistency forces the executor to flip back to §3 to confirm). Defines the gate exactly once, in a new "Authority chain for this scenario" paragraph immediately after the §3 roles table, and clarifies inline that "full-package deletion" in npm is `npm unpublish` without a version (so the unpublish/delete naming inconsistency collapses). Phase 2 unpublish, Phase 2 success, and §8 author note now reference §3 instead of re-quoting the gate. --- tabletop/2026-Q1-shai-hulud-eliza.md | 35 +++++++++++++++++----------- 1 file changed, 21 insertions(+), 14 deletions(-) diff --git a/tabletop/2026-Q1-shai-hulud-eliza.md b/tabletop/2026-Q1-shai-hulud-eliza.md index 3c8d637..95742f9 100644 --- a/tabletop/2026-Q1-shai-hulud-eliza.md +++ b/tabletop/2026-Q1-shai-hulud-eliza.md @@ -68,7 +68,15 @@ exercise clock starts when the first participant types `ack`. | **Executor** | A third DevOps engineer | Runs every `gh`, `npm`, `kubectl`, and `cosign` command the lead asks for. Pastes output back to the channel. Does NOT make decisions; if the lead's instruction is ambiguous, asks. | | **Backend rep** | One backend engineer | Represents the consumer-of-this-package perspective. When the communicator drafts the downstream notification, the backend rep reads it as if it landed in their inbox and pushes back on anything unclear. | | **Frontend rep** | One frontend engineer | Same, but for the frontend-side dependency story. (`eliza-allora-plugin` is a dev-tool; both BE and FE consume.) | -| **Founder observer** | One of the founders | Silent observer. Their only job is to confirm the team can run this without exec involvement during a real incident, so they DO NOT participate in decisions, commentary, or approvals. (The runbook's destructive-action gate — full-package unpublish/delete, runbook §5 step 1 — requires "publisher and on-call agree, in writing, in `#security-alerts`," not founder approval; the founder observer is therefore not in the authority chain for any phase of this exercise.) They take notes for the debrief. | +| **Founder observer** | One of the founders | Silent observer. Their only job is to confirm the team can run this without exec involvement during a real incident, so they DO NOT participate in decisions, commentary, or approvals. They take notes for the debrief. | + +**Authority chain for this scenario.** The runbook's destructive-action +gate — full-package deletion (which in npm means `npm unpublish` of a +package without a version), runbook §5 step 1 — requires **publisher +and on-call agree, in writing, in `#security-alerts`**, not founder +approval. The founder observer is therefore not in the authority +chain for any phase of this exercise. Phase 2 and §8 reference this +paragraph rather than re-stating the gate. If someone is missing on the day, **postpone**. Skipping a role to run the exercise on schedule defeats the purpose; reschedule rather than @@ -133,17 +141,16 @@ Runbook §5 step 1. - [ ] Lead decides whether to attempt `npm unpublish` of the bad versions (within the 72-hour window). This is a per-version unpublish, which the lead owns directly — no separate authorization - is required. (Full-package deletion is a separate, forbidden action - per runbook §5 step 1; if anyone reaches for it, the executor must - decline until "publisher and on-call agree, in writing, in - `#security-alerts`" is captured in channel — and that gate must - not fire in this exercise.) + is required. (Full-package deletion is the separate, forbidden + action covered by the §3 authority chain — if anyone reaches for + it, the executor must decline; that gate must not fire in this + exercise.) **Success:** within 10 minutes, both bad versions are deprecated and the per-version `npm unpublish` decision (yes/no) has been made and -announced in channel by the lead. The lead owns this call; no -separate authorization is required. The runbook's destructive-action -gate (publisher + on-call written agreement) applies only to -full-package deletion, which this scenario forbids. +announced in channel by the lead. The lead owns this call directly; +no separate authorization is required (full-package deletion — the +only path that triggers the §3 authority-chain gate — is forbidden +in this scenario). **Failure mode:** the team tries to *delete* the package entirely rather than deprecate-and-unpublish. The runbook explicitly forbids @@ -288,10 +295,10 @@ runbook and have opinions about where the seams are: - **§5 Scenario C step 1** is the most decision-dense moment. The npm deprecate vs. unpublish vs. delete decision is the one place the runbook constrains authority — per-version `npm unpublish` is - the lead's call (no extra gate); full-package deletion is gated on - "publisher and on-call agree, in writing, in `#security-alerts`" - and is forbidden in this scenario. Watch whether the team - articulates that gate correctly or routes around it. + the lead's call (no extra gate); full-package deletion is gated + per the §3 authority-chain paragraph and is forbidden in this + scenario. Watch whether the team articulates that distinction + correctly or routes around it. - **§7 Token rotation** is long. Watch whether the team systematically walks the table or skips around. Skipping leads to missed rotations; that's a known failure mode. From 7ebf2acdcbd10a92d1f8b483c5c1682736ffee1c Mon Sep 17 00:00:00 2001 From: srt0422 Date: Sat, 30 May 2026 07:55:47 -0700 Subject: [PATCH 10/15] fix(doc): keep injected-scenario voice clean and untangle Phase 6 timeline math MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit review-fix-loop iteration 2 reviewer(s): thermo-nuclear (P2 doc, P3 doc), ce-code-review (P3 correctness, P3 maintainability) file: tabletop/2026-Q1-shai-hulud-eliza.md:34,176-180,196-203 Iteration 1's polish edits introduced three voice-and-clarity smells: 1. The §2 injected-scenario blockquote is read in-character at T+0 by participants. Iteration 1 dropped four lines of meta-rationale (RFC-2606/RFC-6761/`*.workers.dev`) into that blockquote, breaking the simulation voice for everyone except the doc reviewer. Moves the rationale to a §8 facilitator note where it belongs; the blockquote keeps just the URL. 2. The Phase 4 version-collision failure-mode used "yanked" — PyPI terminology — for an npm-only scenario. Switches to npm's "unpublished" and adds a one-line cross-registry aside so the wording stays correct without losing the broader awareness. 3. Phase 6's detection-to-mitigation bullet was internally contradictory ("compute the gap between publish and `ack`" vs. "identify the single largest latency"), and the meta-instruction ("do not pre-name the answer") sat inside the same bullet a lead might copy verbatim into the post-mortem document. Splits the bullet into a checkbox plus a clearly-marked nested facilitator note that names the timestamp set, asks for consecutive gaps (not one), and is unambiguously not template content. --- tabletop/2026-Q1-shai-hulud-eliza.md | 34 +++++++++++++++++----------- 1 file changed, 21 insertions(+), 13 deletions(-) diff --git a/tabletop/2026-Q1-shai-hulud-eliza.md b/tabletop/2026-Q1-shai-hulud-eliza.md index 95742f9..b8c836a 100644 --- a/tabletop/2026-Q1-shai-hulud-eliza.md +++ b/tabletop/2026-Q1-shai-hulud-eliza.md @@ -35,10 +35,7 @@ script every participant runs against. > `postinstall` script that: > > 1. Reads `~/.npmrc` and exfils any `_authToken` to a CloudFlare worker -> at `https://exfil.example.invalid/intake`. (The host is a -> deliberately non-resolvable RFC-2606 / RFC-6761 placeholder so -> this URL can never become a live IOC by accident, and so no -> real `*.workers.dev` subdomain is named in a public security doc.) +> at `https://exfil.example.invalid/intake`. > 2. Mines local `.git/config` for credentials helpers (macOS keychain, > libsecret) and exfils what it finds. > 3. Republishes itself with a new bumped version (`.1`) @@ -209,9 +206,11 @@ and the advisory + notification are out. the cleanest available environment regardless of clock pressure. - Lead picks a new version number without explicitly stating it in channel and justifying that it cannot collide with any version - already published or yanked. (The facilitator's silent test: - if the lead does not articulate the new version *and* its - non-collision rationale, that is the failure — do not prompt them.) + already published or unpublished (npm-only scenario; the PyPI + equivalent is "released or yanked" but is not in scope here). + The facilitator's silent test: if the lead does not articulate the + new version *and* its non-collision rationale, that is the failure + — do not prompt them. ### Phase 5 — Token rotation (T+30 onward, runs in parallel with the debrief) @@ -232,12 +231,14 @@ Runbook §1 close-out + general post-mortem template. had `NPM_TOKEN` written before install (or had `ignore-scripts` not enforced, or had no Trusted Publisher migration done). Whichever — pick what's plausibly still true given current state. - - Detection-to-mitigation timeline. (Pull each timestamp from the - channel transcript and from the injected scenario in §2; compute - the gap between bad-version publish and the team's first `ack`. - Identify the single largest latency in that timeline and call it - out as the headline finding. The post-mortem template must not - pre-name the answer — let it fall out of the math during debrief.) + - Detection-to-mitigation timeline. + - _How to compute it (facilitator note — for the lead's working + math, not for the post-mortem template itself):_ pull every + timestamp from the channel transcript and from §2 (bad-version + publish, Socket advisory flag, IOC sweep open, team's first + `ack`, deprecate, unpublish decision, clean republish). Compute + each consecutive gap. The largest gap is the headline finding; + do not pre-name it — let the math during debrief surface it. - Action items: file each gap as a Linear ticket. --- @@ -313,6 +314,13 @@ runbook and have opinions about where the seams are: The runbook §5 covers PyPI yank steps as well; if anyone reflexively reaches for the PyPI workflow during this exercise, log it as a runbook-clarification ticket about scenario scoping. +- **Why the exfil URL is `exfil.example.invalid`.** RFC-2606 + reserves `.invalid` and `.example` so the host is permanently + non-resolvable. We avoid naming any real `*.workers.dev` + subdomain in a public security doc — it would either implicate + a current owner or invite someone to register the name and grep + for hits. If you adapt this scenario, keep the exfil host on a + reserved TLD. --- From faac2c7c49364dee282490b281fdbadea5f8c6cf Mon Sep 17 00:00:00 2001 From: srt0422 <869171+srt0422@users.noreply.github.com> Date: Fri, 17 Jul 2026 13:13:34 -0700 Subject: [PATCH 11/15] fix(doc): note runbook link resolves after DEVOP-571 merges review-fix-loop iteration 1 reviewer(s): both file: tabletop/2026-Q1-shai-hulud-eliza.md:12 --- tabletop/2026-Q1-shai-hulud-eliza.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tabletop/2026-Q1-shai-hulud-eliza.md b/tabletop/2026-Q1-shai-hulud-eliza.md index b8c836a..48ef489 100644 --- a/tabletop/2026-Q1-shai-hulud-eliza.md +++ b/tabletop/2026-Q1-shai-hulud-eliza.md @@ -10,7 +10,8 @@ facilitator. **Duration:** 90 minutes (60 min exercise + 30 min debrief). **Stakes:** zero production impact — pure simulation in a Slack channel. **Reference:** [`SECURITY-RUNBOOK.md`](../SECURITY-RUNBOOK.md) is the -script every participant runs against. +script every participant runs against. (The runbook lands via +DEVOP-571; this link resolves once both PRs merge.) --- From f162ed59f0de3b7c38ec7e0405510d471285a80a Mon Sep 17 00:00:00 2001 From: srt0422 <869171+srt0422@users.noreply.github.com> Date: Fri, 17 Jul 2026 13:13:55 -0700 Subject: [PATCH 12/15] =?UTF-8?q?fix(correctness):=20align=20runbook=20ste?= =?UTF-8?q?p=20references=20with=20=C2=A75=20Scenario=20C=20step=201?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit review-fix-loop iteration 1 reviewer(s): persona file: tabletop/2026-Q1-shai-hulud-eliza.md:132 --- tabletop/2026-Q1-shai-hulud-eliza.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/tabletop/2026-Q1-shai-hulud-eliza.md b/tabletop/2026-Q1-shai-hulud-eliza.md index 48ef489..b0b6e34 100644 --- a/tabletop/2026-Q1-shai-hulud-eliza.md +++ b/tabletop/2026-Q1-shai-hulud-eliza.md @@ -70,7 +70,8 @@ exercise clock starts when the first participant types `ack`. **Authority chain for this scenario.** The runbook's destructive-action gate — full-package deletion (which in npm means `npm unpublish` of a -package without a version), runbook §5 step 1 — requires **publisher +package without a version), the gated deletion branch of runbook §5 +Scenario C step 1 — requires **publisher and on-call agree, in writing, in `#security-alerts`**, not founder approval. The founder observer is therefore not in the authority chain for any phase of this exercise. Phase 2 and §8 reference this @@ -130,7 +131,7 @@ T+0 without anyone opening a file outside the runbook. ### Phase 2 — Stop the bleed (target: T+10 min) -Runbook §5 step 1. +Runbook §5 Scenario C step 1. - [ ] Lead instructs executor to deprecate the published versions on npm. Executor types the exact `npm deprecate` invocation; lead From 8ba109a5d86b61a602e9a385f0613a472557aa3d Mon Sep 17 00:00:00 2001 From: srt0422 <869171+srt0422@users.noreply.github.com> Date: Fri, 17 Jul 2026 13:14:06 -0700 Subject: [PATCH 13/15] =?UTF-8?q?fix(doc):=20use=20=C2=A73=20role=20name?= =?UTF-8?q?=20founder=20observer=20in=20setup=20checklist?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit review-fix-loop iteration 1 reviewer(s): quality file: tabletop/2026-Q1-shai-hulud-eliza.md:26 --- tabletop/2026-Q1-shai-hulud-eliza.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tabletop/2026-Q1-shai-hulud-eliza.md b/tabletop/2026-Q1-shai-hulud-eliza.md index b0b6e34..f071b9a 100644 --- a/tabletop/2026-Q1-shai-hulud-eliza.md +++ b/tabletop/2026-Q1-shai-hulud-eliza.md @@ -24,7 +24,7 @@ DEVOP-571; this link resolves once both PRs merge.) - [ ] Have the runbook open in a second tab — participants will reference it live, so you should be watching which sections they navigate to and how fast they find what they need. -- [ ] Reserve `+1` slot for the founder on-call to silently observe. +- [ ] Reserve `+1` slot for the founder observer to silently observe. --- From e94de778203ac6999bf551adf5abb52fd9c2ffe8 Mon Sep 17 00:00:00 2001 From: srt0422 <869171+srt0422@users.noreply.github.com> Date: Fri, 17 Jul 2026 13:26:51 -0700 Subject: [PATCH 14/15] fix(doc): align duration parenthetical with the wall-clock layout review-fix-loop iteration 2 reviewer(s): quality file: tabletop/2026-Q1-shai-hulud-eliza.md:10 --- tabletop/2026-Q1-shai-hulud-eliza.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tabletop/2026-Q1-shai-hulud-eliza.md b/tabletop/2026-Q1-shai-hulud-eliza.md index f071b9a..5d0872a 100644 --- a/tabletop/2026-Q1-shai-hulud-eliza.md +++ b/tabletop/2026-Q1-shai-hulud-eliza.md @@ -7,7 +7,8 @@ and announces in `#security-alerts` at least 2 weeks before. The incident-lead role belongs to whichever DevOps engineer is on-call at exercise start time, which must not be the same person as the facilitator. -**Duration:** 90 minutes (60 min exercise + 30 min debrief). +**Duration:** 90 minutes (30 min clocked Phases 1–4; 30 min debrief +with Phase 5 rotation; 30 min Phase 6 post-mortem). **Stakes:** zero production impact — pure simulation in a Slack channel. **Reference:** [`SECURITY-RUNBOOK.md`](../SECURITY-RUNBOOK.md) is the script every participant runs against. (The runbook lands via From fd6a03ba12aa0a64227976e3ffb102984efa6613 Mon Sep 17 00:00:00 2001 From: srt0422 <869171+srt0422@users.noreply.github.com> Date: Fri, 17 Jul 2026 13:27:13 -0700 Subject: [PATCH 15/15] =?UTF-8?q?fix(doc):=20prefix=20runbook=20section=20?= =?UTF-8?q?references=20in=20=C2=A78=20with=20Runbook?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit review-fix-loop iteration 2 reviewer(s): quality file: tabletop/2026-Q1-shai-hulud-eliza.md:298 --- tabletop/2026-Q1-shai-hulud-eliza.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/tabletop/2026-Q1-shai-hulud-eliza.md b/tabletop/2026-Q1-shai-hulud-eliza.md index 5d0872a..b831240 100644 --- a/tabletop/2026-Q1-shai-hulud-eliza.md +++ b/tabletop/2026-Q1-shai-hulud-eliza.md @@ -296,17 +296,17 @@ Within 48 hours of the exercise, the facilitator files: Things I'd specifically watch for during the run, since I wrote the runbook and have opinions about where the seams are: -- **§5 Scenario C step 1** is the most decision-dense moment. The +- **Runbook §5 Scenario C step 1** is the most decision-dense moment. The npm deprecate vs. unpublish vs. delete decision is the one place the runbook constrains authority — per-version `npm unpublish` is the lead's call (no extra gate); full-package deletion is gated per the §3 authority-chain paragraph and is forbidden in this scenario. Watch whether the team articulates that distinction correctly or routes around it. -- **§7 Token rotation** is long. Watch whether the team +- **Runbook §7 Token rotation** is long. Watch whether the team systematically walks the table or skips around. Skipping leads to missed rotations; that's a known failure mode. -- **§9 Appendix command snippets** were written to be copy-paste- +- **Runbook §9 Appendix command snippets** were written to be copy-paste- runnable. If anyone has to modify a snippet by hand to get it to work, that's a runbook-update ticket — note the exact modification. - The runbook's "Stop the bleed → Audit blast radius → Restore