Find the stories git can't name, and reconcile the Deploy Ready backlog

build/ci/README-prod-release-notes.md

@@ -1,135 +1,145 @@
 # Prod rollout → Slack + release notes: manual setup
 
-This repo's changes (below) are necessary but not sufficient — the following
-still needs a human with real credentials, since none of it can be
-generated or guessed by an agent.
+This repo's changes are necessary but not sufficient — some setup below
+requires a human with real credentials.
 
-## How it works
+## Pipeline
 
 ```
 Argo post-promotion analysis (prod)
   -> repository_dispatch (prod-rollout-succeeded, carries `version` + `chartVersion`)
-  -> build/ci/shipped_stories.py   — walks the prod/* tag range in git,
-                                      resolves Shortcut story codes from
-                                      commit subjects / merged-PR branch
-                                      names, hydrates story details
-  -> build/ci/mark_stories_deployed.py — moves each shipped story
-                                      Deploy Ready -> Done via the
-                                      Shortcut API. A failure here (missing
-                                      token, API error, nothing to move) is
-                                      logged and Slack-alerted but never
-                                      blocks the two steps below.
-  -> sefaria-release-notes skill   — reads the JSON, writes prose only
-  -> scripts/post_to_slack.py      — posts both files to Slack
+  -> build/ci/shipped_stories.py        — resolves shipped Shortcut stories for the release
+  -> build/ci/mark_stories_deployed.py  — moves those stories Deploy Ready -> Done
+  -> build/ci/reconcile_deploy_ready.py — org-wide Deploy Ready sweep (independent of this release)
+  -> build/ci/merge_release_backfill.py — folds any current-release backfill into shipped-stories.json
+  -> build/ci/triage_explainer.py       — opt-in: proposes hypotheses for the reconcile sweep's triage bucket
+     + headless `claude -p`
+  -> sefaria-release-notes skill        — reads shipped-stories.json, writes prose only
+  -> scripts/post_to_slack.py           — posts both files to Slack
 ```
 
-Only the release-notes generation step is an LLM. Deciding which stories a given deploy closed
-is a graph walk over git history and Shortcut IDs, and flipping their
-workflow state is a for-loop over a REST API — neither of those is a job
-for a model. The skill's only input is the JSON that
-`shipped_stories.py` already produced; it does not call GitHub or Shortcut
-itself, and it does not mutate any story.
+Only the release-notes prose step and the opt-in triage explainer are LLM
+steps; everything else is deterministic Python or a REST call.
+
+## Running the scripts
+
+```
+python3 build/ci/shipped_stories.py --version 6.111.0-prod.2 [--out shipped-stories.json] [--repo Sefaria/Sefaria-Project] [--chart-version 0.87.5-prod.1]
+python3 build/ci/shipped_stories.py --range <prev-tag>..<cur-tag> [--out shipped-stories.json]
+```
+Requires `git` and `gh` on PATH. `SHORTCUT_API_TOKEN` is optional; without
+it, story ids are still emitted but hydration and the PR-link fallback are
+skipped.
+
+```
+python3 build/ci/mark_stories_deployed.py --input shipped-stories.json [--dry-run] \
+    [--workflow-id 500000005] [--from-state-id 500000045] [--done-state-id 500000010]
+```
+Requires `SHORTCUT_API_TOKEN` unless `--dry-run` is passed.
+
+```
+python3 build/ci/reconcile_deploy_ready.py [--dry-run]
+python3 build/ci/reconcile_deploy_ready.py --apply
+python3 build/ci/reconcile_deploy_ready.py --apply --prod-tag prod/6.111.0-prod.2+chart.0.87.5-prod.1 --out report.json
+```
+`--dry-run` is the default; nothing is transitioned without `--apply`.
+Requires `git`, `gh`, and `SHORTCUT_API_TOKEN` (required even for `--dry-run`).
+
+```
+python3 build/ci/merge_release_backfill.py \
+    --shipped-stories-out shipped-stories.json \
+    --reconcile-report reconcile-deploy-ready-report.json
+    # writes the merged result back to --shipped-stories-out by default; pass --out to write elsewhere
+```
+
+```
+python3 build/ci/triage_explainer.py extract --report reconcile-deploy-ready-report.json --out triage-only.json
+python3 build/ci/triage_explainer.py resolve-enabled --event-name workflow_dispatch --explain-triage-input true --enable-var ""
+```
 
 ## What's already wired up in this repo
 
 - `helm-chart/sefaria/templates/analysistemplate/rollout-complete.yaml` —
-  a `notify-github` container, gated on `deployEnv == "production"` (the
-  prod HelmRelease sets `deployEnv: production`, not `"prod"`), fires a
-  `repository_dispatch` (`event_type: prod-rollout-succeeded`) once Argo's
-  post-promotion analysis confirms the rollout healthy. Reads a
-  `GH_DISPATCH_TOKEN` key from its OWN dedicated secret
-  (`.Values.secrets.ghDispatch.ref`, default `gh-dispatch-token`) — kept
-  separate from `local-settings-secrets` because that secret is mounted via
-  `envFrom` into every web/task/monitor/cronjob pod, which is far too broad
-  a blast radius for a GitHub PAT.
+  fires `repository_dispatch` (`prod-rollout-succeeded`) once Argo's
+  post-promotion analysis confirms the prod rollout healthy. Uses a
+  `GH_DISPATCH_TOKEN` from its own dedicated secret
+  (`.Values.secrets.ghDispatch.ref`, default `gh-dispatch-token`).
 - `.github/workflows/prod-release-notes.yaml` — listens for that dispatch
-  (or a manual `workflow_dispatch`), resolves the version (and optional
-  chart version, for disambiguating a chart-only rollout), runs
-  `shipped_stories.py` and `mark_stories_deployed.py`, runs the
-  `sefaria-release-notes` skill headlessly, and posts both output files to
-  Slack via `scripts/post_to_slack.py`.
-- `.claude/skills/sefaria-release-notes/` — the skill, shipped in-repo,
-  now takes a shipped-stories JSON file as its only input and only writes
-  prose. It no longer talks to GitHub or Shortcut.
-- **preprod needs no changes.** `rollout-complete-preprod` already exists
-  (same chart, templated per `deployEnv`) and already posts to Slack on a
-  successful preprod rollout — assuming its `SLACK_URL` is populated (see
-  below).
+  (or a manual `workflow_dispatch`), runs the pipeline above, and posts to
+  Slack.
+- `.claude/skills/sefaria-release-notes/` — the release-notes skill,
+  shipped in-repo, takes a shipped-stories JSON file as its only input.
+- Preprod needs no changes — `rollout-complete-preprod` already exists and
+  posts to Slack on a successful preprod rollout (as long as `SLACK_URL`
+  is populated; see below).
 
 ## Still required — infrastructure repo (SOPS-encrypted secret)
 
-**This is the step that makes the whole pipeline live — without it,
-`GH_DISPATCH_TOKEN` is simply absent, the dispatch curl gets a 401, falls
-through `|| /bin/true`, and the entire feature is a silent no-op with
-nothing failing anywhere.**
-
 1. Create a GitHub PAT scoped to `Sefaria/Sefaria-Project` only —
-   fine-grained, **Contents: read and write** permission (required for the
-   `repository_dispatch` API endpoint; this token never needs push/admin
-   access, it only fires a dispatch event).
-2. SOPS-encrypt it into the `infrastructure` repo as its OWN dedicated
-   Secret (NOT `local-settings-secrets` — that secret is mounted into every
-   pod in the deployment; see above) under key `GH_DISPATCH_TOKEN`. This
-   repo's `envs/prod/helmrelease.yaml` already points
-   `secrets.ghDispatch.ref` at `gh-dispatch-token-production`; the
-   infrastructure repo needs to create a Secret with that exact name.
-3. Confirm `flux reconcile` picks it up (or wait for the next 5-minute
-   poll) so the key exists on the `rollout-complete-production` Job's pod
-   before the next prod rollout.
+   fine-grained, **Contents: read and write** permission.
+2. SOPS-encrypt it into the `infrastructure` repo as its own dedicated
+   Secret under key `GH_DISPATCH_TOKEN`, named `gh-dispatch-token-production`
+   to match `envs/prod/helmrelease.yaml`'s `secrets.ghDispatch.ref`.
+3. Confirm `flux reconcile` picks it up before the next prod rollout.
+
+Without this, `GH_DISPATCH_TOKEN` is absent, the dispatch curl 401s and
+falls through `|| /bin/true`, and the whole feature is a silent no-op.
 
 ## Still required — Sefaria-Project GitHub Actions secrets
 
 Add these under repo Settings → Secrets and variables → Actions:
 
 | Secret | Purpose | Notes |
 |---|---|---|
-| `SHORTCUT_API_TOKEN` | `shipped_stories.py` story hydration and `mark_stories_deployed.py` state transitions | Shortcut → Settings → API Tokens. Not the same as the OAuth MCP connection used interactively. |
-| `SLACK_PRODUCT_WEBHOOK` | Non-technical release announcement | A second Slack incoming webhook, pointed at whichever channel should get `release-announcement-product-slack.txt`. Until this is set, that post step is a guarded no-op (won't fail the workflow). |
+| `SHORTCUT_API_TOKEN` | Story hydration, PR-link fallback, and state transitions for all three CI scripts | Shortcut → Settings → API Tokens |
+| `SLACK_PRODUCT_WEBHOOK` | Non-technical release announcement | A second Slack incoming webhook. Until set, that post step is a guarded no-op |
 
 Already exist and are reused as-is: `SLACK_DEPLOY_WEBHOOK`, `GITHUB_TOKEN`,
 `ANTHROPIC_API_KEY`.
 
+Optional — to have the triage explainer opt in automatically on the real
+`repository_dispatch` trigger (a manual `workflow_dispatch` run can already
+opt in per-run via its `explain_triage` input): add a repo-level Actions
+**Variable** (not Secret) named `ENABLE_TRIAGE_EXPLAINER` set to `true`.
+
 ## Worth verifying, not something this session could check
 
 `SLACK_URL` in `local-settings-secrets` — confirm it's actually populated
-for **both** `preprod` and `prod` (not just present as a key). The existing
-`rollout-complete` Slack ping silently no-ops if it's empty or missing
-(`optional: true`), so a misconfigured value wouldn't surface as an error
-anywhere — it would just be quiet.
+for both `preprod` and `prod`. The existing `rollout-complete` Slack ping
+silently no-ops if it's empty or missing (`optional: true`).
 
-## End-to-end verification, once the above is done
+## End-to-end verification
 
-1. **Dry-run the whole pipeline without a real deploy.** Run:
+1. Dry-run the whole pipeline without a real deploy:
 
    ```
    gh workflow run "Prod Release Notes" -f version=<a past prod version, bare, no leading v> -f dry_run=true
    ```
 
-   Add `-f chart_version=<chart version>` if that app version has more than
-   one `prod/*` tag (a chart-only rollout) and you need a specific one.
+   Add `-f chart_version=<chart version>` for a chart-only rollout.
 
-   This exercises tag-range resolution, story hydration, release-notes
-   generation, and both Slack posts, with `mark_stories_deployed.py` run in
-   `--dry-run` mode so nothing in Shortcut actually moves. It's the fastest
-   way to validate a change to any of the scripts or the skill without
-   waiting on a real rollout.
+2. Then confirm the real trigger path: promote something small through to
+   prod normally.
+3. Watch for the existing terse Slack ping from `rollout-complete-production`.
+4. Watch the `Prod Release Notes` GitHub Actions workflow run. If it
+   doesn't fire, check the `notify-github` container's logs on the
+   `rollout-complete-production` Job pod for the dispatch curl's
+   exit/response.
+5. Confirm both Slack files post correctly, and confirm the shipped
+   stories actually moved Deploy Ready → Done in Shortcut.
 
-2. **Then confirm the real trigger path.** Promote something small through
-   to prod normally.
+## Running the tests
 
-3. Watch for the existing terse Slack ping from `rollout-complete-production`
-   (confirms the AnalysisTemplate ran and Slack posting works at all).
+```
+python3 -m pytest build/ci/tests/ -q -p no:django -c /dev/null
+```
 
-4. Watch the `Prod Release Notes` GitHub Actions workflow run
-   (`repository_dispatch` → `prod-rollout-succeeded`). If it doesn't fire,
-   check the `notify-github` container's logs on the `rollout-complete-production`
-   Job pod (`kubectl logs -n default -l job-name=...`) for the dispatch
-   curl's exit/response. The curl runs with `-sS -f --max-time 30
-   --connect-timeout 10`, so a bad/missing token (HTTP 401/403) or a
-   timeout now prints to stderr in the pod logs — but the call still falls
-   through `|| /bin/true` by design so it never blocks or fails the
-   rollout, which means a failure here still won't surface anywhere except
-   those logs unless you go look.
+Both `-c /dev/null` and `-p no:django` are needed: the repo-root
+`pytest.ini` sets `DJANGO_SETTINGS_MODULE`, which makes `pytest-django` try
+to `django.setup()` the whole app even when only these standalone,
+stdlib-only scripts' tests are selected. `-c /dev/null` stops that ini from
+being read; `-p no:django` disables the plugin as a second line of defense.
 
-5. Confirm both Slack files post correctly, and confirm the shipped
-   stories actually moved Deploy Ready → Done in Shortcut.
+No network access, `git`, or `gh` binary is required — everything that
+would otherwise shell out or call the Shortcut API is monkeypatched in the
+tests.

build/ci/mark_stories_deployed.py

@@ -1,57 +1,12 @@
 #!/usr/bin/env python3
 """
-Deterministically transition shipped Shortcut stories from "Deploy Ready" to "Done".
-
-Reads a shipped-stories JSON file (as produced by shipped_stories.py) and, for
-each entry in its "stories" list whose workflow_id equals --workflow-id AND
-workflow_state_id equals --from-state-id, PUTs a workflow_state_id update to
---done-state-id via the Shortcut API. Stories already at the done state are
-skipped and logged. Stories sitting in any other state of --workflow-id are
-skipped and logged. Stories belonging to a DIFFERENT workflow than
---workflow-id are skipped and logged separately, because --from-state-id and
---done-state-id are workflow-specific: the Sefaria org has roughly ten
-Shortcut workflows, and a state id that means "Deploy Ready" in one workflow
-can be meaningless (or mean something else entirely) in another. Silently
-lumping those stories into "skipped, some other state" is exactly the
-unattended-run failure mode this script exists to avoid — a shipped story in
-a non-Standard workflow must never look indistinguishable from "nothing
-needed moving". No story is ever mutated in any of these three skip cases.
+Transitions shipped Shortcut stories from "Deploy Ready" to "Done".
 
 Usage:
     python3 mark_stories_deployed.py --input shipped-stories.json [--dry-run] \
         [--workflow-id 500000005] [--from-state-id 500000045] [--done-state-id 500000010]
 
---dry-run prints exactly what would be transitioned and mutates nothing; it
-does not require SHORTCUT_API_TOKEN. Without --dry-run, a missing
-SHORTCUT_API_TOKEN is a hard error (exit 1) raised before any story is
-touched. A per-story API failure is logged and never aborts the loop and
-never fails the process as a whole — the deploy already happened, so this
-script must never be the thing that turns a green rollout red.
-
-Prints a summary JSON at the end with counts and ids for each bucket:
-transitioned, already_done, skipped_other_state, skipped_different_workflow,
-failed — plus a skipped_detail list carrying each skipped story's id,
-workflow_id, workflow_state_id and the reason it was skipped, and hydrated /
-unresolved_story_ids echoed straight from the input file so stage 3 doesn't
-silently lose stories that shipped but that shipped_stories.py could not
-look up.
-
-Whenever the input's "stories" list is non-empty and at least one story was
-skipped for a reason other than already being Done, this script prints a
-WARNING to stderr naming those stories -- even in a MIXED release where some
-other stories transitioned fine, so a partial skip is never invisible just
-because the run "worked". If, in addition, NOTHING was transitioned at all,
-that combination is a silent no-op -- the exact failure mode this script
-exists to prevent -- and, unless --dry-run was passed, the process exits
-non-zero so an unattended run can't look successful when it silently did
-nothing. (--dry-run never exits non-zero for this: it is a preview, and
-dry-run "transitioned" is inherently just a list of candidates, not evidence
-anything actually happened or failed to.)
-
-All ids shown in this file's docstring and comments (e.g. story id 22222)
-are placeholders, not real Shortcut story ids. The workflow and state ids
-(500000005, 500000045, 500000010, 500000728) are real Shortcut workflow/
-state ids, not story ids, and are not covered by that placeholder rule.
+Requires SHORTCUT_API_TOKEN unless --dry-run is passed.
 """
 
 import argparse
@@ -64,12 +19,7 @@
 
 SHORTCUT_API_BASE = "https://api.app.shortcut.com/api/v3"
 
-# Sefaria's "Standard" Shortcut workflow: "Deploy Ready" -> "Done". These are
-# used as defaults so the script is runnable without extra flags in the
-# common case; pass --workflow-id/--from-state-id/--done-state-id explicitly
-# to override for a different workflow. The org has roughly ten workflows,
-# each with its own state ids — e.g. at least one other workflow has its own
-# Done state, 500000728, which is NOT the same as DEFAULT_DONE_STATE_ID.
+# Sefaria's "Standard" Shortcut workflow: "Deploy Ready" -> "Done".
 DEFAULT_WORKFLOW_ID = 500000005
 DEFAULT_FROM_STATE_ID = 500000045
 DEFAULT_DONE_STATE_ID = 500000010
@@ -85,17 +35,7 @@ def warn(message: str) -> None:
 
 
 def classify_stories(stories, workflow_id, from_state_id, done_state_id):
-    """Split stories into (to_transition, already_done, skipped_other_state,
-    skipped_different_workflow) buckets. Pure function, no I/O.
-
-    Workflow membership is checked BEFORE state, so a story sitting at a
-    different workflow's own Done state (e.g. 500000728) is classified as
-    skipped_different_workflow, not already_done — those state ids are not
-    interchangeable across workflows. A story with no workflow_id on record
-    (e.g. from a shipped-stories.json produced before this field existed)
-    falls through to the state-only checks instead of being flagged as a
-    mismatch, so older input files don't spuriously warn on every story.
-    """
+    """Split stories into (to_transition, already_done, skipped_other_state, skipped_different_workflow) buckets."""
     to_transition = []
     already_done = []
     skipped_other_state = []
@@ -156,8 +96,7 @@ def _ids(stories):
 
 
 def _skipped_detail(skipped_other_state, skipped_different_workflow):
-    """Build the human-readable detail list for every skipped (non-already-done)
-    story: id, workflow_id, workflow_state_id and why it was skipped."""
+    """Build the detail list for every skipped (non-already-done) story: id, workflow_id, workflow_state_id, and reason."""
     detail = []
     for story in skipped_different_workflow:
         detail.append({
@@ -254,13 +193,6 @@ def main():
 
     print(json.dumps(summary, indent=2))
 
-    # Make silent skips impossible: any story skipped for a reason other than
-    # already being Done must be visible, whether or not anything else in
-    # this run transitioned. A MIXED release -- some stories transition
-    # fine, others get skipped for a different workflow/state -- used to
-    # produce no WARNING at all as long as `transitioned` was non-empty; that
-    # silently lost visibility into exactly the stories this script exists
-    # to flag. Only the "nothing happened at all" case is fatal.
     if stories and skipped_detail:
         skipped_ids_and_context = [
             f"{d['id']} (workflow_id={d['workflow_id']}, workflow_state_id={d['workflow_state_id']}, "