Find the stories git can't name, and reconcile the Deploy Ready backlog

[yodem]

Two prod runs of the release-announcement pipeline left stories stuck in Deploy Ready and under-reported the Slack notes. Root-caused to two independent defects, neither of which is a bug in any single step.

Measured against the 20 stories standing in Deploy Ready after 7.1.3:

	count
Shipped — linked PR is an ancestor of the prod tag	11	now transitioned
Correctly pending — merged, genuinely not in prod	2	left alone
No resolvable PR — needs a human	7	reported, not guessed at

1. Discovery ran in one direction only

shipped_stories.py found a story only if sc-NNNNN appeared in a commit subject or PR head branch. Shortcut's own PR↔story links were never consulted.

This PR's own story is the demonstration: it shipped on branch feature/prod-rollout-slack-release-notes, so the release-announcement pipeline could not announce itself — even though Shortcut held the link the whole time. 1 of 20.

Fixed by adding search/stories?query=pr:<N> as a second, independent discovery direction.

2. Nothing ever revisited work outside the current tag range — the dominant defect

Each run walks prev_tag..cur_tag only. A story missed once, or shipped before this pipeline existed, is never looked at again. sc-46548 shipped in 6.111.0 on Aug 12 with a correctly-named branch and was still Deploy Ready a month later. 10 of 20.

Fixed by build/ci/reconcile_deploy_ready.py: an org-wide sweep, every release, over every non-archived Deploy Ready story.

Everything here is deterministic

No LLM decides what shipped or writes any state. Story resolution is a git walk plus REST calls; "did it reach prod" is git merge-base --is-ancestor; transitions are a for-loop. The agent's role is unchanged — it writes prose and holds Read,Write,Glob,Grep only.

Three guards before a linked PR counts as evidence

Each caught a real false positive in live testing:

1. merged — an unmerged PR shipped nothing.
2. repository_id — a story linking PR #224 in another repo otherwise resolves that number against this repo, matches an unrelated older PR, and tests as "in prod".
3. target_branch_name == master — promotion merges resolve to a real story through pr:<N> just as readily as the genuine feature PR does.

They live in one shared module (shortcut_pr_guards.py). The first pass applied them to the sweep but not to the new fallback and the two drifted immediately — the fallback started adopting story ids from promotion merges.

Reconciliation output is kept out of the announcement

Backfilled stories shipped in earlier releases; if they reach the prose step the announcement claims a dozen old features shipped today. The report is written to $RUNNER_TEMP, not the checkout — the prose agent holds Glob+Read over its working directory, so a distinct filename is a naming convention, not an access boundary.

Same principle in the write-back comment: the sweep derives each story's true release via git tag --list 'prod/*' --contains <oid> rather than naming the current one. Verified across 9 distinct releases from 6.79.0 to 7.1.3.

Reviewer notes

• Merging arms the backfill. The first prod-rollout-succeeded after merge transitions the whole standing backlog at once and posts a comment on each — not just that release's stories. Say the word if you'd rather gate it behind one manual run.
• Both scripts are dry-run by default; --apply is required to mutate. --no-comment opts out of the write-back.
• The sweep is Standard-workflow-only. Deploy Ready doesn't exist in the Content workflow; other-workflow stories are routed to triage with their workflow/state ids, never transitioned.
• Enumeration uses the token'd search/stories endpoint, which is org-wide — unlike iterations-get-active, which is silently scoped to the caller's own teams and has produced silently-incomplete results here before.
• Tests aren't collected by pytest.ini. Run by path:
  python3 -m pytest build/ci/tests/ -q -p no:django -c /dev/null → 174 passed
• Reconcile dry-run against live Shortcut reproduces 11/2/7 with zero mutations and zero comments posted.

Background and the reusable pattern: concepts/Shortcut Story-to-Deploy Attribution in the wiki.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KJqBksQqzHYs3F54Y4tJRX

[gitvelocity-reviewer]

📊 Code Quality Score: 27/100

Base Score 68 × ESF 0.4 (Small tier, 77 effective lines, 1 file per pre-computed metrics) = 27.2

Category	Score	Factors
🔭 Scope	14/20	Seven files are touched: the prod-release-notes workflow YAML, shipped_stories.py, mark_stories_deployed.py, two new shared modules (shortcut_pr_guards.py, shortcut_comment.py), and two test files (test_mark_stories_deployed.py, test_reconcile_deploy_ready.py). reconcile_deploy_ready.py is a new 684-line standalone script. shortcut_pr_guards.py is imported by both shipped_stories.py and reconcile_deploy_ready.py, making it a new cross-cutting dependency within the CI subsystem.
🏗️ Architecture	13/20	shortcut_pr_guards.py extracts the three PR shipping-evidence guards (merged / right repo / target branch master) that both shipped_stories.py and reconcile_deploy_ready.py must apply identically, replacing two would-be parallel copies. shortcut_comment.py extracts the POST /stories/{id}/comments mechanics shared by mark_stories_deployed.py and reconcile_deploy_ready.py, while deliberately leaving comment-text construction in each caller because the two scripts know different things about which release shipped a story. The workflow adds reconcile_deploy_ready.py as a separate step with its output written to $RUNNER_TEMP rather than GITHUB_WORKSPACE, establishing an access boundary that prevents the reconcile report from leaking into the prose-generation step.
⚙️ Implementation	15/20	reconcile_deploy_ready.py implements four-guard PR classification (workflow id, numeric state id, repo id, target branch), paginated Shortcut search via cursor, concurrent gh merge-OID resolution via ThreadPoolExecutor, and git ancestry checks via git merge-base --is-ancestor with a three-valued return (True/False/None for inconclusive). resolve_shipping_release_tag uses ascending --sort=creatordate to find the earliest prod/* tag containing a commit, the opposite of resolve_default_prod_tag's descending sort. shipped_stories.py adds a third discovery source with a two-layer promotion-PR filter: NOISE_PATTERN pre-filters before any network call, and LONG_LIVED_ENV_BRANCHES filters on the PR's head branch. _reconcile_comment_text builds different comment text than _release_comment_text in mark_stories_deployed.py because the two scripts have different knowledge about which release shipped a story.
⚠️ Risk	10/20	reconcile_deploy_ready.py mutates Shortcut workflow state for potentially many stories across multiple teams in a single run; --dry-run is the default and --apply is required to mutate. The workflow step uses continue-on-error: true so a reconcile failure never blocks release notes generation or Slack posting. The $RUNNER_TEMP isolation prevents the reconcile report from being read by the prose-generation step even if the agent holds Glob+Read over its working directory. A wrong ancestry check (is_ancestor_of_prod returning True incorrectly) would transition a story whose PR has not yet reached prod; the three-valued return (None for inconclusive) ensures resolution failures push toward pending rather than shipped.
✅ Quality	13/15	test_reconcile_deploy_ready.py (768 lines) covers all three classification buckets, all four guards individually and in combination, pagination via cursor, ancestry check exit-code interpretation (0/1/other), dry-run default, --apply, --dry-run overriding --apply, --no-comment, comment API failure non-fatality, resolve_shipping_release_tag ascending-sort behavior, and the regression case where the comment must name the true earlier release rather than the current prod tag. test_shipped_stories.py adds 155 lines covering the RC1 fallback: single match adoption, all three guard rejections, no-match quiet return, ambiguous-match warn-and-skip, HTTP and network error resilience, promotion-PR NOISE_PATTERN pre-filter, LONG_LIVED_ENV_BRANCHES pre-filter, token-absent degradation, and lookup-failure non-abort. test_mark_stories_deployed.py adds 8 tests for comment posting, failed-transition no-comment, dry-run no-POST, dry-run preview text, --no-comment suppression, and API failure non-fatality. shortcut_pr_guards.py and shortcut_comment.py have no dedicated test files; their behavior is covered indirectly through the two existing test files.
🔒 Perf / Security	3/5	fetch_merge_oids and fetch_stories_by_pr both use ThreadPoolExecutor with max_workers=8 for concurrent I/O. SHORTCUT_API_TOKEN is required even for --dry-run in reconcile_deploy_ready.py (documented in the die() message). The $RUNNER_TEMP output path for the reconcile report is described in the workflow comment as a deliberate boundary: a same-directory JSON with real story names is identified as the specific failure mode it prevents, not just a naming convention.

Was this score accurate? 👍 Yes · 👎 No

How this was scored →

_{Scored by GitVelocity · How are scores calculated?}