[Fleet incident] Reconcile active Actions registry with protected-main workflows

[seonghobae]

Live Actions control-plane drift

The repository's Actions API currently reports 52 workflow identities. Numerous historical bootstrap, autofix, cleanup, probe, and PR-specific repair workflows remain in state: active, including examples such as:

• apply-pr-696-review-fixes.yml through v4;
• autofix-pr-693-contracts.yml;
• autofix-pr-706-locks.yml;
• autofix-pr-716-layout.yml;
• bootstrap-current-pr-queue.yml;
• cleanup-pr-716-artifacts.yml and cleanup-pr-731.yml;
• multiple pr-693-*, pr-700-*, and related repair/probe workflows.

Fresh protected-main contents lookup returns 404 Not Found for sampled active path .github/workflows/apply-pr-696-review-fixes-v4.yml. The workflow source is absent, but its registry identity remains advertised as active.

Root cause

Temporary PR repair workflows were removed from Git history after their bounded use, but the independent GitHub Actions registry state was not disabled. Existing quality checks inspect the current tree and therefore cannot detect orphaned workflow identities whose source files are gone.

The repository consequently has no trustworthy answer to “which writers are enabled,” and retains active metadata tied to superseded PR heads and completed repair attempts.

Feasible remediation

Do not restore the old YAML or create another self-deleting repair workflow. Through a normal reviewed operator/control-plane path:

1. paginate the complete workflow registry;
2. bind the audit to the exact protected-main SHA and tree;
3. distinguish present repository workflows, orphaned repository workflows, disabled records, and GitHub-owned dynamic workflows;
4. disable each active orphan after a second exact-state refetch;
5. preserve current CI, security, dependency, release, fuzz, supported review, and active product workflows;
6. produce an immutable before/after ledger.

Coordinate with organization control-plane issue ContextualWisdomLab/.github#945 and detector issue ContextualWisdomLab/appguardrail#929. The repository owner loop retains all mutation authority.

Prevention and acceptance

• executable read-only audit catches active orphan records with workflow ID/path/state/default-branch SHA and pagination receipts;
• missing permissions, transient 403/404/5xx, or branch movement fail closed;
• no name-only heuristic disables a legitimate live workflow;
• no temporary writer source is reintroduced;
• adversarial tests cover pagination, path case/encoding, workflow rename/ID reuse, GitHub dynamic workflows, and a live workflow with a repair-like name;
• exact-head CI/security/review gates pass;
• normal protected-main integration;
• post-remediation Actions inventory proves all reviewed orphans disabled while supported production workflows still execute.

This is part of the organization-wide Hourly Product Development fleet incident. The fleet audit remains read-only and tracks protected-main operational evidence.

[seonghobae]

Additional exact evidence for the fleet incident:

• PR feat: establish forward engineering plan authority #834 predecessor head 982d7548ce250cc20e56cb10eef1ebdd384f85ee, workflow run 31754183311, job 94626307247, failed only the centrally owned strix gate.
• The first failing boundary was scanner workspace materialization: the workflow exposed 46 changed files as if they were the full repository.
• Strix then reported backend/app/auth.py and backend/app/pg_introspect/dsn_guard.py as missing. Both paths exist at that exact commit, so this is not a product-source omission.
• Immediate cause: a patch-scoped workspace was evaluated with full-repository assumptions. Systemic control failure: the required central gate does not distinguish changed-file materialization from a complete checkout.
• Smallest central remedies are either (a) provide a complete exact-head checkout to Strix, or (b) make the scanner contract explicitly patch-scoped and prohibit absence findings for unmaterialized unchanged paths. Acceptance should replay the exact head and prove the gate no longer produces repository-absence findings from a partial workspace.

The current PR head 917907b664d64e7bc1a49b588f3108fc80b1e5ac is newly queued; its result must be evaluated independently. No pg-erd-cloud gate was weakened.

[seonghobae]

Current-head follow-up for the central Strix control failure:

• PR feat: establish forward engineering plan authority #834 exact head 917907b664d64e7bc1a49b588f3108fc80b1e5ac; workflow run 31760244733; job 94644858766; artifact strix-reports ID 9205308319 (digest sha256:eef2ddc3cc9988789fb667aaec70f0052541821beea6493aae86b3b9432e00bf).
• The gate retained a provisional medium IDOR finding for GET /api/connections/by-project/{project_space_uuid}.
• The same final penetration report explicitly states that require_project_member correctly enforces membership, non-members are denied, the dedicated IDOR agent found no IDOR, and “the controls are working as intended.”
• The PoC is not executable confirmation: it contains placeholder UUIDs, no valid authentication, and no observed 200 response.
• First failing boundary: Strix report reconciliation. A provisional concern remained in vulnerabilities/vuln-0001.md after the later verification disproved it, so the required gate counted Vulnerabilities 1 and failed.
• Immediate/root cause: contradictory agent outputs were aggregated without a fail-safe retraction/adjudication state. Systemic control failure: the central gate treats any provisional vulnerability artifact as authoritative even when the final report explicitly rejects it.

Smallest central remedy: require a structured terminal disposition for each candidate finding (confirmed, rejected, or inconclusive) and count only confirmed; fail closed as infrastructure/inconclusive when agents disagree. Acceptance should replay this artifact and prove it cannot simultaneously report “No IDOR vulnerabilities were found” and emit a confirmed vulnerability count. No repository security gate was weakened, and the PR remains blocked.

[seonghobae]

Bounded rerun result (no further retries):

• Exact head 917907b664d64e7bc1a49b588f3108fc80b1e5ac; rerun job 94655237673; artifact 9206218496; digest sha256:b7a751411cf14966fb2ce40d17f1833a7d4756e0841f17f97c9fafd5b9db9468.
• The rerun failed with a different provisional SSRF report whose premise is that app.db_introspect and validate_postgres_dsn_target are missing.
• Both backend/app/db_introspect.py and backend/app/pg_introspect/dsn_guard.py exist at the exact scanned commit. The latter implements validate_postgres_dsn_target; owned DSN-guard tests and all exact-head CI/security checks passed.
• The PoC uses placeholder credentials/UUIDs and never observed an outbound request. The report explicitly reasons from an assumed missing implementation rather than exercising the existing guard.
• First failing boundary remains the 47-file PR-scope materialization. It exposes changed callers while omitting unchanged imported security modules, then treats their absence as a repository fact.
• The one supported job rerun reproduced the systemic scope defect with a different false premise, so further retries are not a feasible remedy.

Central acceptance must provide a complete exact-head source tree (or prohibit absence findings for unmaterialized files) and replay both artifacts 9205308319 and 9206218496. The required Strix gate remains failed; no bypass or repository security weakening was attempted.

[seonghobae]

Additional live cancellation evidence (2026-08-14)

Two unchanged, open, current-main PR heads retain synchronized cancellation storms across independent owned and central workflows:

• PR docs: align ecosystem names and solo-maintainer review policy #737 head 66a6b3e9c3ee4c33691265e9e5f5264bdc414960: 18 check jobs cancelled. CI run 31728877792, Security Scan 31728877761, and dynamic CodeQL 31728876124 all started at 18:04 UTC and transitioned to cancelled around 18:09 UTC without a successor head.
• PR 🎨 Palette: Add contextual ARIA labels to table edit modal actions #752 head 58301d974401b38adf826614e8ad41a5a0f3f8f1: 15 check jobs cancelled. CI run 31744826595, Security Scan 31744826137, and dynamic CodeQL 31744823668 all started at 21:15 UTC and transitioned to cancelled around 21:19 UTC without a successor head.

The first failing boundary is workflow-run cancellation, not a test, scanner, or source assertion. The API exposes the triggering actors and converged timestamps but not the cancellation initiator/reason, so the root actor remains unproven. Blind reruns are deferred because they would repeat the symptom without changing this control-plane condition.

Acceptance for the incident should therefore also require an immutable cancellation ledger (run, exact head, cancellation actor/reason/event when available) and proof that unchanged open heads are not bulk-cancelled by orphan cleanup or queue writers. Cancelled evidence remains non-passing.

[seonghobae]

Live queue-starvation evidence (2026-08-14)

Four independent, open, exact-head PRs currently retain all three repository-owned workflows in queued with no terminal product failure:

• PR feat: establish forward engineering plan authority #834 head 4eca5e9f2e2cdbb4dbfc0990ce2cde5bda46e079: Security Scan 31788663678=queued; SAST Semgrep 31788663609=queued; ci 31788663622=queued; 20 check jobs queued, 0 failed.
• PR fix(a11y): use native form submission paths #835 head 3ca75ed0648feea85b43e5b042188fb5e73c60ef: SAST Semgrep 31789047675=queued; ci 31789047722=queued; Security Scan 31789047562=queued; 14 check jobs queued, 0 failed.
• PR chore(deps-dev): bump @types/react from 19.2.17 to 19.2.18 in /frontend #839 head 1f29b774b6fc222b1cbef665b4093d43f7c9ad0c: ci 31788457558=queued; SAST Semgrep 31788457448=queued; Security Scan 31788457424=queued; 14 check jobs queued, 0 failed.
• PR chore(deps): bump step-security/harden-runner from 2.19.4 to 2.20.1 #844 head 93642226c6284bb5f5b3a1ade1217dac54b822c2: ci 31788084036=queued; Security Scan 31788084130=queued; SAST Semgrep 31788084106=queued; 14 check jobs queued, 0 failed.

The oldest set is PR #844 (runs 31788084036, 31788084130, and 31788084106), created from its current head before the other three sets. The same synchronized state across unrelated diffs places the first observable boundary at Actions scheduling/runner admission, before repository tests or scanners execute. The API evidence does not identify a queue owner, concurrency holder, or cancellation actor, so root attribution remains unproven.

Smallest acceptance evidence is: (1) each unchanged exact-head repository workflow leaves queued and reaches its own terminal conclusion, or (2) an immutable scheduler/concurrency record identifies why it cannot run and which owner can safely remediate it. Pending/queued evidence remains non-passing; no workflow, required check, branch protection, or security gate was bypassed or weakened.

[seonghobae]

Queue-starvation acceptance update (2026-08-15)

The previously recorded scheduler symptom is no longer active for the four unchanged exact heads, and every referenced repository-owned workflow has now reached completed/success:

• PR feat: establish forward engineering plan authority #834 head d972afaf53995e9319ed1df86d624150350dd61c: ci 31808905664, SAST Semgrep 31808905660, Security Scan 31808905806;
• PR fix(a11y): use native form submission paths #835 head 3ca75ed0648feea85b43e5b042188fb5e73c60ef: ci 31789047722, SAST Semgrep 31789047675, Security Scan 31789047562;
• PR chore(deps-dev): bump @types/react from 19.2.17 to 19.2.18 in /frontend #839 head 1f29b774b6fc222b1cbef665b4093d43f7c9ad0c: ci 31788457558, SAST Semgrep 31788457448, Security Scan 31788457424;
• PR chore(deps): bump step-security/harden-runner from 2.19.4 to 2.20.1 #844 head 93642226c6284bb5f5b3a1ade1217dac54b822c2: ci 31788084036, SAST Semgrep 31788084106, Security Scan 31788084130.

Protected main remains 183331e1054fb14b4c017e77fcd0aae99e949277. This satisfies the issue's terminal-workflow evidence alternative for that bounded starvation observation only. It does not resolve the independent orphaned active-workflow registry, cancellation-ledger, or central scanner-materialization/adjudication controls, so the incident remains open.