Report failed unattended runs to Slack, and check every repository does

[GavinLucas]

What

• .github/workflows/slack-on-failure.yaml, a reusable workflow called from a repository's watcher workflow, which workflow_run starts when one of the workflows it names completes. It posts a failed run to the channel behind SLACK_WEBHOOK, naming the repository, the workflow, its trigger, its conclusion and the jobs that did not succeed, and linking the run.
• A new hygiene check, "unattended runs report their failures": every workflow triggered by schedule, push or release must be named in a watcher reporting that trigger, must have a name:, and every name in a watcher must belong to a workflow.
• This repository's own watcher, report-failures.yaml, for its pushes to main, calling the reporter by local path.
• The README documents the watcher a repository adds.
• A review requested by comment now reviews the head it was made against. The gate reads the branch's last push from GitHub's activity record and refuses, with a reply, if it is at or after the comment or not of the current head; the review checks out the head the gate passed. CodeQL raised the race as actions/untrusted-checkout and actions/untrusted-checkout-toctou on claude-review.yaml once this repository's own caller made it visible; it applies to every repository calling the review, and moves with their pins.
• This repository's own pull requests get the Claude review, through code-review.yaml, the same caller as every other repository's but calling the review by local path, so a change to the review is exercised by reviewing the pull request that makes it. A comment request runs main's version, since GitHub runs comment-triggered workflows from the default branch.

Why

A run started by a schedule, a push to the default branch or a release has nobody watching it, so its failure is a red entry in a list nobody opens. One watcher per repository plus this workflow keeps the filtering and posting in one place. The check is in the shared hygiene script because every repository adopts the convention by moving its pin, so the pin move and the watcher land in the same pull request, and a workflow added later without a watcher entry fails CI.

Behaviour

• The caller lists the triggers to report in events. A push is reported on the default branch and on a tag. Only completed runs are reported, with every conclusion but success, skipped or neutral. The filters are in this workflow, not the caller.
• A cancelled run that a newer run of the same workflow, branch and trigger replaced - cancel-in-progress when two merges land together - is skipped with a notice; the newer run is reported if it fails. A run cancelled by its timeout has no newer run and is posted.
• A post that fails - no secret, a webhook Slack refuses, Slack unreachable after retries - fails the run with Slack's answer in the error.
• A call from anything but workflow_run fails in its first step rather than being skipped.
• The caller must grant actions: read and contents: read, or GitHub refuses to start the job. If the failed jobs cannot be listed, the post still goes and says so.
• Accepted limitation: a run cancelled by its timeout while a newer run was already queued looks like a replaced run, and is not posted.
• SLACK_WEBHOOK is passed explicitly by the caller. GitHub uses the lowest level of a secret defined at several: "If a secret with the same name exists at multiple levels, the secret at the lowest level takes precedence." So a repository secret overrides the organisation's.
• workflow_jobs now reads its documents from workflow_documents, which the new check shares; its behaviour is unchanged.

Testing

• actionlint clean.

• The post step's script run locally against a fake Slack server: accepted; refused with 404; refused with 200 and an error body; 500 then 200; unreachable; no secret; called from push; job listing failing; no failed jobs; &, < and > in the workflow name. The 500-then-200 case found that curl's retry appended the second body to the first on standard output, so an accepted post was reported as refused; the body now goes to a file, which the retry truncates (observed with curl 8.7.1, not stated in curl's manual).

• The new check's function against fourteen fixtures, each passing or producing its intended finding: a clean repository, a local call, pull-request-only workflows, on: as a list and as a string, no watcher, a workflow left off, a trigger missing from events, a stale name, no name:, events not JSON, a watcher without workflows:, another reusable workflow, and no on: (refused as cannot evaluate).

• The whole script over a fresh clone of each repository pinning this one: each gets exactly one finding per push, schedule or release workflow it has, and nothing else. Run over this repository's workflows, the new check finds nothing.

• The superseded-cancellation skip, with a stub gh: replaced (skipped), newest (posted), nothing listed (posted), listing failing (warning, posted), a failed run with a newer one (posted), a branch needing URL-encoding. The live runs endpoint, queried against quaffed's checks, returns the newest run first, numbered per workflow.

• The gate's script with a stub gh and GNU date: a pull request event; a request after the push (runs); a push after the request, in the same second, of another commit, no activity, no comment time (each refused with a reply); the lookup failing (job fails, naming it). The live activity endpoint returns a slashed branch's latest push with its head as after.

Not yet tested: the job-level filter and a real post; and the comment-request gate live, since a comment-triggered run uses the default branch's workflow, which has no caller here until this merges. workflow_run fires only from a default branch, so both are tested once a watcher is merged. This repository's own watcher will be the first, on its next push to main.

[github-actions]

Claude review: changes requested

Summary

The PR adds three things: the Slack reporter (slack-on-failure.yaml), this repository's own watcher and Claude review caller, and a new hygiene check, "unattended runs report their failures". The structure is sound. Failure paths in the post step are handled with care: the response body goes to a file because of the retry behaviour, a missing secret is an error, and the job listing degrades to a warning. I found two behaviour bugs in the reporter, one untrue README claim, and a few recorded-state items.

Findings

1. Bug: a tag-triggered release is required by the hygiene check, but the reporter never posts its failures. .github/workflows/slack-on-failure.yaml:35-36

• The job-level filter only reports a push run when workflow_run.head_branch == repository.default_branch.
• For a run started by a tag push (on: push: tags: ['v*'], a common release trigger), head_branch is the tag name, so the filter skips the job and nothing is posted.
• The hygiene check (check-repo-hygiene.py:582) only looks at the trigger keys. It treats such a workflow as push and demands it be listed with push in events. The repository passes, and a failed release still never reaches Slack. That is the silent green this check exists to prevent.
• The same applies, more mildly, to push: branches: [release/*]: it must be listed but is never reported.
• Suggested fix: in the reporter, also allow a push whose head_branch is not a branch (for example, compare head_branch with the run's head_repository/ref, or accept refs/tags). Or, if tag pushes are deliberately out of scope, have the hygiene check say so for a push-on-tags workflow rather than pass it, and add that limitation to the README.
• I could not confirm head_branch for tag pushes against a live run. My basis is the documented workflow_run payload. Commit d548d2a says the job-level filter is untested, so please confirm on a real tag run.

2. Bug: a watcher without types: [completed] posts on every run start. .github/workflows/slack-on-failure.yaml:37

• workflow_run with no types: fires on requested and in_progress as well as completed.
• On those events conclusion is null, and !contains(["success","skipped","neutral"], null) is true. The job runs, the cancelled check does not apply, and Slack gets "ended ``" with every in-progress job listed as (null). This happens for every unattended run.
• The header comment says "The filters are here, not in each caller, so no caller can get them wrong", but the one filter that makes this a completion report is left to the caller's YAML. The hygiene check doesn't check types: either.
• Suggested fix: add github.event.action == 'completed' (or workflow_run.status == 'completed') to the if:. Optionally, have unattended_run_findings flag a watcher whose workflow_run has no types: [completed].

3. Bug (documentation untrue): "without it the post still goes". README.md:145-146

• A called workflow's job cannot raise the token permissions above the caller's.
• slack-on-failure.yaml asks for actions: read at job level. If the caller's workflow does not grant it, whether through explicit permissions: without actions or through a restricted default token, GitHub refuses to start the run with "The nested job 'report' is requesting 'actions: read', but is only allowed 'actions: none'". No post is made.
• The "could not be listed" fallback in the step only covers an API failure, not a missing grant.
• Suggested fix: say the caller must grant actions: read or the watcher fails to start.

4. Bug (minor): suppressing replaced cancelled runs can also hide a timeout, and the comment says it can't. .github/workflows/slack-on-failure.yaml:66-69

• The comment says "A run cancelled by its timeout has no newer run, so it is still posted". That holds only when nothing else started on the branch while the run hung.
• A push-triggered job that hits its timeout while a later merge's run is queued or running has a newer run with a higher run number, so it is suppressed. In a group without cancel-in-progress, the newer run waits as pending and is still listed.
• If that newer run passes, the hang is never reported. An intermittent hang is exactly the case that most needs reporting.
• Telling the two apart is hard, so at least correct the comment and the README "What is reported" bullet to state this as an accepted limitation, rather than claiming a timeout is always posted.

5. Maintainability (recorded state): configuration held elsewhere.

• README.md:136-137: "The organisation's secret posts to the organisation's channel". This describes org-level secret configuration. It goes stale if the org secret is removed or repointed, and the org's Actions secrets settings already answer it. The rule part (a repository secret overrides an org secret of the same name) is fine to keep.
• .github/workflows/code-review.yaml:5: "The same caller as every other repository's". This is a claim about other repositories' files, and it becomes untrue as soon as any one of them diverges. The other repositories' code-review.yaml holds the fact, and commit 411fb93 already records that this caller was copied from them.
• README.md:148-149 ("This repository's own watcher calls the workflow by local path…") and README.md:176-177 ("…reviews each pull request with the Claude review at the pull request's own commit"): both describe what this repository's own workflow files say, which report-failures.yaml and code-review.yaml already answer. The surrounding paragraph already lists CI this way, so this is low weight.
• Also, strictly, a pull_request run uses the caller's commit, which is the merge commit, not the pull request's head. "The pull request's own commit" is slightly off.

6. Preference.

• README.md:127 runs past the wrap width the rest of the file uses.
• check-repo-hygiene.py:561 evaluates body.get("with") or {} twice in one conditional. Binding it once would read more easily.

What I checked

• I read all five changed files in full, plus claude-review.yaml (to check what code-review.yaml passes and how its gate handles forks and bots), premerge.yaml (the watched workflow and its concurrency group), the README, and the commit messages.
• I traced the hygiene check logic against this repository's own workflows by reading:
  • premerge.yaml is push and is listed with push.
  • claude-review.yaml and slack-on-failure.yaml are workflow_call only.
  • code-review.yaml is pull_request/issue_comment.
  • On that reading it would pass here.
• I traced the code-review.yaml if:/depth expressions for /claude-review, /claude-review full, /claude-reviewer, /claude-review fullness and CRLF bodies. They behave as the comment says.
• The repository has no CLAUDE.md or AGENTS.md, so there were no instruction files to review against.

What I did not check

• I could not run the hygiene script or any workflow: the session only allows git diff/log/show.
• I did not check live GitHub behaviour: the head_branch value on tag pushes, the nested-permission error text, or the conclusion GitHub gives a timed-out job. Those findings rest on GitHub's documented behaviour, not on a run.

---

full review of 68eac351110e..411fb939e612. Run.

[GavinLucas]

Addressed in 4d2d875:

1. Tag pushes. Taken. For a push, head_branch is the branch or the tag's name, so the step now looks the name up with git/matching-refs/tags and compares exactly: the default branch or a tag is posted, another branch is not, and an unlistable tag list posts anyway with a warning. This needs contents: read, so the reporter's job and every watcher now ask for it. A push: branches: [release/*] workflow is still required to be listed and still not posted, by design: a push to a non-default branch has its pusher watching, which the README says.
2. types: missing. Taken. The job now requires github.event.action == 'completed', so a watcher's types: no longer matters. The hygiene check does not check it, since the reporter makes it irrelevant.
3. actions: read. Taken; the README said the opposite of what GitHub does. It now says the caller must grant actions: read and contents: read or the job is refused.
4. Timeout hidden by a queued newer run. Taken as an accepted limitation, stated in the step's comment and the README, since a replaced run and a timed-out one with a newer run queued cannot be told apart from the run data.
5. Recorded state. Taken for the organisation's channel (now the rule only), the "same caller as every other repository's" comment, and the watcher paragraph, which is removed. The "Changing anything here" sentence stays, as the reviewer weighed it, in the paragraph that already lists this repository's CI, reworded to "the pull request's own version of the Claude review", since a pull request run uses the merge commit.
6. Wrap and with. Both taken.

Tested against a fake Slack server with a stub gh: a push to the default branch, another branch, a tag, a name that only prefixes a tag, and an unlistable tag list. The live endpoint was queried against a repository with tags, to confirm the prefix matching the exact comparison relies on.

[github-actions]

Claude review: approve

Resolved

• Finding 1, tag pushes never posted: slack-on-failure.yaml:74-83 now treats a push whose head_branch is a tag as reportable. It checks the tag through matching-refs with an exact grep -Fx, and posts anyway if the tags cannot be listed. The README bullet and the header comment say the same.
• Finding 2, a post on every run start: the job's if: now requires github.event.action == 'completed', so a caller's types: no longer matters.
• Finding 3, the README's untrue "without it the post still goes": it now says the caller must grant actions: read and contents: read, or GitHub refuses to start the job. The caller example and report-failures.yaml grant both.
• Finding 4, the comment on timeouts and replaced runs: the step comment and a separate README bullet now state the miss as an accepted limitation.
• Finding 5, three of the recorded-state items:
  • The "organisation's secret posts to the organisation's channel" claim is replaced by the override rule.
  • The README line about this repository's own watcher is gone.
  • The "pull request's own commit" wording is corrected.
• Finding 6, both preference items: the long README line is rewrapped, and given = body.get("with") or {} is bound once.

Still open

• .github/workflows/code-review.yaml:5: "A caller like any other repository's" still claims something about other repositories' files, which will go stale when one diverges. Commit 411fb93 already records that this caller was copied from them. This is a maintainability finding, and low weight.

Declined

• None.

New

• Preference or possible bug, unverified. slack-on-failure.yaml:75: the tag lookup URL-encodes the whole name, so a tag such as release/1.0 is sent as release%2F1.0 inside the path. I am not sure GitHub accepts an encoded slash in that path.
  • If it answers 404, the step takes the "could not tell" branch and posts anyway. A slashed tag therefore still gets reported, with a spurious warning.
  • The same happens for every push to a non-default branch whose name contains a slash, such as feature/x. Those pushes would be posted when they should be skipped.
  • It only matters for a caller that lists push and has workflows running on non-default branches. A single real run would settle it. If it does 404, pass the name as unencoded path segments.
• Maintainability, low weight. slack-on-failure.yaml:75-76: a tag and a branch with the same name are indistinguishable from head_branch. A failed run on the branch would be posted as a tag. This is an edge case, and a one-line note would do.

What I checked

• I read the delta diff and the full reporter step, including the env and token wiring. GH_TOKEN and DEFAULT_BRANCH are set, and the permissions in the workflow, caller and README agree.
• I confirmed that the hygiene check treats a tag-triggered push workflow as push. That is now consistent with the reporter posting tag pushes, so the earlier "silent green" gap is closed.
• I confirmed the if: expression and the unattended_run_findings refactor are behaviour-preserving.

What I did not check

• I ran nothing. The head_branch value for tag runs, the encoded-slash behaviour of matching-refs, and the nested-permission refusal all rest on documented GitHub behaviour, not on a live run.

---

delta review of 411fb939e612..4d2d87548f66. Run.

[GavinLucas]

Second round, in the commit after 4d2d875:

• code-review.yaml:5. Taken: the comment now says only that this file calls the review by local path.
• Encoded slash in the tag lookup. Declined, on evidence: GitHub accepts it. Against docker-mcp's branch dev/claude-plugin-auto-publish, git/matching-refs/heads/dev%2Fclaude-plugin-auto-publish returns refs/heads/dev/claude-plugin-auto-publish, the same as the unencoded path, so a slashed branch is looked up correctly and skipped, and a slashed tag is found.
• A branch and a tag of the same name. Taken as a one-line note where the lookup is: such a push is posted.

[github-actions]

Claude review: approve

Resolved

• code-review.yaml:5: the comment no longer says anything about other repositories' callers. It now says only that the file calls the review by local path.
• Branch and tag with the same name (slack-on-failure.yaml:73-74): the lookup now carries a note that such a push is posted. The code does this already, because the exact grep -Fx finds the tag and the push is reported.

Still open

• None.

Declined

• Encoded slash in the tag lookup. @GavinLucas declined it with evidence: git/matching-refs/heads/dev%2Fclaude-plugin-auto-publish against docker-mcp returns the same ref as the unencoded path. The earlier review had only raised this as unverified. I accept the evidence.

New

• None. The delta is two comment edits. The new comment text states a rule and its reason, not recorded state. The code is unchanged.

What I checked

• I read the full delta diff. It touches only comments in code-review.yaml and slack-on-failure.yaml.

What I did not check

• I ran nothing. The changed comment describes behaviour I confirmed only by reading the code, not by a live run.

---

delta review of 4d2d87548f66..92d1521eb474. Run.

[github-actions]

Claude review: approve

Resolved

• None carried over. The previous review had no open findings.

Still open

• None.

Declined

• Encoded slash in the tag lookup. @GavinLucas declined it with evidence, and the previous review accepted it. Nothing in the new commits touches it.

New

• None. The delta adds a head-at-request-time gate to claude-review.yaml, with matching README text. I read it and found nothing that should block a merge.
  • Fail-safe paths. A failed activity lookup exits 1 and the review does not run. A missing timestamp or SHA refuses the request. If the latest push is not of the head the gate read, the request is refused. This also covers a push landing between gh pr view and the activity call, and an activity record that lags a new push.
  • Comparison. -ge refuses a push at the same second as the comment. That is the safe side, and the reply tells the requester to ask again.
  • Scope. The check runs only for issue_comment. Other triggers keep the old behaviour. That seems intended, since the concern is a push between a request and the run that starts from it.
  • Checkout. The review job now takes head from needs.gate.outputs.head and checks it against ^[0-9a-f]{40}$. Every later use reads steps.pr.outputs.head, namely the checkout ref, HEAD_SHA and the prompt diffs. headRefOid is no longer read in the review job, so the run cannot look up a different head.
  • Comments and README. The new text states rules and their reasons. It records no counts, dates or "currently" claims.

What I checked

• I read the delta diff and every remaining use of headRefOid and steps.pr.outputs in the workflow.

What I did not check

• I ran nothing.
• I did not verify that the repos/{repo}/activity endpoint returns newest-first for ref=refs/heads/... with per_page=1. I also did not verify that its timestamp and after fields behave as the code assumes. I judged both only by reading the code.

---

delta review of 92d1521eb474..15a5b15627d6. Run.