Repository navigation
Cleanup PR artifacts fails while inventorying organization packages #103
Description
The reusable cleanup workflow fails before producing a cleanup plan because its organization package inventory request receives HTTP 403. This makes otherwise-green pull requests fail their cleanup check and causes downstream release gates to refuse publication.
The workflow should either receive an explicitly validated package-read capability or report a clear could-not-run verdict that downstream repositories can distinguish from a product verification failure. The fix belongs in the shared workflow rather than in each caller.
Status
State: Done
Scope: Cleanup workflow reports a clear verdict and uses an explicit package-cleanup token instead of failing with an opaque 403.
Delivered: - #145 (merged)
Remaining: None. Optional: an org admin can add a classic PAT as org secret PACKAGE_CLEANUP_TOKEN (read:packages + delete:packages, SSO-authorized) to make cleanup actually delete PR package versions.
Follow-ups: None
Activity
woksin commentedon Sep 25, 2026
Still failing as of 2026-09-25 with the same
API GET failed (HTTP 403)on the first organization package inventory call. I checked the five most recent runs of Cleanup PR Artifacts in Chronicle, Arc, Fundamentals and Components; all of them failed. None of the last 60 Chronicle runs succeeded. The callers passsecrets: inherit, and the reusable workflow authenticates withPAT_WORKFLOWS, so the token's package-read access (scope, expiry or SSO authorization) is the first thing to check.Example: https://github.com/Cratis/Chronicle/actions/runs/36160957512 was triggered when Cratis/Chronicle#4186 was merged. The failure did not block the merge there, because that pull request was labeled
no-release.cratis-direct commentedon Sep 30, 2026
Still failing as of 2026-09-30. Confirmed on the most recent run in each of Ante, Chronicle, Fundamentals and Workflows itself — no run of this workflow has succeeded in any of them.
Adding the exact call, which the report and the comment above don't pin down: the 403 comes from
GET https://api.github.com/orgs/Cratis/packages?package_type=container, the firstapi.listing(...)in the inventory step (line 224 ofcleanup-pr-artifacts.yml, theorgs/Cratis/packages?package_type={kind}call). A 403 rather than a 401 meansPAT_WORKFLOWSauthenticates but is not authorized to list organization packages — consistent with the missingread:packagesscope or missing org SSO authorization already suggested above.Latest example, triggered when Cratis/Ante#141 was merged: https://github.com/Cratis/Ante/actions/runs/36694143625. That pull request merged despite the red check, so the failure is noise today rather than a block — but it is a guard that cannot pass, and every closed pull request across the organization produces a red X for it.
woksin commentedon Oct 1, 2026
Confirmed again on Arc run https://github.com/Cratis/Arc/actions/runs/36858996904: the first GET to orgs/Cratis/packages?package_type=container returns 403, before any cleanup plan or deletion. The merged #127 bootstrap run authenticates PAT_WORKFLOWS as cratis-bot and explicitly reports a fine-grained PAT. The organization Packages REST inventory endpoints require a classic PAT; workflow permission changes cannot add package scopes to that fine-grained credential. Admin action: provide a dedicated classic package-cleanup PAT with read:packages and delete:packages, the required repository/package access, package-admin rights for deletion, and organization SSO authorization if enforced. Prefer a separate cleanup-only secret over broadening the organization-wide workflow-write token. Preserve the fail-closed inventory and deletion checks. No secrets or package permissions were changed, and no packages were deleted.
cratis-direct commentedon Oct 3, 2026
Still broken as of 2026-10-03, reproduced from a new caller/run:
chore/cratis-ai-update-e7dc6fe, closed 2026-09-29): sameAPI GET failed (HTTP 403)on the firstorgs/Cratis/packages?package_type=containerinventory call, before any cleanup plan is produced.This confirms @woksin's diagnosis still holds:
PAT_WORKFLOWSis a fine-grained PAT, and the organization Packages REST inventory endpoints (GET /orgs/Cratis/packages) reject fine-grained PATs for org-wide listing — nothing in the caller workflows or the reusable workflow's own logic is at fault, and no caller-side change can fix this. I did not find anything actionable to change in thecleanup-pr-artifacts.ymllogic itself (the fail-closed 403 handling is working as designed — it fails loudly rather than silently skipping cleanup).Leaving this with @woksin per the existing assignment and recommended remediation (a dedicated classic PAT with
read:packages/delete:packagesscope, package-admin rights, and org SSO authorization if enforced — provisioned as a separate cleanup-only secret rather than broadening the workflow-write token). I don't have access to create or rotate that credential.woksin commentedon Oct 5, 2026
Code side merged in #145. Remaining: an org admin must create a classic PAT (read:packages + delete:packages, package admin, SSO-authorized) and store it as org secret PACKAGE_CLEANUP_TOKEN; cleanup then runs automatically for callers using secrets: inherit. Leaving this open until that is done.
cratis-direct commentedon Oct 7, 2026
Confirmed again from Cratis/homebrew-cratis: run 36643681300 (PR #14,
chore/cratis-ai-update-e7dc6fe, closed 2026-09-29) failed with the sameAPI GET failed (HTTP 403)on the firstorgs/Cratis/packages?package_type=containerinventory call, before any cleanup plan was produced.I checked
Cratis/homebrew-cratis/.github/workflows/cleanup-pr-artifacts.yml: it only calls the shared reusable workflow withsecrets: inheritand has no caller-side logic to change. Nothing actionable there, consistent with the prior findings in this thread.I also confirmed the current fix in
Cratis/Workflows@main(merged via #145) can't be exercised viaworkflow_dispatchby an agent: dispatching a dry run against this repo (target_repository: Cratis/homebrew-cratis,pull_request: 14) sat inwaitingpending thepr-package-cleanupenvironment's required reviewer, which is the correct gate — I cancelled that test run rather than pushing it through. So this remains blocked on the org admin creating the classic PAT and storing it asPACKAGE_CLEANUP_TOKEN, as already noted.cratis-direct commentedon Oct 7, 2026
Confirmed again, 2026-10-07: Cratis/Lens run 36643689144 (PR #45,
chore/cratis-ai-update-e7dc6fe, closed 2026-09-29) — sameAPI GET failed (HTTP 403)signature on the firstorgs/Cratis/packages?package_type=containerinventory call (job logs have since expired; annotations still show it at::error::API GET failed (HTTP 403)immediately beforeProcess completed with exit code 1).Lens's own caller workflow (
.github/workflows/cleanup-pr-artifacts.yml) is a minimal, correctuses:/secrets: inheritreusable-workflow call — nothing to change there, and nothing actionable in the reusable workflow's logic either (the fail-closed 403 handling and the already-merged #145 fix are working as designed). Leaving this open per the existing remaining action: an org admin provisioning the classicPACKAGE_CLEANUP_TOKENPAT.cratis-direct commentedon Oct 7, 2026
Independently triaged the failing run in Cratis/cratis.studio (https://github.com/Cratis/cratis.studio/actions/runs/36900456979) that led here and arrived at the same diagnosis: the hard
API GET failed (HTTP 403)onorgs/Cratis/packagescame fromPAT_WORKFLOWSlacking package-read authorization, not from cratis.studio's own workflow (which is the unmodified 12-line wrapper callingCratis/Workflows/.github/workflows/cleanup-pr-artifacts.yml@main).Re-ran that same job just now (
gh run rerun 36900456979 --repo Cratis/cratis.studio) to confirm #145 fixed the hard failure: it now completes withconclusion: successand the graceful annotationCould not run: package inventory not authorized. No versions deleted. ..., instead of failing CI. So no caller-side change is needed in cratis.studio — the only remaining action is the org admin step already tracked here (create/storePACKAGE_CLEANUP_TOKEN).No new issue opened; just adding corroboration since I was dispatched to fix the cratis.studio-side failure independently.
cratis-direct commentedon Oct 7, 2026
Confirmed again from Cratis/Workshops: run 36643834802 (PR #13,
chore/cratis-ai-update-e7dc6fe, closed 2026-09-29) originally failed with the sameAPI GET failed (HTTP 403)signature on the firstorgs/Cratis/packages?package_type=containerinventory call (job logs have since expired; annotations still showed it immediately beforeProcess completed with exit code 1).Workshops's own caller workflow (
.github/workflows/cleanup-pr-artifacts.yml) is the same minimal, correctuses:/secrets: inheritwrapper seen in every other affected repo — nothing to change there.Re-ran the same run (
gh run rerun 36643834802 --repo Cratis/Workshops) to check whether #145 fixed the hard failure, same as was done for cratis.studio: it now completes withconclusion: successand the graceful annotationCould not run: package inventory not authorized. No versions deleted. ..., instead of failing CI. So no caller-side change is needed in Workshops either — the only remaining action is the org admin step already tracked here (create/storePACKAGE_CLEANUP_TOKEN).No new issue opened; adding corroboration since I was dispatched to fix the Workshops-side failure independently.
cratis-direct commentedon Oct 7, 2026
Triaged the originally reported failing run in Cratis/Scene (https://github.com/Cratis/Scene/actions/runs/36787107309, PR #57
fix/40-exclude-specs-from-package, closed 2026-09-30) and arrived at the same diagnosis already tracked here: the hardAPI GET failed (HTTP 403)came fromPAT_WORKFLOWSlacking package-read authorization on the firstorgs/Cratis/packages?package_type=containerinventory call, not from Scene's own workflow (which is the unmodified 12-line wrapper callingCratis/Workflows/.github/workflows/cleanup-pr-artifacts.yml@mainwithsecrets: inherit).Re-ran the job (
gh run rerun 36787107309 --repo Cratis/Scene) to confirm #145 fixed the hard failure: it now completes withconclusion: successand the graceful annotationCould not run: package inventory not authorized. No versions deleted. ..., instead of failing CI.No caller-side change needed in Scene, and no new issue opened — adding corroboration since I was dispatched to fix the Scene-side failure independently. Remaining action is still the org admin step already tracked here (create/store
PACKAGE_CLEANUP_TOKEN).cratis-direct commentedon Oct 7, 2026
Triaged the originally reported failing run in Cratis/Architecture (https://github.com/Cratis/Architecture/actions/runs/36901525655, PR #35, closed 2026-10-01) and arrived at the same diagnosis already tracked here: the hard
API GET failed (HTTP 403)came fromPAT_WORKFLOWSlacking package-read authorization on the firstorgs/Cratis/packages?package_type=containerinventory call, not from Architecture's own workflow (which is the unmodified 12-line wrapper callingCratis/Workflows/.github/workflows/cleanup-pr-artifacts.yml@mainwithsecrets: inherit).Re-ran the job (
gh run rerun 36901525655 --repo Cratis/Architecture) to confirm #145 fixed the hard failure: it now completes withconclusion: successand the graceful annotationCould not run: package inventory not authorized. No versions deleted. ..., instead of failing CI.No caller-side change needed in Architecture, and no new issue opened — adding corroboration since I was dispatched to fix the Architecture-side failure independently. Remaining action is still the org admin step already tracked here (create/store
PACKAGE_CLEANUP_TOKEN).cratis-direct commentedon Oct 7, 2026
Triaged the originally reported failing run in Cratis/Chronicle.Kotlin (https://github.com/Cratis/Chronicle.Kotlin/actions/runs/36845894265, PR #120
fix/bcl-guid-mixed-endian, closed 2026-10-01) and arrived at the same diagnosis already tracked here: the hardAPI GET failed (HTTP 403)came fromPAT_WORKFLOWSlacking package-read authorization on the firstorgs/Cratis/packages?package_type=containerinventory call, not from Chronicle.Kotlin's own workflow (which is the unmodified 3-line wrapper callingCratis/Workflows/.github/workflows/cleanup-pr-artifacts.yml@mainwithsecrets: inherit).Re-ran the job (
gh run rerun 36845894265 --repo Cratis/Chronicle.Kotlin) to confirm #145 fixed the hard failure: it now completes withconclusion: successand the graceful annotationCould not run: package inventory not authorized. No versions deleted. ..., instead of failing CI.No caller-side change needed in Chronicle.Kotlin, and no new issue opened — adding corroboration since I was dispatched to fix the Chronicle.Kotlin-side failure independently. Remaining action is still the org admin step already tracked here (create/store
PACKAGE_CLEANUP_TOKEN).cratis-direct commentedon Oct 7, 2026
Triaged the originally reported failing run in Cratis/Arc.Kotlin (https://github.com/Cratis/Arc.Kotlin/actions/runs/36901680411, PR #272, closed 2026-10-01) and arrived at the same diagnosis already tracked here: the hard
API GET failed (HTTP 403)came fromPAT_WORKFLOWSlacking package-read authorization on the firstorgs/Cratis/packages?package_type=containerinventory call, not from Arc.Kotlin's own workflow (which is the unmodified 12-line wrapper callingCratis/Workflows/.github/workflows/cleanup-pr-artifacts.yml@mainwithsecrets: inherit).Re-ran the job (
gh run rerun 36901680411 --repo Cratis/Arc.Kotlin) to confirm #145 fixed the hard failure: it now completes withconclusion: successand the graceful annotationCould not run: package inventory not authorized. No versions deleted. ..., instead of failing CI.No caller-side change needed in Arc.Kotlin, and no new issue opened — adding corroboration since I was dispatched to fix the Arc.Kotlin-side failure independently. Remaining action is still the org admin step already tracked here (create/store
PACKAGE_CLEANUP_TOKEN).woksin commentedon Oct 9, 2026
Closing: the scope is delivered by #145. The cleanup workflow now reports a clear "could not run" verdict instead of failing with an opaque 403, so it no longer turns green PRs red or blocks releases (recent Cleanup PR Artifacts runs pass, e.g. Cratis/Arc run 37928398156). Actually deleting PR package versions is optional: it needs an org admin to add a classic PAT as org secret
PACKAGE_CLEANUP_TOKEN(read:packages + delete:packages, SSO-authorized), and then it works without further code changes.