Skip to content

fix(ci): fail closed when Codemagic desktop preview dies after dispatch (#10145) - #11171

Merged
undivisible merged 6 commits into
BasedHardware:mainfrom
aryanorastar:fix/10145-preview-codemagic-observe
Aug 25, 2026
Merged

fix(ci): fail closed when Codemagic desktop preview dies after dispatch (#10145)#11171
undivisible merged 6 commits into
BasedHardware:mainfrom
aryanorastar:fix/10145-preview-codemagic-observe

Conversation

@aryanorastar

@aryanorastar aryanorastar commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Summary

Test plan

  • python3 .github/scripts/test_observe_codemagic_preview_build.py9 passed
  • check_desktop_preview_publishing()0 errors
  • CI release-process / checks-manifest lanes green
  • After merge: a dead Codemagic preview dispatch must fail the GitHub job (not silent green)

Failure-Class: none

Fixes #10145

Review in cubic

…ch (BasedHardware#10145)

Dispatch returning a buildId was treated as success while Codemagic could die
at startup with no artifact, so macos.omi.me/preview/<slug> silently fell back
to stable. Poll the exact build to a terminal status and keep durable evidence.

Co-authored-by: Cursor <cursoragent@cursor.com>
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 5 files

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread .github/workflows/desktop_publish_preview.yml
Comment thread .github/scripts/observe-codemagic-preview-build.py Outdated
…0145)

Reject non-alphanumeric build IDs before GITHUB_OUTPUT, pass observe args
via env (not ${{ }} in run) to avoid shell injection from provider data,
and drop the unused sys import for Ruff.

Co-authored-by: Cursor <cursoragent@cursor.com>
@aryanorastar

Copy link
Copy Markdown
Contributor Author

Addressed both cubic findings:

P1 — shell injection: dispatch now rejects build IDs outside ^[A-Za-z0-9_-]+$ before writing GITHUB_OUTPUT. The observe step takes CODEMAGIC_BUILD_ID / slug / SHA / app id via env: and "$VAR" args — no ${{ steps.dispatch.outputs.build_id }} inside run:.

P3 — Ruff: removed unused sys import from observe-codemagic-preview-build.py.

python3 .github/scripts/test_observe_codemagic_preview_build.py9 passed.

@Git-on-my-level Git-on-my-level added workflow-review Needs maintainer review for workflow, automation, hooks, or CI behavior needs-maintainer-review Needs a human maintainer to sign off before merge docs-tooling Layer: Documentation, examples, dev tools labels Aug 6, 2026
@Git-on-my-level

Copy link
Copy Markdown
Collaborator

Thanks — this looks like a useful fail-closed improvement for the desktop preview lane.

I reviewed the observer script, workflow wiring, release-process guard update, and the new behavioral tests. The important pieces look sound: the dispatcher now preserves the Codemagic build ID through GITHUB_OUTPUT, passes provider-controlled values via environment variables instead of interpolating them into shell syntax, polls the exact Codemagic build to a terminal status, treats failed/cancelled/timeout/unobservable states as failures, writes durable observation evidence, and registers the regression test in the checks manifest.

Validation I ran locally on the PR head:

  • python3 .github/scripts/test_observe_codemagic_preview_build.py — 9 tests passed
  • PYTHONPATH=.github/scripts python3 .github/scripts/check-release-process-guards.py — passed
  • git diff --check origin/main...HEAD -- .github/scripts/observe-codemagic-preview-build.py .github/workflows/desktop_publish_preview.yml .github/scripts/test_observe_codemagic_preview_build.py .github/scripts/check-release-process-guards.py .github/checks-manifest.yaml — passed

I’m not formally approving because this touches the GitHub/Codemagic preview publishing path and should get a maintainer pass before merge, but my technical read is positive. One small follow-up for the human maintainer to consider: if Codemagic’s single-build endpoint can ever omit workflow/env metadata, the observer currently treats those identity fields as best-effort rather than required; that seems acceptable for the exact build ID returned by dispatch, but it is worth confirming against the live API shape.


by AI on behalf of David — if you need David’s attention urgently, please @Git-on-my-level and escalate with need human response.

@aryanorastar

Copy link
Copy Markdown
Contributor Author

@Git-on-my-level need human response — code-owner APPROVE.

Author-side clear: CI green, MERGEABLE, cubic P1/P3 addressed @a1b34804. Your technical read already covers the fail-closed observer + env-passed build IDs + release-process/manifest wiring.

Branch protection is specifically waiting on code owner review from you (.github/workflows/**). Could you leave a formal APPROVE when ready?

Noted for maintainers: observer treats Codemagic workflow/env identity fields as best-effort if the single-build endpoint omits them — acceptable for the exact buildId from dispatch; live API shape confirm can be a follow-up. Secret provisioning stays on #10221.

@Git-on-my-level

Copy link
Copy Markdown
Collaborator

Thanks for the ping. I re-checked the current head and the workflow-side fix still looks technically sound: the Codemagic build ID is validated before entering GITHUB_OUTPUT, the observer receives provider-controlled values through env/quoted shell arguments, and the observer/test/guard wiring is covered by the new manifest lane.

I also reran the local validation on a1b3480419fb78d8627be96787fe082c9d84f38b:

  • python3 .github/scripts/test_observe_codemagic_preview_build.py — 9 tests passed
  • PYTHONPATH=.github/scripts python3 .github/scripts/check-release-process-guards.py — passed
  • git diff --check origin/main...HEAD -- .github/scripts/observe-codemagic-preview-build.py .github/workflows/desktop_publish_preview.yml .github/scripts/test_observe_codemagic_preview_build.py .github/scripts/check-release-process-guards.py .github/checks-manifest.yaml — passed

I’m still not leaving a formal code-owner approval from automation because this changes the GitHub/Codemagic preview publishing lane and should get an explicit human maintainer/code-owner sign-off before merge. My maintainer signal remains positive pending that human review.


by AI on behalf of David — if you need David’s attention urgently, please @Git-on-my-level and escalate with need human response.

@aryanorastar

Copy link
Copy Markdown
Contributor Author

@Git-on-my-level final approval reminder: PR #11171 is fully CI-green at a1b3480. Your latest technical re-review reran 9 tests, release-process guards, and diff checks with no blocker; the Codemagic build-ID validation and fail-closed observer wiring are sound. The only remaining gate is formal workflow code-owner approval; #10221 secret provisioning remains separate maintainer ops.

Keep Codemagic preview fail-closed observer tip current for workflow CO.
@aryanorastar

Copy link
Copy Markdown
Contributor Author

@Git-on-my-level need human response — formal workflow/code-owner APPROVE is the only merge blocker on #11171.

Author-clear:

  • Your latest AI re-review on a1b3480419: 9 tests + release-process guards + diff check; positive signal, no code blocker
  • Tip refreshed onto current main @09b0f3f76dbe (was ~335 behind; MERGEABLE; prior CI 20 green)
  • Fail-closed Codemagic observe after dispatch; build-ID validation; env-passed provider values; Failure-Class: none
  • Secret provisioning stays on Restore least-privilege credentials for external macOS Preview publishing #10221 (separate maintainer ops)

Please human / workflow-review APPROVE (or explicit hold) when you can. Branch protection is waiting on code owner review from you (.github/workflows/**).

@aryanorastar

Copy link
Copy Markdown
Contributor Author

@Git-on-my-level need human response — formal workflow/code-owner APPROVE is still the only merge blocker on #11171.

Author-clear on tip 0e23b69d1c9c (refreshed onto current main; prior tip was ~63 behind):

  • Fail-closed Codemagic observe after dispatch; build-ID validation; env-passed provider values
  • Local re-check after main refresh: observer suite 9 passed · release-process guards passed
  • Your AI passes: positive / no code blocker; Cubic P1/P3 addressed
  • Failure-Class: none
  • Secret provisioning stays on Restore least-privilege credentials for external macOS Preview publishing #10221 (separate maintainer ops)

CI re-running on the main merge. Please human / workflow-review APPROVE (or explicit hold) when you can — branch protection is waiting on code owner review from you (.github/workflows/**).

@Git-on-my-level

Copy link
Copy Markdown
Collaborator

Thanks — I re-reviewed the current head after the main-branch refresh. I still do not see a code blocker in the PR-specific changes, but this remains a workflow-sensitive desktop preview publishing path, so I’m leaving final workflow/code-owner sign-off to a human maintainer rather than formal approval.

File-specific notes:

  • .github/scripts/observe-codemagic-preview-build.py: the observer writes durable JSON evidence before polling, redacts token-shaped errors, validates the app/build/slug/SHA inputs, checks workflow/preview identity when Codemagic exposes it, and fails closed for failed/cancelled/timeout/unobservable states. The exact-build GET path avoids list matching ambiguity for the dispatched build ID.
  • .github/workflows/desktop_publish_preview.yml: the dispatch step now validates build_id before putting it in GITHUB_OUTPUT, then passes Codemagic-provided values through env/quoted CLI args instead of interpolating them directly into shell syntax. The follow-up observer step makes dispatch success insufficient by itself, and the always() artifact upload preserves the observation file when dispatch succeeded.
  • .github/scripts/test_observe_codemagic_preview_build.py: coverage exercises success, failed/cancelled terminal states, identity mismatch, timeout evidence, missing token behavior, exact GET request wiring, and workflow/evidence-retention wiring.
  • .github/scripts/check-release-process-guards.py: the protected preview dispatcher guard now requires the observer script, terminal-status observation step, evidence-retention step, and existence of the observer script, which should catch future regressions in the manual preview lane.
  • .github/checks-manifest.yaml: the new desktop-codemagic-preview-build-observer manifest entry wires the observer test into both local and CI lanes with triggers covering the script, tests, workflow, and manifest.

Validation I ran locally on this head:

  • python3 .github/scripts/test_observe_codemagic_preview_build.py — 9 tests passed
  • python3 .github/scripts/check-release-process-guards.py — passed
  • git diff --check origin/main...HEAD — passed
  • Manifest YAML parsed and contains one desktop-codemagic-preview-build-observer entry

Final maintainer note: the implementation looks like a useful fail-closed improvement for #10145, but because it changes CI/release-preview behavior and .github/workflows/**, human workflow/code-owner approval should decide merge readiness.

— Automated maintainer review


by AI on behalf of David — if you need David’s attention urgently, please @Git-on-my-level and escalate with need human response.

@aryanorastar

Copy link
Copy Markdown
Contributor Author

@Git-on-my-level need human response — formal workflow/code-owner APPROVE is still the only merge blocker on #11171.

Author-clear on tip 2b29e2aae351 (refreshed onto current main):

  • Your latest AI pass (15m): no code blocker — observer fail-closed wiring, build-ID validation, env-passed provider values, release-process + manifest guards all look sound (9 tests + guards green)
  • Local re-check after this refresh: observer suite 9 passed · release-process guards passed
  • Failure-Class: none
  • Secret provisioning stays on Restore least-privilege credentials for external macOS Preview publishing #10221 (separate maintainer ops)

Please human / workflow-review APPROVE (or explicit hold) when you can — branch protection is waiting on code owner review from you (.github/workflows/**).

@undivisible undivisible added the human Human-authored pull request label Aug 10, 2026
@Git-on-my-level

Copy link
Copy Markdown
Collaborator

Thanks for the refresh. I reviewed the current head (2b29e2aae3518518712bede2b30679fa6ca33def) and still do not see a PR-specific code blocker. This is a workflow-sensitive desktop preview publishing change, so I’m leaving merge readiness to human workflow/code-owner sign-off rather than formal approval.

File-specific notes:

  • .github/scripts/observe-codemagic-preview-build.py: input validation is tight for the Codemagic app ID, build ID, preview slug, and source SHA; the observer writes initial/durable JSON evidence before polling; token-shaped API errors are redacted; the exact dispatched build is fetched by ID; workflow/preview identity mismatches fail closed when Codemagic exposes those fields; terminal failed/cancelled/timeout/unobservable states return non-zero.
  • .github/workflows/desktop_publish_preview.yml: the dispatch step validates the returned build_id before writing GITHUB_OUTPUT, and the observe step passes Codemagic/provider-controlled values through environment variables plus quoted CLI arguments rather than embedding them directly into shell syntax. Dispatch success is no longer treated as preview success, and the observation artifact is retained with always() after successful dispatch.
  • .github/scripts/test_observe_codemagic_preview_build.py: the tests cover success, failed/cancelled terminal states, workflow and slug identity mismatches, timeout evidence, missing-token behavior, exact GET request construction, and workflow/evidence-retention wiring.
  • .github/scripts/check-release-process-guards.py: the preview-publishing guard now requires the observer script, the terminal-status observation step, evidence-retention wording, and the observer file’s presence, which should catch accidental removal from the protected manual preview lane.
  • .github/checks-manifest.yaml: the new desktop-codemagic-preview-build-observer entry wires the observer tests into local/CI coverage and includes triggers for the script, tests, workflow, and manifest.

Validation I ran on this head:

  • python3 -m py_compile for the observer and its test file — passed
  • python3 .github/scripts/test_observe_codemagic_preview_build.py — 9 tests passed

Maintainer note: the technical direction looks like the right fail-closed fix for #10145, but because it changes the GitHub/Codemagic desktop preview publishing lane under .github/workflows/**, human workflow/code-owner approval should decide final merge readiness.

— Automated maintainer review


by AI on behalf of David — for human maintainer review because this changes workflow-controlled desktop preview publishing.

@aryanorastar

Copy link
Copy Markdown
Contributor Author

@Git-on-my-level final merge-gate ping for #11171. Current head 2b29e2aae3 is MERGEABLE and all required checks are green; your latest technical review found no PR-specific blocker. The only remaining gate is formal workflow/code-owner approval for .github/workflows/**. #10221 Codemagic secret provisioning remains separate maintainer ops. Please approve when ready, or state an explicit hold/blocker so I can address it.

@aryanorastar

aryanorastar commented Aug 17, 2026

Copy link
Copy Markdown
Contributor Author

@Git-on-my-level @kodjima33 Workflow review ping: tip 2b29e2aa is MERGEABLE with 30/30 checks green. Fails closed when Codemagic desktop preview dies after dispatch (#10145). Ready for maintainer review/merge.


Still current and still author-side clear. Verified against main on 2026-08-23: not superseded, still MERGEABLE, all checks green, no conflicts. Blocker is the CODEOWNERS workflow-review routing. Grouped with six sibling CI PRs in one consolidated ping at #10959 rather than bumping each thread separately.

…emagic-observe

# Conflicts:
#	.github/checks-manifest.yaml
@undivisible
undivisible merged commit 0d22722 into BasedHardware:main Aug 25, 2026
30 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs-tooling Layer: Documentation, examples, dev tools human Human-authored pull request needs-maintainer-review Needs a human maintainer to sign off before merge workflow-review Needs maintainer review for workflow, automation, hooks, or CI behavior

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Desktop preview lane: dispatched Codemagic builds die silently, previews never publish

3 participants