Skip to content

fix(ci): ensure the break-glass label exists before recording the bypass - #10527

Closed
aryanorastar wants to merge 3 commits into
BasedHardware:mainfrom
aryanorastar:fix/10389-break-glass-audit-resilient-label
Closed

fix(ci): ensure the break-glass label exists before recording the bypass#10527
aryanorastar wants to merge 3 commits into
BasedHardware:mainfrom
aryanorastar:fix/10389-break-glass-audit-resilient-label

Conversation

@aryanorastar

@aryanorastar aryanorastar commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

What

Closes #10389.

The Release Eligibility break-glass path records a prod bypass by opening a GitHub issue with --label release-gate-failure. Under set -euo pipefail, when that label didn't exist the labeled gh issue create failed the whole step — so a real prod Cloud Run bypass (e262b2323b) went unrecorded and had to be reconstructed by hand as #10301.

Fix

The #10163 guard requires the tracking issue to carry that label, so dropping it (or an unlabeled fallback) isn't an option. Instead, create the label first so the labeled issue-create always succeeds:

gh label create release-gate-failure --repo "$REPO" --color B60205 \
  --description "Release Eligibility gate was bypassed or failed; needs follow-up" \
  --force >/dev/null 2>&1 || true
gh issue create --repo "$REPO" ... --label release-gate-failure ...
  • --force is idempotent (creates if missing, refreshes if present); issues: write already covers label creation.
  • || true tolerates only a transient labels-API blip — not a missing label, which the create now guarantees away.
  • Applied to both gcp_backend and gcp_llm_gateway break-glass jobs.
  • No repo checkout added — that would violate the refactor(release): simplify desktop distribution channels #10163 direct-production-admission guard (which is why this stays inline rather than extracting a script).

Also satisfies the issue's "label should exist" acceptance — it's created on the next break-glass run.

Guard update

Because the label name now also appears on the gh label create line, the #10163 checker's "release-gate-failure" fragment would pass even if the issue lost its label. Tightened it to --label release-gate-failure so it still verifies the label is on the tracking issue, not merely present in the job.

Verification

test_check_direct_backend_production_admission.py → 5 passed
  (the mutation that strips the issue's --label is still caught by the tightened fragment)
check-direct-backend-production-admission.py → exit 0
test_check_backend_deploy_source_admission.py → green
both workflows → valid YAML

Failure-Class: none

Review in cubic

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@Git-on-my-level

Copy link
Copy Markdown
Collaborator

Thanks for the focused fix here. The change is directionally right to me: creating release-gate-failure before the labeled gh issue create preserves the audit requirement instead of falling back to an unlabeled issue, and tightening the static checker to look for --label release-gate-failure avoids a false pass from the new label-create line.

I also did a local static/targeted verification pass on the PR head (16ce0abc59b511374ef03f31cafc1e32aa64f90c):

  • python3 .github/scripts/test_check_direct_backend_production_admission.py passed (5 tests)
  • python3 .github/scripts/test_check_backend_deploy_source_admission.py passed (28 tests)
  • python3 .github/scripts/check-direct-backend-production-admission.py passed
  • both edited workflow YAML files parsed successfully

I’m not formally approving because this changes production deploy/break-glass workflow behavior and the current check summary is not fully green (Hygiene/Formatting are not passing in the captured status). Human maintainer review should sign off before merge, but I don’t see a code-level blocker from this pass.


by AI on behalf of David — if you need David’s attention urgently, please @Git-on-my-level and escalate with need human response.

@Git-on-my-level Git-on-my-level added workflow-review Needs maintainer review for workflow, automation, hooks, or CI behavior needs-maintainer-review Needs a human maintainer to sign off before merge labels Jul 25, 2026
@aryanorastar

Copy link
Copy Markdown
Contributor Author

Thanks for the verification pass. One correction on the status blocker: nothing on this PR is failing. Hygiene and Formatting show as CANCELLED, not red — they were superseded by a later run on the same head, which is why a captured status reads them as not-passing. Current rollup on 16ce0ab is 12 SUCCESS / 23 SKIPPED / 1 NEUTRAL / 6 CANCELLED, 0 FAILURE, and GitHub now reports "All checks have passed".

Since the two cancelled lanes never completed here, I ran them locally against that exact head so it isn't taken on trust:

  • check-release-process-guards.py (the repo-wide check the Hygiene lane runs) — 0 errors
  • test_check_direct_backend_production_admission.py — 5 passed; check-direct-backend-production-admission.py — exit 0
  • test_check_backend_deploy_source_admission.py — 28 passed
  • black --line-length 120 --skip-string-normalization --check on the one changed Python file — unchanged (the Formatting lane's scope; the other two changed files are YAML)
  • both workflows parse under yaml.safe_load

So the remaining gate is just the code-owner sign-off you flagged, which is the right call given this touches the production break-glass path — not a CI problem. Head is unchanged at 16ce0ab, so your earlier verification still stands.

@aryanorastar

Copy link
Copy Markdown
Contributor Author

@Git-on-my-level need human response — this PR is mergeable (no conflicts, checks green) and has been sitting in needs-maintainer-review / workflow-review. Could a maintainer give it the human sign-off, or let me know if anything is blocking?

Summary: this PR is part of a batch that was just cleaned up (rebased onto current main, conflicts resolved, superseded PRs closed). All remaining open PRs are mergeable and waiting on review. Please review/merge or request changes.

aryanorastar and others added 3 commits August 1, 2026 10:25
…udit issue (BasedHardware#10389)

Both break-glass audit jobs ran gh issue create --label release-gate-failure
with no guarantee the label exists in the repo. GitHub rejects issue creation
with an unknown label, so the audit issue was never filed (the prod Cloud Run
bypass in BasedHardware#10301 was preserved manually). Ensure the label exists first with
an idempotent gh label create, then file the issue — the audit trail can no
longer be dropped by a missing label.

Failure-Class: none

Co-authored-by: CommandCodeBot <noreply@commandcode.ai>
…ission check

The gateway validator required the bare `release-gate-failure` string
anywhere in gcp_llm_gateway.yml. The new `gh label create
release-gate-failure` step (and its comment) satisfy that weak fragment,
so a workflow that dropped `--label release-gate-failure` from the audit
issue would pass the check. Tighten the fragment to the exact
`--label release-gate-failure` flag on the issue-create command, which
is the property the check exists to pin.

Co-authored-by: CommandCodeBot <noreply@commandcode.ai>
…sured

The idempotent `gh label create ... 2>/dev/null || true` swallowed every
failure — permissions, rate limits, API errors — not just the tolerated
"already exists" case, so a real label-create failure would still drop
the audit issue silently. Capture the output and only suppress the exact
"already exists" message; any other failure now fails the job loudly
before the audit issue is attempted.

Co-authored-by: CommandCodeBot <noreply@commandcode.ai>
@aryanorastar
aryanorastar force-pushed the fix/10389-break-glass-audit-resilient-label branch from 16ce0ab to 2f44b69 Compare August 2, 2026 06:13
@aryanorastar

Copy link
Copy Markdown
Contributor Author

Head updated with the full 3-commit series on fix/10389-break-glass-audit-resilient-label (forced update, same base 3fcce0fd3):

  1. f5cdbe8f9 — ensure the release-gate-failure label exists before filing the audit issue
  2. 27859652b — tighten the static checker to --label release-gate-failure (closes the false-positive from the new gh label create line)
  3. 2f44b6926 — fail loudly on label-create errors other than "already exists" (replaces the broad || true)

CI status is green on the tip: check-direct-backend-production-admission.py exits 0, test_check_direct_backend_production_admission.py 5/5, test_check_backend_deploy_source_admission.py 28/28, both workflows validate as YAML. @Git-on-my-level for the human sign-off this production break-glass path needs.

@aryanorastar

Copy link
Copy Markdown
Contributor Author

@Git-on-my-level gentle ping — note that #10966 now carries the refined label_output approach (already reviewed) which supersedes this alternative --force implementation. Worth closing #10527 if #10966 is the path forward.

@Git-on-my-level Git-on-my-level added docs-tooling Layer: Documentation, examples, dev tools superseded Superseded by a newer change on main labels Aug 2, 2026
@Git-on-my-level

Copy link
Copy Markdown
Collaborator

Thanks for the focused work on the break-glass audit path here. I’m going to close this PR as superseded by #10966, which carries the same fix direction but with the refined label-create handling: it only tolerates the explicit “already exists” case and fails loudly on other label creation/API/permission failures before the audit issue is filed.

That is the safer path for this production workflow surface. #10966 should remain under maintainer workflow review before merge.


by AI on behalf of David — if you need David’s attention urgently, please @Git-on-my-level.

@github-actions

github-actions Bot commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

Hey @aryanorastar 👋

Thank you so much for taking the time to contribute to Omi! We truly appreciate you putting in the effort to submit this pull request.

After careful review, we've decided not to merge this particular PR. Please don't take this personally — we genuinely try to merge as many contributions as possible, but sometimes we have to make tough calls based on:

  • Project standards — Ensuring consistency across the codebase
  • User needs — Making sure changes align with what our users need
  • Code best practices — Maintaining code quality and maintainability
  • Project direction — Keeping aligned with our product principles and locked invariants

Before your next PR, please skim:

  • PRODUCT.md — product north star
  • Product invariants — locked rules (shared chat, memory tiers, agent control plane, integrations, brand)

If this was declined for direction or taste, maintainers should cite an invariant ID or open a proposed one — ask if that citation is missing.

Your contribution is still valuable to us, and we'd love to see you contribute again in the future! If you'd like feedback on how to improve this PR or want to discuss alternative approaches, please don't hesitate to reach out.

Thank you for being part of the Omi community!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs-tooling Layer: Documentation, examples, dev tools needs-maintainer-review Needs a human maintainer to sign off before merge superseded Superseded by a newer change on main workflow-review Needs maintainer review for workflow, automation, hooks, or CI behavior

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix: create release-gate-failure label (break-glass audit job fails without it)

2 participants