release-train: develop -> staging - #1030
Merged
Merged
Conversation
…printed here-documents as text (#1022) * test(installer): copy guard shares one lexer, honours escapes, reads printed here-documents as text One AWK_LEX walker for the derivation, the here-document lister and the comment stripper (they held three copies). Escaped quotes (bash backslash, PowerShell backtick and doubled quote) no longer move the brace depth. A printed here-document is scanned in full, a here-document that generates a file keeps its # lines as comments; body lines are never also code lines, so an offender is counted once. RFC numbers of three or more digits. The shared closer rule also un-hides eleven PowerShell helpers that a "@.Trim() closer had swallowed from the vocabulary. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * test(installer): copy guard fails closed on a missing scratch dir; only stdout-to-file redirects make a here-document a generated file mktemp gets a template and a guard error, so the EXIT cleanup can never expand to the root. A here-document with 2>, >&2 or a /dev/ target stays printed text; only >, >> or 1> to a real file classifies it as a generated file whose # lines are comments. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * test(installer): the one-lexer census counts the closer rule too heredoc_delim and closes join the census, and the closer regex may be spelled only inside closes(); the closer was the half that had drifted. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
…he workload to :tag (#1008) * fix(image-refresh): re-pin the digest when a helm re-render reverts the workload to :tag (client-runtime#199) image-refresh only re-pinned on a registry digest CHANGE. After `helm upgrade --reset-then-reuse-values` (the hourly auto-upgrade) re-renders the workload back to repo:tag and discards the `set image repo@digest` pin, the next tick saw `recorded == latest` and no-op`d -- leaving the workload on the bare tag (IfNotPresent), where a stale node :tag layer silently runs an OLD control-plane image. That is how a pre-#416 jobs-manager ran under a sealed egress netpol on the stg/prod fleets. Read the live workload image in the recorded==latest branch and re-pin when it is not repo@latest. shellcheck-clean; helm-unittest guard added. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(chart): rebump to 1.9.109 (develop advanced to 1.9.108) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(image-refresh): re-pin the requests-proxy when only it reverts, not just the api container (backend#199) Bugbot Medium on #1008: the no-op decision for tracebloc/jobs-manager read only the deployment api container. When it already matched repo@latest the loop continued and never wrote rp_set_args -- so a tick that pinned the api then died before the requests-proxy rollout, or a helm re-render that reverted only the proxy, left the proxy on :tag and later ticks skipped it forever off the api match alone. Read the requests-proxy proxy container too (when it follows the jobs-manager digest) and fall through to the re-image path -- which re-derives both jm_set_args and rp_set_args -- when it is off want. Guard the fall-through logs so they stay accurate. Regression-guarded in the script test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(image-refresh): compare the re-pin on the @sha256 digest, skip on unreadable, and align the docs (backend#199) Addresses LukasWodka's review on #1008. Ask 1 — a BEHAVIOURAL test, not text-presence. Adds scripts/tests/image-refresh-repin-on-revert.bats, which extracts the shipped `recorded == latest` branch from the rendered chart and drives it with the registry HEAD and the two live-workload reads stubbed: a `:tag` revert re-pins, api+proxy both on the digest no-ops, a proxy-only revert re-pins the proxy, an unreadable read skips the tick, and a registry-prefix rewrite is not a revert. Inverting the comparison reddens these, which the helm-unittest text asserts could not detect. Design question — persistent mismatch under a mutating image webhook. Compare on the @sha256 DIGEST, not the whole reference: a webhook that rewrites the registry prefix to an internal mirror keeps the digest, so a prefix-only rewrite no longer reads as a revert -- which otherwise re-pinned every tick and tripped the #563 flap lockout for all control-plane images after three ticks. A genuine revert to `:tag` carries no @sha256 and still re-pins. Nit — unreadable live image now SKIPS the re-pin this tick and retries, instead of re-asserting (which burned a #563 flap attempt on a healthy edge and logged a revert that may not have happened). The two helper comments are corrected to this fail-closed stance, and the "reverted the pin" log is neutral wording that also covers the fresh-install case. Ask 2 — the header no longer contradicts the code. Rewrites the HELM RE-RENDER limitation as HANDLED (bounded to one tick), the first-tick contract as "pins on the tick after first observation", and the two matching values.yaml bullets; and drops the "DESIGN NOTE (for review)" block from the shipped ConfigMap, since a customer operator cannot follow a PR thread. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(image-refresh): surface a re-pin whose rollout never settled, don't no-op it (backend#199) Bugbot High on #1008: a re-pin does `set image` (which moves the spec to repo@digest immediately) then waits on `rollout status`. If that rollout times out on requests-proxy or the resource-monitor DaemonSet -- both OUTSIDE the top-of-tick settled guard, which only covers jobs-manager -- the tick exits under `set -e` with ATTEMPT_KEY still raised, and the next tick's live-image check reads the (already-moved) spec as on-digest and no-ops. The stuck rollout is never retried and stale :tag pods keep running while the CronJob stays green. The digest-CHANGED path self-heals here (recorded stays old until the rollout settles, so it retries); the re-pin path could not, because recorded is already == latest. Read ATTEMPT_KEY once before the loop: a raised value means a prior re-image never reached its success-reset, so the "both on digest" no-op re-enters the re-image path instead of continuing. `rollout status` then runs again -- a settled workload resets the counter (idempotent `set image`, one fast status call), a genuinely stuck one advances the counter to the #563 flap lockout, which SURFACES it rather than hiding it. Best-effort read: unreadable/absent is treated as 0, since the restart block's own read stays the fail-closed authority. Adds two bats cases: on-digest + raised ATTEMPT_KEY re-runs the rollout (restart_needed=1, no no-op); on-digest + no pending attempt stays a clean no-op. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(image-refresh): gate the forced retry below MAX so a latched flap stops suppressing the annotation write (backend#199) blocking 1+2 (@shujaatTracebloc on #1008): the pending_attempt>0 forced re-run is a no-op once ATTEMPT_KEY latches -- the flap guard exit 0s before any set image/rollout status -- and worse, every latched tick then skips the annotation write (first-observation records, stale-pin clears) forever. Gate it on pending_attempt < MAX_REFRESH_ATTEMPTS so a latched image falls to the no-op path and the tick completes. blocking 3: state the fresh-install re-pin cost honestly in the header (the resource-monitor DaemonSet rollout can latch the shared lockout on a NotReady-node fleet; jobs-manager Recreate downtime). bats 9/9. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(image-refresh): surface a latched flap (WARN + FLAP_KEY) on the on-digest no-op tick (client-runtime#199) With the < MAX gate, a latched tick (pending_attempt >= MAX_REFRESH_ATTEMPTS) keeps restart_needed=0 and never enters the downstream flap guard -- the only other writer of FLAP_KEY and the MANUAL ATTENTION WARN. Refresh is then dead for ALL control-plane images while the CronJob stays green, which #1964 forbids ("images did not update" must never be inferable only from the Job colour). Emit the WARN naming the refresh-attempt clear and annotate FLAP_KEY in the latched arm before the no-op continue. Two bats cases pin it (WARN+FLAP at pending=3; silence at pending<MAX), mutation-proof on the -ge bound. Also sweep before-squash nits flagged by all three reviewers: backend#199 -> client-runtime#199 (public repo, correct issue), proxy_off_digest typo, and the "hourly auto-upgrade" overstatement. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(image-refresh): rebump chart to 1.9.110 (develop took 1.9.109 via client#1017) (client-runtime#199) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Syed Saqlain <syedsaqlain@MacBook-Pro.local> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…SPAN mutation (backend#3509) (#1025) * test(gate-default-prose): restore the mixed-polarity case (c) + its _SPAN mutation (backend#3509) Case (c) had been reduced to an all-true LIST claim after backend#947 baked every *ByEnv gate true everywhere, so nothing in the suite still crossed an opposite- polarity word: a revert of _SPAN to the greedy [\w,\s]* would have stayed green (Bugbot Medium on the staging promotion client#1012). Restore the false-positive guard as a genuinely mixed fixture. A new mixfixture() builds a throwaway tree where narrowEdgeuserByEnv ships true for dev and false for stg/prod, with every source that names the gate (chart, schema description, helper comment, runbook) stating that same mixed reality. Case (c) asserts the correct mixed claim is GREEN under the shipped _SPAN; case (c-span) reverts _SPAN to the greedy span in the fixture's own guard copy and asserts the SAME fixture reddens with "says 'true' for stg" -- proving the refuse-to-cross _SPAN is load-bearing (repo CLAUDE.md rule 9: assertion and mutation call the real _SPAN, not a copy). gate-default-prose-mutations: 13 passed, 0 failed; shellcheck -S warning clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(gate-default-prose): scrub internal tracker ids from the public script, tighten the fixture assert Lukas: this is a public repo -- the comment block named the private tracker five times. Describe the finding in words instead, matching the repo precedent (keeping internal ticket identifiers out of customer-facing copy). Pre-existing header refs are left untouched (out of scope). Also the nit: the mixfixture flips only the narrowEdgeuserByEnv block, so exactly stg+prod go false -- assert == 2, not >= 2. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Syed Saqlain <syedsaqlain@MacBook-Pro.local> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…t-enable ensure_cluster_autostart no longer bails the whole function when the bounded docker ps node read times out; it skips only the node restart-policy loop (k3d already sets --restart unless-stopped at create time) and still runs the Linux docker.service boot-enable, logging the skipped read. Finding recorded off the client#1011 promotion review.
Contributor
Author
|
bugbot run |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 2c15020. Configure here.
tracebloc-release-train
Bot
added
gate-nudge
Contributor
Author
Functional review — staging batch 2026-09-10 (10:43 hop): HELD, not passedEvidence pulled 11:56–12:20 UTC against these tips: fr-assist channels all OK (1 SKIPPED, credentials); frontend smoke on The cause is not this batch: it is chart v1.9.109 (#1017, kubelet reservation) combined with client-runtime's v3 envelope contract (tracebloc/client-runtime#544), both already on No card from this batch advances until the journey trains again on the corrected chart. No prod hop until then. |
Contributor
Author
Functional review — staging batches of 2026-09-10 (10:43 and 14:49 hops) — PASS (FR reviewer: LukasWodka; self-signoff per RFC-BACKEND-1405 D6)Decision: PASS, by the reviewer's instruction at 15:45 UTC. All 132 cards that sat at What changed since the hold. The 12:00 hold was not about this batch: the edge refused every training envelope with Evidence at signoff.
Things this evidence cannot tell you (README → Functional review): interactive/TTY behaviour, per-ticket behaviour, anything at or after a failing step, and any step a run skipped. |
Contributor
Author
|
Correction to the evidence paragraph above (15:50 UTC). The two journey runs named there (34497041338, 34497050908) were dispatched with |
Contributor
Author
|
Journey verdict appended (16:20 UTC) — the fix is confirmed on staging's candidate chart.
Same node, same runtime image, same envelope; only the chart differs. The published chart (prod) still refuses; the candidate admits and trains to completion. Prod hop proceeds so 1.9.112 supersedes 1.9.109. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Automated promotion by the release train (RFC-0008 D14). Head is the train-managed
release-train/to-stagingbranch (a mirror ofdevelop), so it never collides with a human PR. Merged only when the fr-gate is green.Note
Medium Risk
Changes edge control-plane image reconciliation and rollout/flap behavior on every cluster running image-refresh; incorrect live-image logic could cause extra rollouts or missed re-pins, though tests heavily guard the new paths.
Overview
Bumps the client Helm chart to 1.9.110 and hardens the image-refresh CronJob so control-plane images stay on the intended digest when Helm re-renders workloads back to
repo:tagafter fleet auto-upgrade.When the registry digest is unchanged, the script now reads live deployment/DaemonSet image refs (including requests-proxy separately from jobs-manager
api) and re-pins on@sha256mismatch, with fail-closed skips on unreadable API reads. It also retries unfinished rollouts viaATTEMPT_KEY, surfaces flap latch warnings when refresh is stopped, and documents the second-tick digest pin on fresh installs.Installer / guard fixes:
ensure_cluster_autostartno longer aborts the whole function when enumerating k3d nodes times out—Linux docker.service boot enable still runs. The customer-copy-no-ticket-refs guard gets a shared quote/here-doc lexer, scans here-document bodies for tracker tokens, and safer temp-dir handling. gate-default-prose mutation tests restore mixed-polarity false-positive coverage for_SPAN.Adds bats coverage for re-pin behavior (
image-refresh-repin-on-revert.bats) and related helm-unittest regex locks.Reviewed by Cursor Bugbot for commit 2c15020. Bugbot is set up for automated code reviews on this repo. Configure here.