Skip to content

release-train: develop -> staging - #462

Merged
tracebloc-release-train[bot] merged 1 commit into
stagingfrom
release-train/to-staging
Sep 10, 2026
Merged

release-train: develop -> staging#462
tracebloc-release-train[bot] merged 1 commit into
stagingfrom
release-train/to-staging

Conversation

@LukasWodka

@LukasWodka LukasWodka commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Automated promotion by the release train (RFC-0008 D14). Head is the train-managed release-train/to-staging branch (a mirror of develop), so it never collides with a human PR. Merged only when the fr-gate is green.


Note

Medium Risk
Changes gate behavior for Dockerfiles with parameterized bases: some repos may newly fail CI with cannot-parse until ARG defaults are set, but avoids silently missing CUDA-on-CPU torch issues.

Overview
dead-weight cuda-torch-on-cpu: Dockerfile stage GPU/CPU detection no longer treats unset FROM ${ARG} references as GPU when the ARG name contains cuda, and no longer silently skips those installs.

_stage_gpu_map now tracks per-stage unresolved base images (no resolved image name after ARG expansion). GPU hints are evaluated on stripped refs so placeholder text cannot fake a GPU stage. Stages with only a known name but an unset tag/registry (e.g. nvidia/cuda:${TAG}, ${REGISTRY}/python:3.11-slim) stay classifiable; truly unknown bases (bare ${CUDA_IMAGE}, nested defaults like ${IMAGE:-${GPU_BASE}}) are flagged.

Pip installs in an unresolved, non-GPU stage emit a cannot-parse scan-integrity finding (fail closed, like full-python-base), instead of a clean pass or a false cuda-torch-on-cpu hit.

Tests: New selftest cases and mutation anchors cover unset ARGs, nested defaults, registry-only placeholders, and the cannot-parse path.

Reviewed by Cursor Bugbot for commit 2645c82. Bugbot is set up for automated code reviews on this repo. Configure here.

…lent GPU pass (.github#457) (#459)

* fix(dead-weight): unresolved base-image ARG is cannot-parse, not a silent GPU pass (backend#3562)

`_stage_gpu_map` discarded the unresolved flag from `_expand_args` and ran
GPU_HINT on the raw FROM text. An unset `ARG` whose name contains cuda/gpu/
nvidia/rocm (e.g. `FROM ${CUDA_IMAGE}`) therefore matched GPU_HINT via its own
placeholder name, marked the stage GPU, and cleared the `cuda-torch-on-cpu`
finding without knowing the image -- the same ambiguity check_full_python_base
already reports as `cannot-parse`.

- Judge GPU-ness on the resolved text only (strip unresolved `${ARG}`), so a
  placeholder name can't spuriously satisfy GPU_HINT; a tag-only placeholder
  over a real GPU name (`nvidia/cuda:${TAG}`) is still GPU.
- Thread the unresolved flag through `_stage_gpu_map`/`_installers_of`; an
  install in an unresolved-base stage is now `cannot-parse` (scan integrity),
  not a silent clean pass.
- Selftest cases + mutations for the new behaviour; re-anchor the mutations
  whose source lines moved.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(dead-weight): treat any placeholder left after ARG expansion as unresolved (backend#3562)

Bugbot (.github#459): `_expand_args` does not recurse into a nested default
like `${IMAGE:-${GPU_BASE}}`, so the inner `${GPU_BASE}` survives expansion;
the GPU-strip then discarded it and the stage read as a known CPU image,
raising a false `cuda-torch-on-cpu` instead of `cannot-parse`.

Use ANY placeholder still present after expansion (`ARG_REF.search(ref)`) as
the unresolved signal, not just `_expand_args`'s non-recursive flag. Selftest
case + mutation for the nested-default path.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(dead-weight): cite the public thread; judge unknown base on empty resolved NAME (.github#457)

Review (.github#459): this is a public repo, so drop the private-tracker
citations -- code comments now reference .github#457 (the public Bugbot thread).

Also adopts the reviewer's nit: an image is "unknown" only when expansion leaves
no resolvable image NAME, not when any placeholder remains. A name templated
only in its registry or tag (`${REGISTRY}/python:3.11-slim`, `nvidia/cuda:${TAG}`)
is known -- the registry-prefix case is a normal cuda-torch-on-cpu finding with
the real "+cpu / index" remedy again, not a false cannot-parse. Nested defaults
and fully-templated names (`${IMAGE:-${GPU_BASE}}`, `${CUDA_IMAGE}`) stay
cannot-parse. Moved the CPU-via-ARG counter-assertion beside its GPU twin.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Syed Saqlain <syedsaqlain@MacBook-Pro.local>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
@LukasWodka

Copy link
Copy Markdown
Contributor Author

bugbot run

@tracebloc-release-train tracebloc-release-train Bot added the gate-nudge Toggled by the release train to (re-)fire the fr-gate label Sep 10, 2026

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 2645c82. Configure here.

@tracebloc-release-train tracebloc-release-train Bot removed the gate-nudge Toggled by the release train to (re-)fire the fr-gate label Sep 10, 2026
@tracebloc-release-train
tracebloc-release-train Bot merged commit 9574704 into staging Sep 10, 2026
74 checks passed
@tracebloc-release-train
tracebloc-release-train Bot deleted the release-train/to-staging branch September 10, 2026 11:21
@LukasWodka

Copy link
Copy Markdown
Contributor Author

Functional review — staging batch 2026-09-10 (10:43 hop): HELD, not passed

Evidence pulled 11:56–12:20 UTC against these tips: fr-assist channels all OK (1 SKIPPED, credentials); frontend smoke on staging 34473123260 green; journey baseline 34473117815 red and fleet 34473120598 red on 15/16 tasks, all at Train a model: the edge refuses every envelope with NEVER_STEADY_STATE ("each run needs cpu=1, memory=3Gi, ephemeral-storage=2Gi, and the closest node leaves cpu=900m, memory=2390Mi").

The cause is not this batch: it is chart v1.9.109 (tracebloc/client#1017, kubelet reservation) combined with client-runtime's v3 envelope contract (tracebloc/client-runtime#544), both already on main since this morning's prod hop. The 06:00 battery on chart 1.9.108 was 13/16 green with the same jobs-manager image. Fix in flight: tracebloc/client#1031 reverts the chart half (1.9.112).

No card from this batch advances until the journey trains again on the corrected chart. No prod hop until then.

@LukasWodka

Copy link
Copy Markdown
Contributor Author

Functional review — staging batches of 2026-09-10 (10:43 and 14:49 hops) — PASS (FR reviewer: LukasWodka; self-signoff per RFC-BACKEND-1405 D6)

Decision: PASS, by the reviewer's instruction at 15:45 UTC. All 132 cards that sat at FR on staging — the 86 held since the 10:43 hop plus the 46 this hop pushed — are advanced to Ready for prod.

What changed since the hold. The 12:00 hold was not about this batch: the edge refused every training envelope with NEVER_STEADY_STATE because chart 1.9.109 (kubelet reservation) plus the runtime's v3 envelope contract left a 2-core / 8 GiB node 100 m and 682 MiB short of the smallest envelope. The interim control-plane trim (tracebloc/client#1032, chart 1.9.112, render 650 m / 2272 MiB with the collector counted) merged at 14:47 UTC and reached staging with this hop at 15:37 UTC; rc v1.9.112-rc.1 is published.

Evidence at signoff.

  • Staging tips: every repo in the hop is on its 15:20–15:37 UTC mirror; the e2e-test-agent mirror was HELD by the train on a Bugbot High (teardown ceiling) and its harness stays at the 11:22 tip — that affects nothing under review here.
  • Artifact channels (fr-assist, 11:34 UTC run 34471966935): all OK, 1 SKIPPED (credentials).
  • Frontend smoke on staging (11:47 UTC, 34473123260): green.
  • Journey battery on the corrected chart is IN FLIGHT at signoff (dispatched 15:39 UTC on the staging harness): baseline https://github.com/tracebloc/e2e-test-agent/actions/runs/34497041338 and the 16-task fleet https://github.com/tracebloc/e2e-test-agent/actions/runs/34497050908. The verdict that matters is Train a model scheduling on the 2-core k3d node. It will be appended below when the runs finish; if it is red, the prod hop does not proceed and the cards come back.

Things this evidence cannot tell you (README → Functional review): interactive/TTY behaviour, per-ticket behaviour, anything at or after a failing step, and any step a run skipped.

@LukasWodka

Copy link
Copy Markdown
Contributor Author

Correction to the evidence paragraph above (15:50 UTC). The two journey runs named there (34497041338, 34497050908) were dispatched with chart_source/chart_ref blank, which installs the published chart from the Helm index — the index is stable-only, so they exercise chart 1.9.109, the customer path as it is on prod today, not the 1.9.112 candidate on staging. They are a control, not the proof. The proof runs are the candidate-chart baselines dispatched with chart_ref=v1.9.112-rc.1: https://github.com/tracebloc/e2e-test-agent/actions/runs/34498185079 (tabular_classification) and the image_classification twin dispatched a minute later. The fleet workflow has no chart_ref input, so the candidate is proven on the baseline cell only. Verdicts will be appended here; the prod hop waits for them.

@LukasWodka

Copy link
Copy Markdown
Contributor Author

Journey verdict appended (16:20 UTC) — the fix is confirmed on staging's candidate chart.

run chart installed node arithmetic (k3d-tracebloc-server-0) Train a model
34498185079 tabular_classification client-1.9.112-rc.1 (candidate) allocatable 1900m / 5538Mi, requested 750m / 2284Mi, leaves 1150m / 3254Mi → envelope fits ✅ experiment COMPLETED, leaderboard found
34498269145 image_classification client-1.9.112-rc.1 (candidate) same: leaves 1150m / 3254Mi → fits ✅ experiment COMPLETED, leaderboard found
34497041338 control, published chart client-1.9.109 (prod today) requested 1000m / 3148Mi, leaves 900m / 2390Mi → refused UnschedulableEnvelope at 15:54 UTC
34497050908 control fleet, published chart client-1.9.109 same refusal on 14 legs; 1 leg died earlier on a cosign download inside the installer ❌ 15/16 red at Train

Same node, same runtime image, same envelope; only the chart differs. The published chart (prod) still refuses; the candidate admits and trains to completion. Prod hop proceeds so 1.9.112 supersedes 1.9.109.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants