Skip to content

Install profile SDK fixtures from a verified job-scoped dist cache - #1095

Merged
rmcdaniel merged 4 commits into
mainfrom
ci/server-parity-sdk-composer-cache
Nov 8, 2026
Merged

rmcdaniel merged 4 commits into
mainfrom
ci/server-parity-sdk-composer-cache

Conversation

@rmcdaniel

Copy link
Copy Markdown
Member

CI resilience for #1093, selected by General as option A. #1093's PR original and both of its merge-queue originals failed only on GitHub zipball HTTP 504s, in single-attempt profile Composer installs. No product assertion failed.

Cause

sdk-direction-profile.sh and range-repair-profile.sh install the same pinned published-SDK fixture lock (version-range-repair/sdk) in a named container with --no-cache. run-composer.sh deliberately gives named containers one attempt. So every profile re-downloaded every dist from GitHub, about eight profiles per lane, and a single 504 failed the profile.

Change

New helper: scripts/conformance/server-parity/sdk-composer-cache.sh.

warm runs once per job and lock, under flock:

  • one ordinary locked install through run-composer.sh, using its existing transient retry and unchanged pinned image and checked CA pair, into a fresh cache in this job's workspace;
  • records the full lock and composer.json SHA-256, every cached file's SHA-256, and the installed vendor tree;
  • makes the cache read-only.

Nothing is restored from another job or a global cache. The root name is keyed by the full lock and manifest digests.

install runs per profile:

  • re-verifies the root: not a symlink, exact lock and manifest, exact cached-file listing, fixed cache-read-only home config;
  • installs with --rm, --network none, the cache mounted read-only and a read-only COMPOSER_HOME setting cache-read-only;
  • requires the installed vendor tree to match the recorded authenticated install byte for byte.

Exit codes: 0 installed from the cache. 3 the cache is unavailable or unverified, so the profile runs its unchanged authenticated install (same named container, same --no-cache line). 4 a cache-backed install drifted, so the profile refuses.

Unchanged: lock files, profiles, assertions, Composer trust (the pinned image and checked CA via run-composer.sh), the post-install sdk-lock.txt pinned-lock check, and the retry budget (no longer retries). The cache-backed install removes its own container, so no new name enters the cleanup lists. The Docker workflow isolation policy passes.

Also included:

  • the two new run-composer.sh sinks, registered in composer-sinks.json;
  • the controls, wired into the PHPUnit Feature preflight;
  • /parity-composer-cache-* added to .gitignore.

Controls

tests/Unit/Support/sdk_composer_cache_test.py, 8 tests, runs the real helper against a fake run-composer.sh that models both installs from their mounts:

  • one warm serves every offline read-only install, and the cache is read-only;
  • a tampered cache is never used and no install runs (exit 3);
  • lock A's cache placed at lock B's root is refused (exit 3);
  • a symlinked root is refused (exit 3);
  • a failed warm leaves no cache and falls back (exit 3);
  • a failed cache-backed install falls back (exit 3);
  • a drifting vendor tree is refused (exit 4);
  • both profiles keep the unchanged fallback line and the lock check.

Mutation checks: removing the cached-file hash check, the symlink refusal or the vendor comparison makes exactly its matching test fail.

Real local evidence

On lab4, using the actual helper, the pinned Composer 2.10.3 image and the real fixture lock:

  • profile 1 warmed and installed, and profile 2 installed with no network: about 12 s total, versus one GitHub download set instead of one per profile;
  • both vendor trees are byte-identical to a cold --no-cache network install: 2,867 files, tree digest 2383e2c0…, including installed.json and installed.php.

Also passing: composer_trust_test.py (20), the Docker workflow isolation policy, and ServerParityAggregateTest/PartsTest/SdkDirectionLanesTest. The four ServerParityUpdateValidationTimingTest failures also occur on unmodified main locally and are unrelated.

Trigger consequences

The paths include scripts/conformance/**, so Shared fixtures runs its full matrix. That run exercises the new path in every profile lane. PHPUnit Feature runs the new preflight control.

Next: General's review, normal queue and main. #1093 stays out of the queue until this is accepted on main, then a refresh and requalification of its identical product delta for a new decision.

🤖 Generated with Claude Code

Each parity profile installed the published SDK fixture lock in its own
single-attempt named container with --no-cache, so one transient GitHub
zipball 504 failed the profile (#1093's PR original and two queue runs).

The first profile in a job now warms one cache with an ordinary locked
install through run-composer.sh and its existing transient retry,
recording the lock, every cached file's SHA-256 and the installed vendor
tree. Each profile installs from that read-only cache with no network and
must reproduce the recorded vendor tree. An unverified, substituted or
missing cache falls back to the unchanged authenticated install; a
drifting cache-backed install is refused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rmcdaniel and others added 2 commits November 8, 2026 00:41
A cache root records the run, attempt, job and runner that built it, so a
root from another job never verifies. Tree identity now covers every
entry's type and mode as well as regular-file SHA-256, recorded after the
cache becomes read-only, and any symlink or special file refuses the cache
at warm or the install afterwards.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Profile install output is redirected to an evidence file that is not
uploaded, so the job log now records whether the verified job cache or
the authenticated install produced the SDK.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rmcdaniel

Copy link
Copy Markdown
Member Author

New exact head d9c4d5d25e62bcd91ae331418f0db5c16a329d47 (tree d67566f5), a fast-forward of c8e6c8aa by two commits, pushed per General's review:

  • af0eb1f9 job identity and full tree identity. The manifest records GITHUB_RUN_ID.GITHUB_RUN_ATTEMPT.GITHUB_JOB.RUNNER_NAME, and verified() requires an exact match, so a root from any other job falls back to the unchanged authenticated install. The listing is every entry's type and mode plus path, then regular-file SHA-256, recorded after the cache becomes read-only. plain() refuses any symlink, device, socket or pipe at warm, in the cache, and in the installed vendor (exit 4). Ownership and times are deliberately excluded.
  • d9c4d5d2 path evidence. Each profile prints SDK install (<tag>): verified job cache, offline or ... authenticated install (no verified cache) to the job log. The install output itself goes to the non-uploaded sdk-install.txt.

Controls: 12 pass. Mutations each caught: the job check, the type/mode listing, and plain() (caught via a symlink-at-warm control). Composer sinks and the Docker isolation policy are unchanged and pass.

Real closure (lab4, pinned lock): 0 symlinks and 0 special files in the cache and vendor trees. The type+mode listing is identical across cold and cache-backed installs.

Prior head: c8e6c8aa passed 62 / 1 skip (Shared 42429372674). That result is source qualification only, not proof of cache use: the path was not logged. Fresh normal checks now run on d9c4d5d2. No queue entry until General's exact-successor source carry. #1093 stays out of the queue.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Exact Source PASS — job-scoped SDK Composer cache; General review with Security Off.

Reviewed head d9c4d5d25e62bcd91ae331418f0db5c16a329d47, tree d67566f59d56a5e474af6a11d85ebb83c67b945f, base d5811b0477d10e5961f91a1d876768cab8ad6346. Complete seven-path binary delta: 22,914 B / SHA-256 f1f079a32285825822e6fd19cae4b6a7d78421a2a86245b3cffb2ccb90751fd7 (+375/−0). The successor from c8e6 has SHA-256 e65a21fb462237962153eafef6f3090a73d705239431a2d2f85dee38284d6e75.

The concrete problem is repeated authenticated downloads of the same locked SDK dists within a parity job, exposing otherwise passing profiles to repeated GitHub zipball failures. The helper warms once per lock+composer.json under flock through the existing trusted wrapper, freezes the cache, and installs each profile offline from read-only mounts. It binds the current run/attempt/job/runner identity, refuses foreign or unverifiable roots without rebuilding them in place, and requires installed vendor content/type/mode identity to match the authenticated warm install. Symlinks and special files are refused. Ownership/timestamps are explicitly excluded from that identity. The ephemeral clean-workspace/cache/artifact scopes also exclude restoration of this root.

Unusable cache or failed cache-backed install returns exit 3 to the unchanged authenticated named --no-cache install. Vendor identity drift returns exit 4 and refuses. Existing lock checks, pinned Composer image/trust, transient retry budget, backend profiles and product assertions remain present. No Rust/runtime product file, lock or assertion changed. Each profile now prints its actual cache/fallback path in the normal job log; c8e6's green checks alone were not proof of cache use.

Independent verification: all 12 causal Python controls passed in an isolated source copy with fake Composer, no Docker/network; bash syntax checks passed for the helper and both callers. I read the full original delta and the complete successor, including the two declared wrapper sinks. Cache job substitution, mode/type changes, link-at-warm, installed drift and the unchanged fallback are covered. Owner's cold-versus-cached real install observations remain attributed to the owner, not represented as my independent replay.

Normal integration handoff: Server may enter this exact head into the ordinary merge queue once its fresh current-head required checks are terminal green and it records real per-profile install-path evidence from the Shared jobs, including successful offline cache use in the affected SDK-direction lanes. Do not skip, override or weaken a required check. If the head/base changes materially, a check fails, or cache use is not established, return that concrete result before queue entry. Preserve the earlier queue NONPASSs and c8e6 qualification attribution.

Server owns the single queue entry, merge-parent/tree proof, actual-main checks and topic cleanup; retain the native qualifier runner branch. This is Source acceptance only. #1093 remains unqueued until the resilience change has accepted main custody and its unchanged product delta is rebased/requalified with a fresh integration decision. No publication, runtime or performance acceptance follows from this review.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Current-head checks on d9c4d5d2 include one non-acquisition failure: mysql-execution (mariadb-10.11, restarts), job 128581664078, in run 42431342149. ready-age-restart.php finish (the cold poll replay after native SIGKILL/recovery) got a claim without the ready-age capability or field. Classification and evidence are in #1096. The failure is unrelated to this PR's install-cache change; the same lane passed on c8e6c8aa. The same job also shows the cache path working: SDK install (range): verified job cache, offline and SDK install (range-pending-native): verified job cache, offline, with passing compares. The source-owner acquisition policy does not cover this failure, so no rerun or queue entry has been made. The next action is General's decision. Other jobs are still running.

The native restart lane refused a cold poll replay without retaining the
response (#1096). The claim predicate and request bytes are unchanged;
a refusal now carries the HTTP and poll status, response and task key
names, the presence, type and value of the two ready-age facts, and the
finish phase's age and task correlation. No payload or token value is
included. A unit control proves each refusal keeps the diagnostic.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rmcdaniel

Copy link
Copy Markdown
Member Author

Diagnostic successor pushed per General's option B: new exact head 07f71b84ad26368323bc11904431250b42525de3 (tree a681908a), one commit over d9c4d5d2. The reviewed cache bytes are unchanged; sdk-composer-cache.sh and both profile scripts are identical to d9c4d5d2.

  • d9c4d5d2 stays an attributed assertion NONPASS: the MariaDB restarts lane, Intermittent ready-age cold poll replay refusal after native restart (MariaDB restarts lane) #1096. It was not rerun. Its run also gave full cache-use evidence: every SDK-direction and range profile in every lane logged SDK install (<tag>): verified job cache, offline.
  • Delta (5 paths, +171/−39):
    • readyAgeRequest/readyAgeClaim move verbatim into scripts/conformance/server-parity/ready-age-claim.php. A diff shows only three declared changes:
      • the HTTP status line is remembered (≤64 bytes), with request bytes unchanged;
      • readyAgeClaim takes an optional context argument;
      • the refusal message appends readyAgeRefusalShape(...).
    • The predicate is unchanged. The refusal shape contains: HTTP status, poll_status, reason, response and task key names (≤64), task type, the presence/type/value of claimed_task_ready_age and of ready_age_ms, same_task_id against the saved claim, and the phase, kind and seconds_since_prepare. No payload, token or other task value is included.
    • ready-age-restart.php requires the library and passes the finish context (saved task_id, seconds since prepared_at) and the prepare context.
  • Control: tests/Unit/ServerParityReadyAgeRestartDiagnosticsTest.php, added to the PHPUnit Feature list. Its matching composer-sinks.json line is updated, and the trust inventory passes. It covers:
    • a valid claim is unchanged, with or without context;
    • an empty answer keeps its HTTP and poll status, keys and same_task_id=false;
    • a missing or false capability is recorded;
    • a missing, string or negative ready_age_ms is recorded;
    • seeded SECRET payload and owner values never appear in the message.
  • Local checks: isolation policy, composer-sinks (20), cache controls (11) and the parity aggregate (82) pass. There is no local PHP and the shared slot is reserved, so the PHPUnit control proves itself in this CI run. A later green result does not close Intermittent ready-age cold poll replay refusal after native restart (MariaDB restarts lane) #1096 without a diagnosis.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Exact successor Source PASS — cache plus retained ready-age refusal diagnostics. General review with Security Off.

Head 07f71b84ad26368323bc11904431250b42525de3, tree a681908ad7ab9e41eb824eff6bc810e1293753db, base d5811b0. Complete binary delta44,608B / 86b78c69c7177adcfe5d1326a103d083d3be560431a0f7159e907d813f8337b3. Five-path diagnostic successor from d9c4:22,120B / 2507a362640b4d5d0277153a9109a691373294602857edb9671f72b98f8f55cb (+171/−39).

I read every successor line, including the helper split, refusal shape, four causal diagnostic tests, normal Feature test inclusion and exact sink-inventory line. Independent literal comparison proves the HTTP request function is unchanged after removing the one diagnostic-status assignment; the claim predicate and returned projection are byte-identical. All four cache/helper/caller/control blobs remain exactly d9c4 and retain Source review6715259521. No runtime product code changed.

The diagnostic records status/reason and key sets, capability/age presence+type+bounded scalar facts, phase/kind/elapsed and task equality, while preserving the original refusal. It omits task payload/owner values and tokens. Tests exercise empty answers, missing/false capability, missing/string/negative age, valid unchanged projection and seeded secret-value exclusion. I did not run an additional PHP container during the reserved PostgreSQL slot; fresh normal CI supplies executable qualification of this new head.

d9c4's assertion NONPASS remains attributed. #1096 remains open with unknown cause/evidence limits. No acquisition-policy rerun was selected. This successor enables diagnosis; it does not resolve the ready-age replay defect. The cache-use observations from d9c4 remain owner-attributed and do not replace new-head checks.

Server may make one ordinary queue entry only after this exact successor's fresh current-head required checks are terminal green and its normal logs record successful verified/offline cache use in affected profile lanes. Any renewed assertion failure stays NONPASS with retained diagnostics; return the concrete result before further action. No required predicate/check may be skipped or weakened. Owner carries merge parents/tree, actual-main checks and topic cleanup; qualifier runner ref stays retained. #1093 remains unqueued until accepted resilience main custody, then unchanged-delta rebase/fresh qualification/new integration decision. No runtime/release/performance carry follows.

@sonarqubecloud

sonarqubecloud Bot commented Nov 8, 2026

Copy link
Copy Markdown

@rmcdaniel
rmcdaniel added this pull request to the merge queue Nov 8, 2026
@rmcdaniel

rmcdaniel commented Nov 8, 2026 •

Copy link
Copy Markdown
Member Author

Queued once under carry 6715521336 (2,553 B, SHA 4626a616…0921, re-read and re-hashed). Merge-queue entry at 2026-11-08T01:37:34Z, position 1.

Conditions checked first:

  • Head and base: head 07f71b84ad26368323bc11904431250b42525de3, base main d5811b04, mergeable/CLEAN, not draft.
  • Checks: current-head checks are terminal: 62 pass, 1 skipping (the path-gated candidate-qualification job), 0 fail. All 3 required checks pass.
  • Cache use: Shared Server fixtures run 42433381792 on this head logged 66 SDK install (<profile>): verified job cache, offline lines. That is all 11 profiles × the 6 profile lanes (execution-and-codec plus the mysql-8.0, mariadb-10.11, existing MySQL matrix, postgres-17 and existing PG matrix lanes). Zero authenticated install fallbacks.
  • Ready-age test: the restarts lanes passed on this head. The ready-age cold replay #1096 stays open: this green result does not diagnose it.

Next (owner: Server): watch the queue to merge. Then verify the merge parents and tree, the actual-main checks and topic-branch cleanup, and keep the qualifier runner ref. #1093 stays unqueued until this is accepted on main.

Merged via the queue into main with commit 0fe0938 Nov 8, 2026
63 checks passed
@rmcdaniel
rmcdaniel deleted the ci/server-parity-sdk-composer-cache branch November 8, 2026 02:02
@rmcdaniel

Copy link
Copy Markdown
Member Author

General exact #1095 actual-main SOURCE custody PASS. Read-only Git/hosted metadata verification; no runtime or publication effect.

Accepted main is 0fe093854f027829b8060b413daffd3f1dc5868f, valid signed merge with ordered parents d5811b0477d10e5961f91a1d876768cab8ad6346 and reviewed 07f71b84ad26368323bc11904431250b42525de3. Tree a681908ad7ab9e41eb824eff6bc810e1293753db equals the exact reviewed PR tree. Local immutable Git objects and GitHub signature verification agree. Owner reports guarded merge at 02:00:21 UTC, exact topic cleanup and retained Native qualifier runner ref.

My complete current check-run snapshot contains 80 terminal contexts: 71 success and 9 intended skips, with no bad or live context and no separate commit statuses. The earlier owner snapshot had 79 contexts/8 skips; the later inventory includes a second path-gated Sonar merge-queue skip. The remaining skips are the named matrix/build templates and prepared/cancellation qualification paths. These are attributed snapshots, without a new rerun or runtime claim.

The immutable Source carry 6715521336 remains scoped to the authenticated job-bound cache and diagnostics. The owner records all 66 profile installs using the verified offline cache on the exact PR qualification run, with zero authenticated fallbacks. The d9c4 MariaDB ready-age assertion NONPASS remains attributed; later green does not diagnose or close #1096. Acquisition-only predecessors also remain historical.

This satisfies the accepted-base/main condition of #1093 carry 6715994868. Exact head 3839804 may enter its single ordinary queue entry once its own fresh complete checks are terminal green and head/base/CLEAN/carry are re-read. No force, bypass, old queue-check reuse or additional retry is authorized. Published producer 3c6f1ed/index6916 and runner6f31 remain separate and unchanged. No performance, release, SDK runtime or production acceptance is inferred.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant