Skip to content

feat(fleet): durable movement foundations and bounded reconciler - #37

Draft
forhappy wants to merge 29 commits into
mainfrom
codex/fleet-operations-foundations
Draft

forhappy wants to merge 29 commits into
mainfrom
codex/fleet-operations-foundations

Conversation

@forhappy

@forhappy forhappy commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Latest checkpoint ac739bc: add CellNode::fleet_snapshot for request-bound original native pages. Pin the full journal head/registry, physical node/boot, nonce, category, native continuation, bounds and deadline. Authorize before and after capture through the existing two-job action bank; keep accepted work and native page charges owned across canceled waiters and shutdown. Validate original intervals and available native identities. Unbound owners remain explicit incomplete coverage. The SQLite adapter checks the full barrier and endpoint intent transactionally; reader producer pages expose their original scope and physical node. Persisted/wire formats are unchanged.

Focused evidence for ac739bc: 118 distinct scoped cases pass: complete host library 29, complete public node 88, and one independent SQLite authorization case (90 example cases filtered). Eight cases are new. Host all-target Clippy, host/runtime API docs with warnings denied, format/diff and static validators pass. All 628 Rust/Cargo/lockfile blobs match tested manifest SHA256 9c41f0ee52ae80ddb5929e6158ed6e625377d236d1e7c768c6ec14554554fea6. The complete example and broad workspace/process suites were not rerun locally for this checkpoint. See the execution record.

Readiness: this PR remains draft. Complete authenticated aggregate observation, current authority, replacement policy, role evacuation/finalization and full qualification remain outstanding. New-head CI is pending. Parent bb08c90 passed workspace/MSRV, contracts, smoke, both capacities, website, fuzz, fast/negative TLC and object-only routing; broad TLC/simulator were skipped. Leased routing failed the unchanged performance gate for leased/local_command/c1 after all eight functional executions passed; cause remains unestablished. Its terminal failure supersedes the running status recorded below for that checkpoint.

Previous checkpoint bb08c90: expose CellNode::fleet_durability_supervisor through the original retained supervisor owner. Capture distinct lifecycle states, separate original supervisor/request-stop failures, and the existing bounded rotation bank including automatic claims. Capture waits on no join/provider/native work and remains readable during drain after byte admission closes. Commit the consumed task join before bookkeeping so a failed stop cannot repoll its JoinHandle. The fixed four-KiB metadata owner uses the existing retained-byte ledger before lease admission; native byte admission stays lease-fenced and terminal drain closes both allocation paths.

Focused evidence for bb08c90: 234 distinct scoped cases pass: complete host library 26, public node 84, complete fleet example 90, isolated application integration 29, and isolated runtime lease selection five (196 filtered; 16 documented manual application cases ignored). Ten cases are new. Runtime/host/application all-target Clippy and runtime/host API docs pass with warnings denied. Format/diff, boundaries/layout, 110 Rust snippets, 1,172 Markdown links and SQL/peer gates pass. All 623 Rust/Cargo paths match unchanged manifest SHA256 f1f06c9f28802aef58710332984425116ab4f01681f903c10533d6c651ab6340, rechecked after each gate and against staged blobs. The unchanged existing startup test caught an intermediate Fenced regression, which the final metadata path fixes. Real successful/panicked tasks with a poisoned original bank retain separate source identities across repeated joins. Details, original intermediate failures and exact commands are in the execution record.

Outstanding qualification: the full plan remains unfinished and this PR stays draft. Supervisor capture is advisory; complete authenticated role/current-authority envelopes, replacement policy, cross-session recovery, role evacuation/finalization and full process/provider/compatibility qualification remain required. This commit requires its own CI. On PR head 9faedb9 (merge source 3b3bb0a13eae209a845ded4e4ef9316fdc9a930d), workspace CI passed 89 fleet cases and failed the controller-restart requirement of two retained lost release replies. Compose smoke failed reader lane 6 replacement progress: all three calls started in the 14,247,556-microsecond interval returned behind, with no wholly contained successful call. Original logs, artifact, source/binary identities and derived interval records are archived; causes remain unestablished. Contracts, MSRV, capacities, website, fuzz, fast/negative TLC and leased routing pass for that prior head; broad TLC/simulator were skipped and object-only routing was still running at the recorded capture. These results do not qualify this commit or complete W1–W10.

Previous reader fixture checkpoint (9faedb9)

Reader fixture checkpoint 9faedb9: make the 128-native-reader pagination qualification independent of opening speed. A retained 127-view workload reproduced the original admission refusal: 1,598,029,824 native bytes plus 24,969,216 retained bytes crosses 600 permille of the original combined budget, with zero jobs/disk and measured Constrained. Give this explicit fixture four GiB of native credit, require Normal pressure across 1,500 ms of real samples, and preserve all 128 requests/views/readbacks and zero joined ledgers. An intermediate full run exposed the original 30-second advertisement expiry; renew both signed boots through canonical directory CAS and advance the receiver guard only after confirmation. A new canonical renewal case survives original real expiry and then confirms fencing. Ordinary memory budgets, production profiles, pressure thresholds, publication bounds and expected proof assertions remain unchanged.

Focused evidence for 9faedb9: 220 distinct cases pass (host library 20, public node 81, complete fleet example 90, isolated application integration 29; 16 documented manual application cases ignored). Host/application all-target Clippy and host API docs pass with warnings denied; format/diff, boundaries/layout, 110 Rust snippets, 1,172 Markdown links and SQL/peer gates pass. All 619 indexed Rust/Cargo paths match unchanged manifest SHA256 fbc5eb815cfc35d883bc56f84451e8fa52f196ad4934f9070fe2b3090538d835, rechecked after each gate. Detailed reproduction/failure evidence is in the execution record. The separately authored control-plane plan/audit from 353904e is preserved.

Qualification recorded for that checkpoint: the full plan remains unfinished and this PR stays draft. Subsequent CI failures are recorded in the latest checkpoint above. Previous publication-hint and controller-restart failures remain open. Five two-CPU/four-GiB/zero-swap baseline Linux runs all failed, including original OS descriptor exhaustion and additional timing/controller failures; their original binary, all source hashes and logs are archived. The corrected Linux ARM64 campaign is terminal: with two CPUs, four GiB, zero swap and 8,192 descriptors, run 1 passed 89 cases and failed measured_overload_moves_real_cells_after_durable_controller_reconstruction with real movement did not settle both attempts; its cause remains unestablished. Runs 2–5 each passed all 90 cases (28.83, 31.62, 57.06 and 51.62 seconds). The 128-reader and canonical renewal cases passed in all five runs, including beyond the original 30-second expiry. Every before/build/after source manifest exactly matches the qualified 619-path fbc5eb81… source; binary SHA256 is 6ba713e78de3c0de84d6d6b997681b7a6736199bd94255bba47dccad5ee51a14. Build JSON, exact executable/binary, limits, all original logs and terminal no-OOM container state are archived at /tmp/cellule-host-linux-evidence-reader-pressure-fix/. Repetitions are diagnostic evidence and not extra distinct tests; the first failure prevents claiming this entire campaign passes. This checkpoint addresses the diagnosed fixture admission and lease defects, not the remaining full-plan work.

Implement fleet operations foundations through the canonical runtime and host lifecycle, with a self-contained design and implementation plan and execution evidence.

  • Add durable operation/controller transitions, physical-node intents, enrollment records, immutable action inputs/results, receiver credit and atomic permit/history retirement.

  • Publish signed pressure/mode/capacity samples and bounded role inventories. Drive the existing planner/reducer through a caller-driven reconciler with durable cooldowns, unknown-result inspection and controller replacement.

  • Reuse canonical Cell release, restoration, activation and recovery. Preserve original acknowledged outcomes and exact roots; retain accepted effects across waiter cancellation and host deadlines.

  • Hold configured fleet startup until exact established boot/current-intent confirmation. Preserve maintenance mode and management readiness; join runtime work before canonical withdrawal and journal retirement.

  • Bind reader activation/removal to the shared journal. Join retained peer clones, accepted queries, refreshes and native SQLite opens before durable closure.

  • Retain complete original follower fences before authority close. Epoch-bound maintenance requests use the existing durability supervisor and validate expected boot, physical leader and newer epoch during replacement.

  • Previous follower checkpoint (489c5a9): expose bounded managed follower epoch progress without waiting on provider/journal/native I/O. Preserve all original selected requests, unknown acceptance, exact signed-attempt/proof identities and original retirement/error Arcs; hash opaque ensembles/tokens in place. Capture preparation before any row via protocol state. Charge one MiB per page, reuse the canonical 32-epoch bound, invalidate changed/missing continuations and preserve typed component capture failures. Add live requested-rotation failure evidence with the next native retry held, so original response identity is deterministic. Closed or exhausted ledgers cannot fabricate empty coverage. Publication-hint failure diagnostics now retain partial peers and occurrence/time; the two-second bound and five-second periodic tick are unchanged.

  • Previous reader checkpoint (8cff2c3): expose bounded original reader enrollment progress through paused acceptance, native opening and publication. Keep the canonical activation lane and retained job owner; use brief record locks so Pending and unknown work remain visible. Track accepted preparation before a row exists, unobserved task completion and joining handles. Charge one MiB per page, stop copying before variable payloads exceed that budget, preserve original errors independently, and reject changed continuations. Native execution errors are retained before publication awaits. Add real 128-request/native-view/readback/cleanup qualification and align the follower deadline scenario with its retained native owner.

  • Previous producer checkpoint: install a managed follower producer in that same supervisor. Every original selected member must be durably Pending before the one immutable directory CAS. Unknown results inspect only the retained attempt or its original-token refusal fence. Original establishment events replay before configuration delivery; complete member fences, canonical close and every durable retirement precede inventory/resource release. Undelivered attempts remain owned through drain.

  • Atomically refuse joined nonexecution in the shared journal transaction domain, including absent keys and lost acceptance replies. Retain terminal exclusion rows so delayed acceptance cannot reopen reader/follower obligations. Preserve original native and journal errors independently.

  • Supply one local SQLite transaction domain and executable overload, controller-restart, and inspect-journal examples. The movement scenarios use twelve real Cells across three independently leased hosts, move a bounded batch, reconstruct the controller and check original receipts and joined ledgers.

  • Previous roster checkpoint (5872122): traverse the complete durable intent/enrollment roster through canonical bounded pages before native observation. Supply FleetRoster to the observer, preserve original failed boots and both role endpoints, and recheck the full head/registry after capture. Count balancing additionally requires exact established signed boot coverage, no Pending enrollment and writer rows matching signed counts. Unknown/missing boots, racing enrollment and omitted actors cannot override these checks through an adapter's completeness flag. Retained original records enter the planner input digest; partial pressure relief retains its existing gates.

Applications own authorization, HTTP, providers, credentials, management transport, complete native-role observation and replacement policy.

Integration changes

  • FleetObserver::observe now takes &FleetRoster; use roster.snapshot() for the original head/registry and inspect every retained responsibility. The reconciler owns traversal and the subsequent version recheck. Both reference/model implementations are adapted.
  • NodeLogAuthority::close now receives NodeLogRetirementObservation; ordinary implementations obtain the existing rotation barrier through barrier(). Managed authorities require all original member confirmations. Test/application implementations in the workspace are adapted.
  • FleetEnrollmentJournal adds required atomic refuse_unexecuted_enrollment. The reference adapter implements it with the same registry-version transaction as acceptance/publication.
  • Configured fleet hosts must use install_fleet_node_durability_provider for follower production. Read-only providers return exact prepared inputs; transports pin original follower boots and authorities reconcile exact source epochs and original close receipts.
  • Persisted record formats, signed peer bytes and object paths are unchanged. Production qualification profiles and required evidence remain unchanged.

Focused verification for 489c5a9

Check Result
Complete host library target 20 passed; none filtered/ignored
Complete public host node target 81 passed; none filtered/ignored
Complete fleet example target 89 passed; none filtered/ignored
Isolated complete application integration target 29 passed; none filtered; 16 documented manual cases ignored
Host/application all-target Clippy and host API docs Passed with warnings denied
Format/diff, boundaries/layout, docs and SQL/peer gates Passed

219 distinct scoped cases passed, including six new cases, against one unchanged manifest of 618 Rust/Cargo paths including all lockfiles: SHA256 9eb1e0bef426cfd02bc3212e12d058fe7d123f7a0bafaa731a2f554d60d96fbf. Both drivers recheck the whole path set and source after every command. All indexed source and the entire staged tree match the verified isolated snapshot. Its final documentation resolves 1,152 local links/anchors, 110 Rust snippets, 28 SQL/peer assertions and 567 validator links. Logs/drivers/manifests are /tmp/cellule-follower-inventory-final-*. Active-checkout documentation includes separately edited control-plane files outside this diff.

New cases cover cursor/budget/hash contracts, paused preparation before a request, partial acceptance, preserved original members/proof identities, ledger refusal and exact charges, and original failed-member/error Arcs during requested rotation. The next native retry is paused before comparing captures. The existing deadline case requires RuntimeClosed when its original unresolved epoch remains retained; it does not infer empty coverage. Intermediate compilation and closed-ledger assertion failures, plus the earlier run before the deterministic retry gate, are archived and not added to this count.

The baseline publication-hint CI failure could not be reproduced: one initial selected call, 100 serial and 100 concurrent repetitions, ten complete macOS integration runs, and twenty complete two-CPU/four-GiB/zero-swap Linux ARM64 integration runs passed on baseline source. Linux image rust@sha256:0e2bcaef56d041a486784e54104a81aebe0da44bd03019bd70bc0401e42e4a97; binary SHA256 fe3f3c993893686b1c14b0adff85a1ce929382b85ded5fe7d366b7f4e08cb8f5. These repeats are diagnostic evidence, not extra distinct tests or a fix. Linux post-run source hashes were not retained. Original x86 CI failure and its unknown cause remain open. The regression now reports occurrence, partial/missing peers, elapsed time and pending transports without changing its timing or proof assertions.

Follower pages remain advisory interval captures. An idle protocol or zero epochs cannot prove a joined supervisor, complete native-role absence, current authority or replacement policy. Complete authenticated role envelopes and maintenance actions/finalization remain required; the reference collector remains incomplete. The new head requires independent CI qualification.

Focused verification for 8cff2c3

Rust/Cargo 1.97.0, all features and locked dependencies, CARGO_INCREMENTAL=0, and this checkout's Workspace target directory:

Check Result
Complete host library target 17 passed; none filtered/ignored
Complete public host node target 81 passed; none filtered/ignored
Complete fleet example target 86 passed; none filtered/ignored
Host all-target Clippy/API documentation Passed with warnings denied
Format/diff, boundaries/layout, Rust/Markdown and SQL/peer gates Passed

184 distinct scoped cases passed, including twelve new cases, against one unchanged manifest of 615 Rust/Cargo paths including all lockfiles: SHA256 bee6e0b5bdbe8c136830f152462bc7ad006840462ea906374c76bb03c25e9112. The driver rediscovers the full path set and verifies the manifest after every command; committed source matches all 615 entries. The isolated final documentation snapshot passes 1,151 local links/anchors, 110 Rust snippets, 28 SQL/peer assertions and 567 validator links. Active-checkout links additionally include the independently edited control-plane plan/audit; those files are outside this diff. Logs/manifests/reproduction are /tmp/cellule-reader-inventory-*.

New cases cover paused/lost acceptance, native/publication progress, accepted preparation without a row, stable/stale paging, memory refusal and exact charge release, real panic/unknown completion, joining state and original error Arcs. One fixture preserves 128 original long-partition requests across pages while final acceptance is paused, then requires all 128 native views, receipt-bound reads and zero native/retained/job/disk ledgers after joined shutdown. Its explicit larger bounds leave ordinary eight-slot/16-MiB retained/128-MiB native fixtures and all production profiles unchanged. Intermediate compile, runtime-closed, capacity, capture and manifest-driver failures are recorded in the evidence document and are not added to the final count. Broad/process/provider suites were not executed locally.

The original follower CI assertion read completion immediately after an 80 ms waiter deadline. It now awaits the original failure and retirement observation within the existing three-second fixture capture bound, preserving that deadline and all original proof/resource assertions. The original CI log does not establish which native phase remained pending. The new source still needs CI qualification.

These are advisory local producer pages. Complete authenticated role envelopes, current authority, leader/failed-session obligations, replacement policy, maintenance actions/finalization and full plan qualification remain required. The reference collector still reports incomplete role coverage.

Focused verification for 5872122

Rust/Cargo 1.97.0, all features and locked dependencies, CARGO_INCREMENTAL=0, and this checkout's Workspace target directory:

Check Result
Complete host library target 11 passed; none filtered/ignored
Complete public host node target 81 passed; none filtered/ignored
Complete fleet example target 80 passed; none filtered/ignored
Host all-target Clippy and API documentation Passed with warnings denied
Format/diff, boundaries/layout, Rust/Markdown and SQL/peer gates Passed

172 distinct scoped cases passed, including 17 new cases, against one unchanged manifest of 610 Rust/Cargo paths (including all lockfiles), SHA256 4892861ee8518547b233e6bbc5d454131112ab85729cdd6ebd48ba9062cfdefc. The final pipeline checks that manifest after every command; committed source matches it. New cases cover multiple pages (132 intents/130 enrollments, all statuses), reconstruction, original failed boots, lost acceptance replies, head/registry races, page continuations, aggregate limits, preserved backend errors, deadlines, signed boot coverage, omitted writer rows, enrollment during capture, and Pending enrollment with pressure relief. No production profile or required assertion was weakened. Initial compilation mistakes and the intermediate manifest selection are recorded in the evidence document. Broad/process/provider suites were not executed locally.

The reference collector remains incomplete for native reader/follower roles. This checkpoint proves roster traversal and stricter count inputs; it cannot finalize maintenance or complete the full plan.

Focused verification for 7375109

Rust/Cargo 1.97.0, all features, the lockfile, CARGO_INCREMENTAL=0, and this checkout's Workspace target directory:

Check Result
Complete public host node target 81 passed; none filtered/ignored
Complete fleet operations example target 71 passed; none filtered/ignored
Public runtime durability selection 24 passed; 176 filtered; none ignored
Runtime node library selection 101 passed; 408 filtered; none ignored
Runtime fleet operations selection 66 passed; 443 filtered; none ignored
Runtime admission selection 5 passed; 504 filtered; none ignored
Host/app all-target and runtime library/public runtime Clippy Passed with warnings denied
Host/runtime API docs Passed with warnings denied
Format, diff, boundaries/layout, docs and SQL/peer gates Passed

348 distinct cases passed against one unchanged manifest of 607 Rust/Cargo paths, SHA256 ce6f869aec2f6705a260aea141b3837c0db37e0d77cf07c7db9b956c02bb95c9. Earlier/intermediate runs are not added to this total. Nine new public example scenarios use signed directory authority and two native follower stores. They cover Pending-before-CAS, partial acceptance/receiver cordon, lost replies, failed member fences, cancelled/deadline drain waiters, invalid limits before acceptance, and an acknowledged SQL mutation with nonzero object coverage, exact-root restoration and original sys_requests response. Additional cases cover atomic exclusion/restart/races, configured-host provider rejection, refusal codecs, startup hold and evidence replay.

Opened reader cases still require Retired; the two explicit nonexecution cases require Refused and terminal replay. Intermediate failures and corrections are recorded in the evidence document. App process drivers compiled under Clippy; process/provider suites were not executed locally.

Current-head CI failures (489c5a9)

Workflow APIs confirm both failures on exact PR head 489c5a970e91c607dee1f42390a13fa64bcf6e6a:

  • Workspace job 110716061446 passes the full application integration target (29 passed/16 manual ignored), host library (20 passed), public node target (81 passed), and all six new follower inventory cases. The fleet example ends 87 passed, 2 failed: new_controller_adopts_lost_releases_after_real_expiry_and_joins_receiver_credit reports one lost release reply and one retired permit rather than two retained unconfirmed releases; byte_budget_yields_a_continuation_over_128_native_long_partition_readers returns the original retained Capacity("node pressure") at variable_rows.rs:19. The workspace stops at that example gate.
  • Contract job 110716061458 ends 28 application cases passed, 1 failed, 16 manual ignored. publication_hints_reach_readers_beyond_the_activation_concurrency times out on publication 3 after 2.001088492 seconds, with 10 of 19 healthy hints received, nine missing peers, no held peer and one intentionally pending transport. The enhanced assertion now narrows the failure phase; it does not establish the cause.

Original logs are /tmp/cellule-fleet-489c5a9-{workspace,contract}.log. Causes remain unestablished. No bound, production profile or required evidence is weakened. These failures stay open; the local 219-case pass cannot substitute for their qualification. Other current-head provider/performance campaigns need their own terminal evidence. The full goal remains active and this PR stays draft.

CI and readiness

This PR remains draft, and the full W1–W10 goal remains active. Complete authenticated revisioned role observations, failed-process/boot evidence, replacement-policy proof, maintenance role actions/finalization, remaining primitive/fault matrices (including Cron and Blob external owners), sustained convergence, complete maintenance/receiver-loss examples, process/provider/mixed-binary qualification and operator rollout/runbooks remain required. The reference collector still reports incomplete role coverage. This checkpoint cannot complete SettleRoles, Finalize or the full plan.

Baseline 8cff2c3 workspace CI job 110698431351 is terminal failure: cellule-app --test integration passed 28 cases, ignored 16 documented manual cases, and failed host::replicas::pending_reader_activation_retains_a_new_publication_hint at replicas.rs:285 with Elapsed(()). Its two-second bound requires publication wake-up before the five-second periodic tick; no timing/profile change was made. The workspace stopped before the host example target. Original logs are archived at /tmp/cellule-fleet-8cff2c3-workspace.log; cause remains unestablished and this failure stays open.

Baseline 8cff2c3 passes MSRV, both capacity campaigns (36962309457), contracts (36962309438), website (36962309421), fuzz (36962309504), fast/negative TLC (36962309590) and Compose smoke plus both routing comparisons (36962309460). Leased job 110698430939 completed success at 2026-10-02 04:23:26 UTC; object-only job 110698430922 at 04:41:35 UTC. Broad TLC/simulator remain skipped. Green baseline comparisons do not establish the historical failures’ causes or qualify the new head/full goal.

Baseline 5872122 passes MSRV, both capacity campaigns, contracts, website, fuzz smoke, fast/negative TLC and Compose smoke. Its workspace job 110679369745 failed after 79 example cases passed at the follower completion assertion described above; original logs remain archived.

Both baseline routing comparisons also completed failure after all eight functional executions in each mode passed. The unchanged median-of-four gate requires p95/p99 ≤ 110% and throughput ≥ 90%. Failures:

Case Throughput p95 p99
leased/local_query/c1 0.98936 1.07740 1.18368
object_only/local_query_expired_bursts/c16 1.00000 1.00871 1.15413

Leased job 110680072679 completed failure at 2026-10-02 03:11:43 UTC; object-only job 110680072513 at 03:12:18 UTC. Artifacts cell-routing-leased-36956132515-1 (ID 11207545900, ZIP SHA256 0d86d9cb16c83846e0432468b80425bcdd01f6fc02d9bb2e45bb3cb8b291f7f7) and cell-routing-object_only-36956132515-1 (ID 11206579342, ZIP SHA256 accb887edde0b11f11f9e02338a7e685847216a78be20a1608a6ca71f0fc7dcb) preserve raw rows/windows, logs, manifests and binaries. Both pin synthetic merge 4b6b091679636b3eeeba985e26f4c9a5164cc44c of PR 5872122 into fbfd84f9c4dfcb6de497072efd9aafa5a6f409cc, against frozen baseline 0dc04a658bd99668936f7ec58032d054f6fbc141. Candidate binary SHA256 is 5f2914db0810cce008b272fb2e27fb0dbfcd733c1d6bbbed65e1ace77a50f136; baseline is 08108609e742a3f8091616675152cab3de21f0e021f41588465d60b49a25a83c; harness is 425b55c168f55ef4b7395c69948027dd77b2bd72836e962daa399120d74149d7. The candidate binary matches the prior routing candidate exactly, which does not by itself establish these failures' cause. Original artifacts are archived under /tmp/cellule-fleet-5872122-routing-*. No profile or required evidence was weakened; the new head needs independent qualification.

Baseline 7375109 passes workspace/MSRV, both capacity campaigns, contracts, website, fuzz smoke, fast/negative TLC and Compose smoke. Its leased routing job 110671204057 completed failure at 2026-10-02 02:31:11 UTC, after all eight functional runs passed. The unchanged gate is median of all four runs: p95/p99 ≤ 110%, throughput ≥ 90%. Failing ratios:

Case Throughput p95 p99
leased/forwarded_command/c1 0.94804 0.94596 1.25613
leased/forwarded_command/c16 0.95954 1.15368 1.11897
leased/local_query_expired_bursts/c16 1.00000 1.05568 1.34762

Cause remains unestablished; no local process rerun or assertion change. Original artifact cell-routing-leased-36953458504-1, ID 11205313244, retains raw rows/windows, logs and frozen binaries; ZIP SHA256 dd328ccd12d670f1bcb2f05d3af441ab85abb26dd8a4577d1b524e24b880e724. Candidate source 55ae5c21290319e996b6c4b7506abf6ffdf1d00a is the synthetic merge of PR 7375109 into fbfd84f9c4dfcb6de497072efd9aafa5a6f409cc, compared with frozen baseline 0dc04a658bd99668936f7ec58032d054f6fbc141. Candidate binary SHA256 5f2914db0810cce008b272fb2e27fb0dbfcd733c1d6bbbed65e1ace77a50f136; baseline binary 08108609e742a3f8091616675152cab3de21f0e021f41588465d60b49a25a83c. Both use harness 425b55c168f55ef4b7395c69948027dd77b2bd72836e962daa399120d74149d7. Original logs and extracted evidence are archived under /tmp/cellule-fleet-7375109-routing*. Its object-only routing job 110671203955 completed success at 2026-10-02 02:35:43 UTC. The combined Compose workflow 36953458504 is terminal failure because the leased comparison failed. The new head requires independent qualification; these failures remain open evidence.

Baseline cc7aa30 passed workspace/MSRV, both capacity campaigns, contracts, website, decoder fuzz smoke, fast/negative TLC and Compose smoke. Broad TLC/simulator were skipped. Its routing comparison failed in job 110653610627. All 16 functional executions passed, but leased/local_command/c16 failed the unchanged comparison gate: candidate/baseline median throughput ratio 0.88824 (required ≥ 0.90), p95 ratio 1.12470 (required ≤ 1.10), and p99 ratio 1.06528. Candidate medians were 60.5975 commands/s and 262.4498545 ms p95, versus 68.222 commands/s and 233.351874 ms for baseline. Reads were 2192 for both. This does not establish the cause.

Artifact cell-routing-36947731323-1 (ID 11204931295) retains original rows, stage windows, logs and frozen binaries. ZIP SHA256 140559620e498df8bd134f9902e422cf95c5ba4658846ed15f463af716e32ed2; candidate source is synthetic merge 51a7ee50ed76d2a37311317d5df6291888714859 of cc7aa30 into 0dc04a658bd99668936f7ec58032d054f6fbc141. Baseline/candidate binary SHA256 values are 08108609e742a3f8091616675152cab3de21f0e021f41588465d60b49a25a83c and 86d341d694ebc8e114971b96e8f1133210438a2cb128d14520243e121b269e94; both use harness SHA256 425b55c168f55ef4b7395c69948027dd77b2bd72836e962daa399120d74149d7. Raw logs and extracted evidence are archived locally under /tmp/cellule-fleet-cc7aa30-routing*. The new head needs its own qualification; no assertion or profile was weakened.

Prior 6092203 Compose smoke failed in job 110643075559: three initial provider-backed cases passed, then mixed three-node reader traffic returned ReplicaUnavailable at process_scaling.rs:548. Artifact cell-reference-compose-36944383259-1 (ID 11201985421) retains original logs, TSVs, source and binary hashes. It pins synthetic merge source 68943f6c8b0e1bd7fd57e766590779c067983d34 of PR head 6092203 into base 0dc04a658bd99668936f7ec58032d054f6fbc141. ZIP SHA256 2ec63e359d13b01ca86ef2fd16ae3763d53e0517480e32d723772fc14a6c06da. Its routing job is authoritatively Cancelled. The cause remains unestablished; later green smoke does not identify it. Earlier capacity timing and routing/controller-restart failures remain recorded in the execution evidence. No qualification profile or required assertion was weakened.

Add journal-backed fleet operations, retained intents and enrollment contracts, resource-backed receiver preparation, finite host actions, fresh actor inspections, and a caller-driven movement controller. Include the local SQLite journal example, focused evidence, CI model coverage, and the full remaining fleet implementation plan.
Share the leased-node admission-pressure scenario between the executable and tests. Move a bounded journal-backed batch to two receivers, reopen the controller client, verify original outcomes and restored state, fence old source handles, and join all runtimes and journal jobs. Preserve incomplete production role coverage and remaining plan scope. Fix the owner-loss fixture to inspect the retained original fencing source and require empty resource ledgers.
…s-foundations

# Conflicts:
#	scripts/check-web-rust-examples.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant