Skip to content

feat(protocol): publish Git and Xet state through capsules - #208

Open
forhappy wants to merge 46 commits into
crab-v2from
feat/request-minimal-protocol
Open

forhappy wants to merge 46 commits into
crab-v2from
feat/request-minimal-protocol

Conversation

@forhappy

@forhappy forhappy commented Sep 14, 2026 •

Copy link
Copy Markdown
Contributor

Status

Protocol-v2 capsule roots, per-ref publication, external Xet xorbs/shards, and stable layered packs are implemented. This PR is not production-qualified and v1 must remain available. Current head: c7c88bfd57d. Correctness and performance gates have not been relaxed.

Local RustFS 1.0.0 GA evidence

A fresh GitHub Kubernetes clone supplied a 5,000-commit first-parent replay. All 5,000 pushes, ten exact-tip fetch-before-repack intervals, strict Git fsck, seed/final remote Crab fsck, independent cold/warm clones, and sampled blob digests passed. Incremental push mean/p95 was 228/422 ms with 7.012 mean origin requests and no growth across 500-commit windows. Fetch p95 was 5.996 s but 34 origin requests versus the unchanged ≤10 gate. Full data and retained artifact hashes: crab/docs/benchmarks/capsule-v2-kubernetes-5000-rustfs-ga.md.

A matched, isolated 500-commit diagnostic compared the retained 32-way per-ref capsule compaction policy with an uncommitted four-way experiment on the same upstream commit range. Four-way reduced fetch requests from 32 to 14 and interval-repack requests from 63 to 27, but raised mean push requests from 7.012 to 7.488; observed local fetch time was 6.079 s versus 4.423 s. Both runs passed exact tips, clones, strict Git/Crab fsck, and sampled blob bytes, but both failed the ≤10 fetch-request gate. The fan-in experiment was reverted; single-pair latency differences are not treated as causal proof. Details and report hashes are in the same benchmark document.

A separate default 100 GiB Xet workload exercised three file versions, historical hydration, cross-repo dedup, restore, republish, exact bytes, and remote fsck. All 3,850 file comparisons passed. The harness still failed because the metered seed push saw three 60-second proxy timeouts on duplicate xorb PUTs. Path tracing reproduced those requests; direct and later metered conditional-PUT probes returned 412 promptly, but the original stall cause is not established. Cold 100 GiB hydrate took 37.9 minutes, so performance is not qualified. Full evidence: crab/docs/benchmarks/capsule-v2-xet-100g-rustfs-ga.md.

Open gates

  • Reduce the 500-commit incremental-fetch request p95 from 34 to ≤10 without weakening authentication, exact-tip binding, or cancellation.
  • Resolve the 100 GiB meter timeouts and read/hydrate cost; rerun the full workload cleanly.
  • Bring CI green. The current deterministic reds are the no-op-repack mirror fixture and stale maintenance-request/Cellule-label assertions. The SDK timing rerun passed; an image job is retrying after public registry rate limiting. New-head checks are pending; no green claim is made.
  • Complete matched-v1 performance, hosted-provider/platform, replica, tiering, mount, browser, backup/restore, restart, and reclamation parity before any v1 retirement.

Design and release gates: crab/docs/design/capsule-layered-packs.md and crab/docs/design/capsule-xorbs-shards.md.

@forhappy forhappy changed the title perf(protocol): cut object-store push requests below five average feat(protocol): publish Git and Xet state through capsules Sep 15, 2026
@forhappy
forhappy force-pushed the feat/request-minimal-protocol branch 3 times, most recently from 1962719 to 9850e91 Compare September 16, 2026 08:40
@forhappy

Copy link
Copy Markdown
Contributor Author

Follow-up qualification and hardening (commit 9393caf):

  • Terminal upload-pack now retains delta bases external only when the request is unfiltered, non-shallow, non-deepen, advertises thin-pack, and every client have is authenticated in the pinned v2 visibility plan. OFS deltas are rewritten to REF_DELTA; the full dependency sort/materialization path remains for filtered, shallow, incomplete, and legacy requests.
  • Catalog-selected objects retain authenticated physical pack order; legacy materialized selection keeps canonical OID ordering.
  • Fresh release binary v2-tiny-external-final-20260917 on RustFS: two incremental pushes 304–327 ms at 8 requests each; two fetches 225–247 ms at 9 requests; final clone 501 ms at 18 requests; matching tip and strict fsck passed.
  • Focused tests: crab-remote-git 133, crab-read 188, upload-pack wire 35; cargo check -p crab, release build, architecture gates, and format/diff checks pass.

The Kubernetes 5,000-commit stress artifact remains explicitly negative at the 500-commit fetch/repack boundary because the interrupted pre-fix repository state still requires a large historical pack; it is not being reported as parity proof. Hosted-provider, multipart, replica/tiering, mount/browser, S3 gateway, migration/recovery, and backup/restore rows remain release gates.

@forhappy

Copy link
Copy Markdown
Contributor Author

Parity follow-up (commit d6431f8): removed the stale client rejection for protected pushes carrying a v2 mirror-plan ID. The plan ID is already authenticated in CapsuleTransaction::for_plan; the auth-server capsule publisher commits the same transaction-scoped capsule plan receipt. The protected capsule receive test now exercises that planned transaction, verifies the receipt, and retries successfully. cargo test -p crab-auth-server --lib (101), capsule-push tests (9), and the release build pass.

@forhappy

Copy link
Copy Markdown
Contributor Author

Added external_thin_subset_pack_keeps_the_proven_base_outside_the_pack (commit 36f2ea6). It generates the public external-base thin-pack API from a real REF_DELTA fixture, verifies the one-object thin pack with strict git index-pack --fix-thin, and passes.

@forhappy

Copy link
Copy Markdown
Contributor Author

Documentation follow-up (commit 0233c25): the main capsule publication design now records the authenticated external thin-base rule and protected mirror-plan receipt path alongside the parity inventory and RustFS evidence.

@forhappy
forhappy force-pushed the feat/request-minimal-protocol branch from 0233c25 to 6ad3b0b Compare September 17, 2026 07:53
@forhappy

Copy link
Copy Markdown
Contributor Author

V1 parity pass

Implemented and pushed in 6ad3b0b0e9a (rebased onto current main):

  • Legacy read replicas now use the v1 manifest/index/object readiness contract only when v2/root is absent; a present or corrupt v2 root never downgrades. The resolver accepts verified legacy manifests, and tests cover both acceptance and corrupt-root rejection.
  • migrate import, migrate export, and adopt --rewrite-history now use the built-in verified fast-export/fast-import engine with v2 Xet staging/hydration. Shared Git blobs are converted inline only for selected paths; ref rollback is attempted on post-import failures, checkout failures are surfaced, and staging is closed before success.
  • Remote snapshot download/export, mount, hydrator, browser/HTTP, protected publication, and checkpoint readers carry one authenticated v2 view and immutable pointer catalog through reconstruction.
  • Documentation now contains a complete v1 product-parity inventory, cross-surface contracts, closure order, and Level-3 acceptance gates.

Proof after rebase:

  • cargo check -p crab --locked passed.
  • cargo test -p crab --locked --lib: 4,250 passed, 0 failed, 3 ignored.
  • cargo fmt --all -- --check, git diff --check, and python3 crab/scripts/check-architecture-gates.py passed.

Remaining release blockers are intentionally explicit: hosted-provider checksum/multipart and 5,000-commit current-format replay; managed replica failover/repair; tier/archive restore; mount range/cancellation/unmount; browser/HTTP load and fault matrix; S3 gateway operation/concurrency/restart matrix; lifecycle/workflow/admin inventory; backup/restore export inventory; and migration fault/resume/provider plus older-Git/interrupted/adversarial qualification. Full v1 production parity is not claimed until those Level-3 gates pass.

@forhappy

Copy link
Copy Markdown
Contributor Author

Parity closure update (c2d87d8):

  • Hydrate and remote mount now share one restore-availability adapter. It is built from the resolved physical store identity, so managed and replica read views target the bucket that owns the authenticated v2 catalog instead of reconstructing a provider from the logical crab:// URL.
  • --no-restore does not construct a cloud restore client, but archived shard/xorb reads still fail closed with the typed archive-class admission error.
  • A standalone crab:// mount now refuses to start if its authenticated v2 read context cannot be built; it no longer starts with stub readers and defers the failure until first pointer access. Local Git-native mount fallback is unchanged.
  • Design matrix and closure notes are updated in crab/docs/design/capsule-xorbs-shards.md.

Local proof after this change:

  • cargo test -p crab --locked --lib: 4,254 passed, 0 failed, 3 ignored.
  • cargo test -p crab --locked --lib cmd::mount: 120 passed before the final guard, plus the two new fail-closed/fallback tests passed individually.
  • cargo check -p crab --locked, cargo fmt --all -- --check, git diff --check, and python3 crab/scripts/check-architecture-gates.py all pass.

This closes the local wiring gap, but is not a claim of complete v1 production parity. Release gates remain: live S3/GCS/Azure lifecycle and restore behavior; replica readiness/failover/repair; restored-content verification; the full FUSE/NFS range/cache/cancellation matrix; browser and smart-HTTP load/fault coverage; S3 gateway restart/concurrency/request-count coverage; and migration, backup inventory, and delete/restore qualification on populated v1/v2 repositories.

@forhappy

Copy link
Copy Markdown
Contributor Author

Follow-up test hardening: the focused mount module now passes 122/122. I also serialized the unmount test's HOME override through the existing test guard; this removes a process-global HOME race that could make local_pipeline_config_rejects_active_cache flaky when mount tests ran in parallel. This is test-only and does not alter local Git-native fallback behavior.

@forhappy

Copy link
Copy Markdown
Contributor Author

Final local rerun after the test-only race fix: cargo test -p crab --locked --lib --quiet passed 4,254 tests (3 ignored) in 99.51s; cmd::mount passed 122/122. No working-tree changes are pending other than pre-existing generated Python __pycache__ directories, which were not added.

@forhappy
forhappy force-pushed the feat/request-minimal-protocol branch from 4b94ade to cc8700f Compare September 18, 2026 05:18
@forhappy

Copy link
Copy Markdown
Contributor Author

Layered-pack qualification update (commit 1640af9):

  • Fixed the ref-only layered-run regression: physical pack source validation now runs only for runs that carry Git members; ref-only runs still retain authenticated visibility and transition checks.
  • Exact release build verified on local RustFS/Kubernetes fixture: 1.1 GiB cold git clone --no-checkout completed in 3.75 s; default checkout clone completed in 8.04 s; exact tip 71f0fc6e72d53d5caf50b1314ca4d754463117f0; connectivity fsck passed; 26,892 files checked out.
  • Test proof: crab-read 200/200, remote-helper 139/139, metadata 210/210, release build, formatting, and diff checks pass.
  • The earlier ~20 s shared-volume result is destination write contention (RustFS and destination sharing the qualification volume), not object-store request or connectivity amplification.

The 5,000-commit replay with 500-commit fetch/repack checkpoints and hosted-provider matrix remain explicit release gates; v1 is not being retired until those pass.

@forhappy

Copy link
Copy Markdown
Contributor Author

Follow-up pushed in ee72375b4b09b3df74d90b15ffcecfcd5edd8868.

The first fresh PR-208 5,000-commit run found a correctness blocker before replay could continue: an append-only in-memory visibility dictionary could be non-canonical when serialized into the layered capsule (CRAB-E0020: layered visibility dictionary is not in canonical order). The fix canonicalizes the OID dictionary at the capsule boundary and remaps ref, transition, and history ordinals; it preserves the append-only runtime representation and keeps the decoder fail-closed.

Proof for the fix:

  • cargo test -p crab-metadata --lib --locked: 211 passed
  • cargo test -p crab-read --lib --locked: 200 passed
  • cargo test -p crab --lib git::remote_helper --locked: 139 passed
  • cargo fmt --all -- --check and git diff --check: passed

I am rebuilding the exact release binary from this commit and rerunning the fresh local-RustFS qualification. The earlier run remains recorded as a failed negative qualification at seed + replay 1; no 5,000-commit success is claimed until all replay, checkpoint fetch/repack, clone, and fsck gates pass.

@forhappy

Copy link
Copy Markdown
Contributor Author

Layered-pack follow-up pushed in 4ed9664ab70 (on top of 042c129cd55):

  • The writer now canonicalizes append-only visibility ordinals at the wire boundary and remaps refs, transitions, history closures, and authenticated member admission through the same old-to-new map. This closes the two fresh-replay correctness failures found after the ref-only fix.
  • crab-metadata passes 212/212 and crab-read passes 200/200 on the patched tree.
  • The local-RustFS smoke (seed + 10 replay pushes, fetch/repack checkpoints at 5 and 10) now passes every push, fetch, repack, tip, and fsck gate; incremental pushes remain about 1.04 s after the seed.
  • The design doc records the remaining cold-clone blocker: a multi-member checkpoint correctly declines the one-source/one-member direct installer, then the normal path performed 357,313 uncoalesced range reads in 163 s before the run was intentionally stopped (1.89 GB read, 15.2 GB inflated). This is not being reported as a cold-clone or 5,000-commit qualification pass.

The PR is updated with the fixes and evidence, but v1 is not retired and the 5,000-commit/multi-pack cold-clone gates remain open until the authenticated multi-member install or equivalent whole-member union path is implemented and requalified.

@forhappy

Copy link
Copy Markdown
Contributor Author

Post-push test completion for 4ed9664ab70:

  • cargo test -p crab-metadata --lib --locked: 212 passed
  • cargo test -p crab-read --lib --locked: 200 passed
  • cargo test -p crab --lib git::remote_helper --locked: 139 passed
  • cargo fmt --all -- --check and git diff --check: passed

The branch remains clean apart from the pre-existing untracked local crab.toml RustFS config, which is not part of the PR.

@forhappy

Copy link
Copy Markdown
Contributor Author

Updated in commit 1cfdd63 (perf(fetch): preload layered locators for cold clones).\n\nWhat changed:\n- Complete layered views now coalesce and authenticate index, reverse-index, and kind-bearing locator sidecars, then expose inline locators to the planner.\n- Ordinary incremental haves stay on the footer/tip-bound path; cold, filtered, shallow, and tag requests promote only when complete visibility is required.\n- Multi-member cold clones no longer fall back to per-object visibility reads; direct one-pack installation remains fail-closed.\n- Design history and qualification evidence are recorded in crab/docs/design/capsule-layered-packs.md.\n\nVerification:\n- crab-read: 200 passed.\n- remote-helper: 139 passed.\n- upload-pack wire: 39 passed.\n- Local RustFS, Kubernetes-derived fixture: blob:none clone 14.50 s with 173 ms planning; cache-miss shallow blob:none clone 6.12 s with 1,095 planning reads and 372 terminal response-pack reads; unfiltered multi-member clone 85.94 s, exact source tip, native git fsck --full clean.\n- Full, filtered, and shallow clone tips all match the source.\n\nThe fresh 5,000-push/fetch/repack matrix, hosted-provider latency, and v1-retirement gates remain open; this update does not claim those are complete.

@forhappy

Copy link
Copy Markdown
Contributor Author

CI follow-up: GitHub did not emit a synchronize run for the new head, so I manually dispatched the current commit (1cfdd63) against the repository workflows:\n- CI: https://github.com/crabbuild/crab/actions/runs/35675208869\n- Git protocol v2 partial-clone qualification: https://github.com/crabbuild/crab/actions/runs/35675210814\n- Large repository RustFS qualification: https://github.com/crabbuild/crab/actions/runs/35675212409\n\nAt this update they are queued, not yet green; the prior completed run was for an older head.

@forhappy

Copy link
Copy Markdown
Contributor Author

Pushed dba2cb4 (fix(protocol-v2): retain negotiated fetch haves).

What changed:

  • Accumulate and de-duplicate protocol-v2 haves across negotiation rounds before any tip-bound → complete-view promotion.
  • Copy the complete negotiated have set into the terminal done request, so incremental planning remains wants-minus-haves instead of regenerating the repository.
  • Added design evidence in crab/docs/design/capsule-layered-packs.md.

Proof:

  • cargo test -p crab --lib upload_pack_wire --locked: 39 passed
  • cargo test -p crab-read --lib --locked: 200 passed
  • cargo test -p crab --lib remote_helper --locked: 140 passed
  • Full-history K8s + local RustFS: incremental fetch after pushes 1 and 5 passed with exact tips and no missing objects; response packs were 49.9 MiB and 55.7 MiB instead of the prior 1.09 GiB complete response.

Qualification status remains honest: the replay later reached an existing 503,980,520-byte Crab/Xet pointer commit and stopped with CRAB-E0086 because the replay harness had not staged its local chunks. The 5,000-push/xorb qualification gate is still open; this is a staging-contract failure, not evidence that the haves fix is incorrect.

@forhappy

Copy link
Copy Markdown
Contributor Author

Pushed follow-up commit 1696f53 to PR 208.

The fresh full-history GitHub-origin Kubernetes RustFS qualification (pr208-v2-fresh-github-smoke2-20260921, binary dba2cb4) passed seed publication, protocol-v2 incremental fetches at pushes 1/5/10, exact tip checks, cold and warm full clones, and native fsck. Measured incremental fetches were 33.457s / 15,494 storage range reads / 50.1MB response at push 1 and 24.227s / 16,999 reads / 55.5MB response at push 5. Cold full clone was 217.5s with 10 store requests; warm clone was 99.8s with zero store requests.

The run stopped only at the blobless-clone qualification assertion blob-none-ordinal-metadata-lookup: the exact ordinal-metadata lookup branch emitted no locator_lookup_mode event, so the harness observed zero metadata events even though the request completed through the catalog-filter plan. The new commit adds that trace at the crab-metadata reader boundary; no data-path or authorization behavior changes. Focused crab-metadata tests pass (212/212).

Fresh workflows have been dispatched at this new head:

I am not marking the PR green until those runs complete.

@forhappy

Copy link
Copy Markdown
Contributor Author

Pushed f0db75ac994 to PR 208.

This fixes the v2 incremental-fetch regression caused by generation-owner checkpoint compaction: the control-only reader previously saw no live capsule transitions after compaction and fell back to a visibility traversal. Layered checkpoints now carry a bounded, authenticated recent per-ref transition suffix in the footer. Control-only fetches consume that suffix; older/incomplete have chains still fail closed to the existing catalog/traversal planner. The complete ordinal visibility body remains authoritative for cold/strict paths.

Proof:

  • cargo fmt --all --check
  • cargo test -p crab-metadata --lib --locked: 213 passed
  • cargo test -p crab-read --lib --locked: 200 passed
  • local RustFS tiny fixture, committed binary SHA matched source SHA; seed + 10 replay pushes, incremental fetches at 1/5/10, clones, and fsck reached exact tips. Incremental logs report strategy=tip_bound_transition, visibility_plan_ms=0, and exact-pack-member responses.
  • Qualification report is intentionally not called fully green: the tiny fixture's blob-none-ordinal-metadata-lookup check is false because that small clone selected the valid catalog_filter path; the incremental correctness gates all passed.

Fresh manual workflows for this commit:

@forhappy

Copy link
Copy Markdown
Contributor Author

Qualification update for f0db75ac994:

  • The local RustFS K8s replay is still running with the refreshed GitHub-origin source and 5,000 replay commits. It has completed 302 pushes so far with no push failure, missing-object error, or tip mismatch.
  • Seed publication, cold clone, and incremental fetch gates at pushes 1, 10, and 100 have passed with exact tips. At push 10, the v2 transition planner took 0 ms, used 3 object-store reads for 149 objects, and generated the response pack in about 30 ms.
  • This is not a green qualification result yet: the full replay, 500-push fetch/repack gate, final clone, and fsck are still pending.
  • The three fresh workflows for this head remain pending/queued: CI, protocol-v2 qualification, and large-repository RustFS qualification.

@forhappy

Copy link
Copy Markdown
Contributor Author

Update: pushed a1d745f41abc9234c18bdebfd4d092318fa913a8 (perf(metadata): bound visibility updates and fetch history).

  • crab-metadata lib tests: 214 passed; targeted visibility tests: 17 passed.
  • Release binary built from this commit.
  • Local RustFS/Kubernetes-shaped K8s qualification is running with the exact binary, seed + 5,000 first-parent pushes, fetch/repack every 500, and end-to-end verification.
  • At 500 pushes: 501/501 pushes succeeded; visibility planning used tip_bound_transition in 1 ms; incremental fetch completed in 6.95 s with 51 measured object-store requests (53 in the operation log). The old pre-change 500-fetch control was 69.7 s and 74,135 requests.
  • The remaining fetch latency is Git pack/index materialization; the v2 server still generated a response pack for this protocol-v2 request. The 5,000-commit run is intentionally still in progress, so this is not being claimed as final green qualification yet.

Known local gate: workspace cargo clippy -p crab-metadata --lib -- -D warnings still reports six pre-existing/unrelated lint failures in adjacent capsule/visibility code; no lint suppression was added.

@forhappy

Copy link
Copy Markdown
Contributor Author

Qualification update from the exact a1d745f binary: the 1,000-push checkpoint completed with 1,001/1,001 pushes successful. Owner/checkpoint maintenance was 608.9 s (two active packs, peak child RSS ~0.86 GiB). The 1,000-commit incremental fetch completed in 59.7 s, with 100 storage requests and 6.53 s pack generation; visibility planning remained 24 ms. This exposes the remaining v2 bottleneck clearly: the protocol-v2 server is still materializing response packs as the requested delta grows, so PR 208 is not yet a final sub-second/under-10-request qualification. The local 5,000 replay remains in progress.

@forhappy

Copy link
Copy Markdown
Contributor Author

Focused regression gate from a1d745f: cargo test -p crab-read --lib capsule_protocol --locked -j2 passed 21 tests (179 filtered), covering layered member admission, authenticated pack identity, coalesced/split range windows, control-only checkpoint reads, and incremental transition retention.

@forhappy

Copy link
Copy Markdown
Contributor Author

Qualification update: the 1,500-boundary checkpoint completed with 1,501/1,501 pushes successful. Owner maintenance was 511.8 s (peak child RSS ~1.69 GiB, two active packs). The incremental fetch then completed in 15.4 s with 146 storage requests and 17,445 logical objects; visibility planning was 1 ms. This reinforces the outstanding protocol-v2 response-pack scaling gap; correctness remains intact and the 5,000 replay is continuing.

@forhappy

Copy link
Copy Markdown
Contributor Author

Qualification update: the 2,000-boundary checkpoint completed with 2,001/2,001 pushes successful. Owner maintenance was 659.3 s (peak child RSS ~0.79 GiB, two active packs). Incremental fetch completed in 5.88 s with 126 storage requests and 20,279 logical objects; visibility planning was 3 ms. Fetch wall time varied versus the 1,500 sample, but the request count remains dominated by protocol-v2 response-pack reads. The 5,000 replay continues with no correctness failure.

@forhappy

Copy link
Copy Markdown
Contributor Author

Follow-up at 00b7177: source-matched release binary rebuilt and the full isolated RustFS 1.0.0 GA protocol-v2 partial-clone smoke passed all 92 checks. Actual filtered incremental fetch advertised version 2 / command=fetch; the preserved filtered-transfer-smaller gate now measures delivered Git pack bytes, 19,902 filtered (including initial lazy fetches) versus 188,716 full, rather than remote-reader counters that are zero on direct cold clones. Strict Git and lifecycle checks passed. Retained report SHA-256: 6d7dbc98a998c4233ca7d47e4da498d8a7326ce3c32c85626663d61fa62c8c55. Cell RustFS workflow now supplies the public-test environment scope; the new CI run is still underway. Kubernetes 5,000-push correctness remains passed but fetch p95 is 34 requests versus the unchanged 10-request gate.

@forhappy

forhappy commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor Author

Qualification update (head 1a0165b): the corrected layered-checkpoint reader contract passes locally, 38/38 crab-remote checkpoint tests. CI has restarted and is still in progress.

A fresh local RustFS GA Xet run exercised 40 GiB of large files across three versions (120 GiB logical history). It passed 96 checks through seed push, two incremental pushes, three layered repacks, retained refs/history, cross-repository chunk reuse, and byte-identical consumer hydration. Retained xorb bytes were 21.54 GB (16.7% of logical history); incremental pushes were 72 object-store requests each (large-file/xorb traffic, not the small Git-commit path). Seed upload was 749 requests.

This is partial qualification, not a 100 GiB pass: I interrupted the full cold-clone hydrate when the shared workspace reached its 20 GiB safety floor. The final report records that deliberate interruption as a failed run; no product corruption was observed before the stop. Full cold-clone, 100 GiB, v1 comparison, and the remaining red CI jobs are still open gates. I preserved run artifacts and remote objects.

@forhappy

Copy link
Copy Markdown
Contributor Author

Follow-up at head 158e104: the Xet qualification harness now releases only its own caches after each verified phase, budgets the modeled peak checkout/origin/cache footprint, and rechecks free space before every hydrate. A regression test reproduces the prior 40 GiB run being admitted at 153 GiB free; 31 local harness/meter tests pass. This is a safety and qualification-harness fix, not a completed 100 GiB run.

The retained 5,000-commit trace also confirms the final warm fetch is already one complete GET per each of 24 capsule sources plus eight control/admission requests. The unchanged ten-request gate still fails; reader range coalescing alone cannot solve that physical source floor. CI has restarted on this head.

@forhappy

Copy link
Copy Markdown
Contributor Author

Qualification update (c172bf1): the 500 MiB × 5-version RustFS 1.0.0 GA rehearsal passed all pushes/repack, independent consumer byte-identity, and first cold-clone hydrate; the second hydrate was stopped by the harness headroom guard (22,398,337,024 free < 22,523,412,480 required), not by data corruption. The harness now releases verified consumer worktrees, dehydrates the published source after copying the consumer fixture, and records available/required bytes. Focused Python tests: 33 passed. The source-dehydration E2E repeat and 100 GiB gate remain open; current mounted-volume free space is below the 24.6 GB minimum rehearsal preflight. Reports/logs retained, isolated disposable bucket/checkouts cleaned.

@forhappy

Copy link
Copy Markdown
Contributor Author

Qualification follow-up: the fresh xet-500m-source-dehydrate-20260928-r3 run passed on RustFS 1.0.0 GA with the pinned f16f5d71 binary: 158/158 checks, 174 commands, five versioned pushes/repack passes, cold and same-cache rehydration, historical byte checks, restore/republish, Git fsck and remote Crab fsck. No proxy errors. Report SHA-256 99bc1582ab80184490eaa97496e713fb8608e9cd5de96c3517d986cd0b443385; transport SHA-256 3385e7bee0ae6172b0280513db1571f271fe03c9a6bc3bdec4c4d24455f4bdc9. Isolated data cleaned, reports/logs retained. This validates the harness lifecycle fix at 500 MiB; the 100 GiB and broader release/parity gates remain open. I will batch the design-note update with the next PR push to avoid cancelling the CI currently running on c172bf1.

@forhappy

Copy link
Copy Markdown
Contributor Author

Qualification update (HEAD 9b91d0b): two fresh traced 1 GiB × five-version RustFS GA Xet runs passed 159 checks/174 commands each with zero proxy errors or 5xx responses. The first 1 GiB rehearsal had one proxy TimeoutError despite a passed user-visible report; this push adds a zero-proxy-error gate so that condition now fails qualification. Incremental Xet pushes in the latest clean repeat were 369–452 ms / 34 object-store requests. This is bounded lifecycle evidence, not completion of the 100 GiB, paired v1/v2, provider, or release gates. CI currently has a known mirror smoke fixture issue (a second repack does no work) and a prior public ECR rate-limit failure; both are being tracked explicitly.

@forhappy

Copy link
Copy Markdown
Contributor Author

Current-head local RustFS 1.0.0 GA diagnostic (binary SHA-256 733823bf5f68b22e5c24b9fea9127959425c1551ec5209c4c4a40fc673f4f59a): the mirror smoke reproducibly reached 117 checks / 449 commands, then failed its source-ahead stale-plan assertion. The preceding second repack was a no-op (packs 1→1, bytes read/written 0), so applying the still-valid plan succeeded. A focused mirror unit test confirms plan identity changes when the destination snapshot actually changes. This is fixture setup, not evidence that stale plans are accepted after a real metadata mutation. Separately, the in-memory repack request assertion reproduces 17 operations for the current two-phase logical+physical publication against its <=12 expectation; the publication/recovery tradeoff remains open. CI and the 100 GiB Xet gate are not yet complete.

@forhappy
forhappy changed the base branch from main to crab-v2 September 30, 2026 01:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant