Add verified packed-catalog storage and publication foundations

[forhappy]

Large Git histories need canonical object lookup and publication without a heap-sized OID inventory or authoritative per-object placement rows in the Repository Cell. This PR adds immutable pack/catalog storage, verified preparation, owner-fenced atomic publication, bounded directory maintenance and shared final-command dispatch. Long imports can now retain their creating namespace through staging and acquire a catalog generation floor after physical verification.

Resulting behavior

• Completed native capture now explicitly unlocks its exclusive fence only when the last owned file pin drops. Closing a CLOEXEC descriptor alone can leave an unrelated pre-exec child holding its lock and spuriously refuse the next native admission. The fix preserves queued-upload ownership, active-worker exclusion and descendant drain. A deterministic regression reproduces the defect before the fix on macOS and Linux and verifies release while an unrelated child remains paused.

• Adopted native inputs now publish through authenticated physical custody. CatalogPreparation::begin_retained_pack checks exact full-descriptor membership in the current successor checkpoint and freshly verifies its bound attempt/floor. A private proof gates closure verification while the ordinary raw namespace guard remains strict. Original native/metadata incarnations are preserved; new source-index nodes use the successor namespace. Catalog certificate v3 binds the immutable checkpoint digest through reconciliation, issuer checks and final transactional pin/MAC/expiry/authority checks. The old v2 certificate domain is rejected. Completed exact/logical recovery preserves original results. The shared bounded input-index cache reuses existing data structures; no per-object SQL inventory is introduced. Production producer selection, complete collection/isolated restore and large-team capacity qualification remain open.

• Durable native input checkpoints now reuse NativePackDescriptor, the shared RangeIndex/NodeRef and independent lease rows, with a bounded purpose-separated envelope and immutable digest. Command 29 and query 30 recheck current access, owner/attempt, expiry, scope and authentication. Staging and bound preparation recovery adopt the exact retained root without copying index nodes or uploading native pairs; final registration rechecks source custody and root equality atomically. ReadyStaging::claim retains exact SDK dispatch through cancellation and uncertain replies. StagingTicket::register_inputs now synchronously admits one checkpoint and its mutation identity into that same dispatcher; accepted registration precedes Bind or graceful stop, and a due renewal precedes registration. Dropped observers recover through pending_inputs, while unknown commands retain their exact evidence. One additional 4 KiB reservation bounds the admitted checkpoint slot. Known registration stores the original receipt before a fresh live query, preserving committed recovery while refusing revoked custody. Checkpoints establish descriptor custody; independent physical/canonical/closure verification remains required. Production staging wiring, exact bound-preparation registration supervision and collection integration remain open.

• Native staged receives now have an explicit run_native_receive API that retains pack/index pairs and shares the bounded CGI collector. stage_native_packs captures at most 32 request-private inputs under an admitted StagingContext, reuses native descriptors and the pinned-file uploader, and holds cache/disk ownership plus an exclusive native fence through queued work and cancellation. Scope, size, loose-input, checksum and active-worker failures reject before descriptors escape. A SHA-1/SHA-256 composition test receives through actual Git, stages authenticated pairs, drops receive/source caches, independently verifies, atomically publishes, and clones/fscks from a rebuilt cache selected through the committed catalog. The fixture supplies authority and orchestrates the APIs; production HTTP/SSH conversion remains open.

• Refused and empty pushes now issue a bounded, purpose-separated outcome proof directly from PreparationSession, without a catalog loader, upload, scratch or native worker. Session and catalog readers share the existing authoritative lease/deadline/renewal fence. Outcome proofs reuse command 19, response/options/signed-witness tables and foreground dispatch. New outcomes check current write access, owner/attempt/pin/expiry and immutable floor; completed recovery preserves original results. A moving catalog does not invalidate a ref-free outcome. Exact uncertain commands retain only the session rather than prepared catalog artifacts. Production producer conversion remains open.

• Native shutdown now permanently closes the shared admission pool across all cloned scopes, joins accepted/tracked work, and waits for healthy zero foreground and maintenance claims before Cell shutdown, workspace release, heartbeat stop and advertisement withdrawal. Startup failure after enrollment follows the same ordering. Canceled drain observers cannot reopen admission or release claims; poison, underflow and quarantined native owners keep drain unproven. Closed admission maps to HTTP 503.

• Every shared native Git spawn now requires a private four-dimension claim for process slots, CPU admission units, memory and descriptors. One pool is configured per node and shared through production gateways/caches/readers; preparation APIs require its explicit scope. Foreground and maintenance shares are disjoint. Claims follow the native drain guard through cancellation/quarantine; repack workers release after drain before validation and returned caches retain their original scope. Typed exhaustion becomes HTTP 503. Required native_limits is a configuration hard cutover; checked-in config producers/examples and benchmark metadata now supply it. These claims are estimates; OS containment, account/fair preparation scheduling and full-history qualification remain open.

• Native transport, verification, decoded-object/history, cache maintenance, blob extraction, candidates and ref-list workers now reuse one process ownership guard. Unix completion waits for inherited-descriptor EOF before leader reaping. Cancellation signals the unreaped private group and transfers the child, descriptor and generic owner to a bounded independent reaper; failed, saturated or shutdown drain quarantines owners. The fence proves drain only for participating descendants that preserve it. Hard OS CPU/RSS/file/process containment, fair account/preparation admission, profile qualification and equivalent non-Unix descendant drain remain open.

• Metadata, directory and closure construction reuse the existing SQLite tables and DiskBudget with admitted geometric growth. Empty default spools initially charge 192 KiB rather than 768 MiB. Fully rolled-back SQLite capacity failures can double the page cap only after disk admission; commit precedes cursor/digest advancement, and verified edge files rehash on every replay. Closure lookup creation and pending-degree initialization use indexed pages of at most 512. Immutable sealing shrinks to exact file bytes. Explicit ceilings, cancellation ownership and conservative cleanup remain enforced.

• Physical verification shares one admitted append-only dependency file per at-most-512-object metadata page, reducing dependency files from as many as 512 to one. Existing decoded witnesses retain exact private offset/length/digest ranges and rehash each range on SQL replay. Exclusive object-writer permits survive queued cancellation, storage failure poisons reuse, and full file credit remains held until the last producer/witness/worker drains. This is a per-page bound; global file/I/O admission and native profile qualification remain open.

• Ancestry reuses admitted SQLite growth and the existing visits/answers tables with a 192 KiB initial charge. A mutable walker borrow spans each traversal; memo answers bind to the exact catalog descriptor. Failure or cancellation permanently fences reuse, queued workers retain scratch credit through drain, and fresh lease checks cover identical tips and cached answers. Queue reset uses indexed transactions of at most 512 keys while preserving memo answers.

• File-backed SHA-1/SHA-256 native Git indexes, authenticated operation-scoped artifacts, canonical metadata segments, directory/source indexes and admitted catalog readers provide immutable lookup foundations. Physical pack verification and disk-backed canonical/dependency/closure verification compose through private factories; raw descriptors or caller-selected certification flags cannot authorize publication.

• A fresh schema removes legacy Git bodies and per-object placement. Independent attempt pins, immutable generation facts, monotonic creating namespaces, exact deferred operation/pin bindings and bounded indexed reaping preserve custody through Claim, Abort and owner restoration. SQL reaping does not authorize remote deletion.

• Staging reuses the same tokens, operation rows and lease rows with a NULL generation. Begin/Renew/Check/ClaimStaging retain inputs without holding intervening catalog facts. BindStaging selects the current floor once, preserves namespace and expiry, and needs no additional pin. A generated non-null binding value prevents SQLite's nullable composite-FK escape. Staging cannot open a preparation base or certify publication; the existing assembler accepts pre-bind physical witnesses after authoritative binding.

• StagingCoordinator owns Begin/Claim/Renew/RegisterStagedInputs/Bind commands and input producers independently of observers. Defaults bound 32 operations/eight per actor and 64 worker/result slots/eight per actor. Automatic renewal uses fresh authoritative queries; completed typed results remain charged until single handoff and are retrievable after a canceled observer. Seal blocks new workers and renews custody while slots drain before Bind. Failure cancels and joins producers and drops completed resources before credits. Unknown commands retain exact identities/evidence, sharing invocation/resolution with final publication; close/drain preserves uncertainty and original receipts.

• Trusted catalog/ref proofs bind exact ref-plan bytes, canonical membership and ancestry. Final commands recheck admitted owner fencing, current ACL/policies/checks, expected ref identities/versions and selected catalog CAS. Catalog, refs, exact native response, options and signed-push annotations commit atomically. Exact/logical recovery preserves original durable receipts. Reconciliation reuses physical inputs and admitted closure scratch against a queried current catalog.

• Streamed bounded output files share an indexed ingress root. StoredRun reuses physical file facts and logical RunCoverage; partial projections retain exact parent/prefix/suffix inventories. Directory-root v3/range-index v2 reject old layouts. Nodes remain within 64 KiB and point selection within 48 candidates; projections share physical cache admission.

• Certified compaction merges 2–32 ingress roots and supports bounded adjacent-level windows, retaining verified suffixes in existing physical files. Path-copy updates preserve unrelated runs, source/version identity, refs and old readers; reconciliation rejects changed or newly overlapping inputs. Geometric policy rotates ingress, levels and indexed cursors while bounding urgent bursts.

• One service-owned publication coordinator dispatches privately prepared pushes and compactions through typed outcomes and exact uncertainty recovery. Defaults reserve four of 32 slots for maintenance, cap maintenance waits at two of eight total, and allow at most three foreground starts before eligible maintenance. Separate class/account quotas and byte credits retain ambiguous work; proof ownership drops before admission is released. Cancellation cleanup releases private workspace roots before reader/disk credits.

Release scope and dependencies

Node configuration now requires native_limits; missing limits, unknown fields and negative capacities reject parsing, and invalid resource shares reject before workspace/provider work. The example and small evaluation profiles require measured hardware/OS headroom before production use.

The fresh schema and registry are not selected by production HTTP/SSH handlers yet. Remaining work includes producer/reader conversion, production staging lifecycle wiring, exact bound-preparation checkpoint supervision, full-history input/resource qualification and ancestry acceleration, continuous preparation/maintenance and CPU/I/O admission, publication progress against moving roots, native physical pack rewriting and accelerated reads, complete retained-root collection, isolated restore and deployment hard cutover. Outcome growth, provider durability/grouping and full-history Kubernetes/Linux/Chromium mixed-load campaigns remain unqualified. This PR does not claim capacity for 10,000+ engineers.

Binding preserves borrowed input expiry; a five-minute staging renewal can leave a five-minute catalog floor. Remaining-floor capacity and lease/admission configuration must be qualified before production selection. Ordinary direct preparation retains its two-command minimum; bulk staging adds Bind, measured renewals and each selected input checkpoint registration.

This branch includes the pack-storage prerequisite from Canopy #19, integrated with the Rust workspace. Cellule manifest/lockfile entries pin cea9b9a7913f88cca114a2010e6b5c0a0aacbcf0 from Cellule #38; that owner-fencing dependency remains a draft.

Validation

• Capture fence release increment (f59617d): all 380 unique workspace library tests passed locally (6 Git-format, 14 object-storage, 360 server; server 120.67s) and under an unprivileged Linux Docker user (same counts; server 147.38s). Linux used Rust 1.97.1/Git 2.39.5, while local macOS used Rust 1.98.0. The new fork regression, original capture rejection fixture, queued-upload ownership, actual receive/publication/cold clone and both permissions checks passed. The initial macOS regression failed because /bin/true was absent; correcting executable lookup exposed the intended WouldBlock failure before the production fix. The pre-fix Linux run passed 357 server tests and failed three (225.22s): the new regression and two permissions fixtures incorrectly run as root. Its original capture rejection fixture passed. This proves the inherited exclusive-lock defect; the precise phase of the earlier CI failure remains unproven. Diagnostic instrumentation was removed before final Linux validation and all-target Clippy (warnings denied, 29.98s). Formatting, diff, whitespace and local documentation targets passed; the disposable container was cleaned up. Fresh CI is pending. These checks do not qualify full-history import, production cutover, collection/isolated restore or large-team capacity.

• Retained physical custody increment (63299e3): all 379 unique workspace library tests passed (6 Git-format, 14 object-storage, 359 server; server 152.75s). Six new custody tests cover moving nonempty bases and cold Git clone in both formats, absent successor checkpoint, raw namespace rejection, absent inventory membership, exact index incarnation, signed digest substitution/old-domain refusal, and issuer/final-command revocation, expiry and Claim. The actual owner-restoration test now publishes after old-pin expiry and verifies independent pin/original checkpoint receipt recovery after completion. The initial run passed 378 tests and failed its post-publication active-query assertion: completion correctly retires the active operation. The corrected assertion checks active custody before publication and retained pin/exact receipt afterward; production guards and deadlines are unchanged. All-target workspace Clippy with warnings denied passed (1m 09s); formatting, diff, whitespace for 16 changed/new files and local documentation targets passed. Fresh CI is pending. This does not qualify production cutover or full-history/large-team capacity.

• Staging checkpoint supervision increment (c200d63): all 373 unique workspace library tests passed (6 Git-format, 14 object-storage, 353 server; server 145.02s). Five new service tests cover canceled observers, exact absent/lost/panicked dispatch in both formats, registration-before-Bind ordering, one-slot and foreign/duplicate/closed refusal without execution, committed recovery with revoked current access, and expiry after absence without attaching an inventory. The real receive/publication/cold-clone fixture uses supervised registration in both formats. The initial run passed 372 unique tests and failed the new expiry fixture (server 352 pass/one fail, 172.62s): changing only pin expiry violated the deferred operation/pin foreign key. The fixture now changes both expiries in one transaction, with production deadlines, guards and assertions unchanged. Final all-target workspace Clippy with warnings denied passed (1m 22s); formatting, diff, whitespace for 11 changed files and 18 local documentation targets passed. Both CI runs at c200d63 failed the existing native-capture rejection fixture with Input(Io(Kind(WouldBlock))) after 352 server tests passed and one failed (101.72s and 99.44s): run 1, run 2. The Linux capture-fence failure remains unresolved; local success is not a clean CI pass. These results do not qualify production cutover or full-history/large-team capacity.

• Input checkpoint increment (b30feb0): all 368 unique workspace library tests passed (6 Git-format, 14 object-storage, 348 server; server 126.73s). Seven new tests cover bounded multi-node inventories in both formats, exact replay/immutable SQL, authority and tamper rejection, source expiry, actual owner restoration with independent decoding of a retained real pack after old-pin expiry, bound preparation adoption without node copying, and Claim uncertainty after absent/lost replies and panic. The real receive/publication/cold-clone composition now checkpoints inputs in both formats. All-target workspace Clippy with warnings denied passed (37.16s), along with formatting, diff checks, whitespace for 15 changed/new files and 18 local documentation targets. Both CI runs passed at b30feb0, including workspace integrations, Python harness and RustFS compatibility. These are bounded correctness/composition checks, not full-history or large-team capacity qualification.

• Native-input increment: the initial workspace run passed all 361 unique library tests (6 Git-format, 14 object-storage, 341 server), two CLI checks, ten Directory Cell checks and two Git HTTP checks. All three new capture tests passed. Multi-server passed 84, failed 12 and ignored nine (420.76s), exposing legacy loose-file/external-blob fixture assumptions under globally forced packed receive, plus Cell deadlines and system-wide file exhaustion. Packed receive is now scoped to the new API pending production hard cutover. Final composition/rejection checks passed (1.75s), and queued-upload ownership passed separately. All 12 unchanged failing integration cases passed sequentially after the correction (360.52s). The original concurrent workspace run remains failed; this is not a clean full-workspace or capacity pass. Final queued-upload ownership passed (0.10s). Rebuilt owner-restart, Repository Cell and stock-Git smart-HTTP checks passed (4.11, 27.88 and 51.35s). Final workspace/all-target Clippy with warnings denied passed (33.40s). Formatting, diff checks, whitespace for all 12 changed/new files and 17 local documentation targets passed.

• Outcome-only increment: workspace library tests passed 357 and failed one existing native ancestry case with Base(Inactive) (server: 337 pass/one fail, 240.26s). All six new tests passed, covering both object formats, exact response-only persistence, moving/unavailable roots, payload/purpose rejection, write/read revocation, canceled observers, exact uncertain dispatch recovery and genuine owner restoration. The unchanged ancestry case passed in isolation (31.86s), with original assertions and lease deadline. The concurrent library run remains failed.

• Outcome-only increment owner-restart and Repository Cell checks passed (5.60 and 43.36s). Stock-Git smart-HTTP passed after rebuilding (80.15s); its first attempt could not execute because the test binary was missing from the shared target directory.

• Final cargo +1.98.0 clippy --workspace --all-targets --locked -- -D warnings passed (1m 12s). Formatting, diff checks, new-file whitespace and 31 local documentation links passed.

Earlier native shutdown increment:

• cargo +1.98.0 test --workspace --locked passed all 352 unique library tests (6 Git-format, 14 object-storage, 332 server), two CLI checks, ten Directory Cell checks and two Git HTTP checks. Five new resource tests cover terminal closure, admission/closure races, multiple and canceled observers, two-worker final-release races and poisoned/underflowed drain refusal. The new real-server check retains both native classes longer than one lease, verifies live renewal and workspace exclusion before Cell shutdown, cancels an observer, then verifies withdrawal and workspace reuse after release. Existing closed-stream and escaped-session fault fixtures now await the pool drain directly.
• The parallel multi-server run passed 94, failed two and ignored nine (403.96s). Large-object push returned HTTP 503 and bulk mirror returned HTTP 500; logs record Cell SQL deadlines and pending ingestion mutations. Both unchanged cases passed together sequentially (204.54s), with original assertions and deadlines. The mixed run remains failed; this is not a clean full-workspace or capacity pass.
• Owner-restart, Repository Cell and stock-Git smart-HTTP passed separately (4.06, 30.17 and 72.68s). These checks exercise the current production path; they do not qualify the missing packed-catalog cutover.
• Final cargo +1.98.0 clippy --workspace --all-targets --locked -- -D warnings passed (48.74s). Formatting, diff checks, new-file whitespace and 32 local documentation links passed. Workspace doctest targets passed with zero declared tests.
• The library suite includes native SHA-1/SHA-256 physical verification, canonical collision/typed closure rejection, admitted growth and rollback, shared edge-file witness replay, exact catalog ancestry, output projection/compaction inventories, staging renewal/ownership, account/class publication dispatch, owner-fenced atomic catalog/ref publication and original-receipt recovery. These are bounded primitive/composition checks, not full-history or 10,000-engineer throughput measurements.
• Both CI runs passed at native shutdown commit b36a3a6, including workspace integrations, Python harness and RustFS compatibility. Both CI runs also passed at outcome-only completion commit a34ea8d, including workspace integrations, Python harness and RustFS compatibility. Both CI runs passed at native-input capture commit 95ef9ea, including workspace integrations, Python harness and RustFS compatibility.

Design and remaining requirements

• Native input checkpoints and adoption

• Native receive input capture

• Outcome-only push completion

• Node native resource admission, configuration and claims

• Native process ownership and descendant drain

• Shared dependency spool ownership and replay

• Admitted SQLite construction growth

• Implementation status and executable implementation plan

• Large-team workload budgets and mandatory qualification

• Staged input retention and late binding

• Service-owned staging lifecycle

• Physical/logical run coverage, geometric maintenance, and shared dispatch