diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 18dddb13..79184d4d 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -88,8 +88,9 @@ jobs: .github/workflows/ci.yml|Cargo.toml|Cargo.lock|crates/*/Cargo.toml) run_rustfs_ci=true ;; crates/omnigraph/src/storage.rs) run_rustfs_ci=true ;; crates/omnigraph/src/db/manifest.rs|crates/omnigraph/src/db/manifest/*) run_rustfs_ci=true ;; - crates/omnigraph/tests/s3_storage.rs|crates/omnigraph/tests/write_cost_s3.rs|crates/omnigraph/tests/lance_surface_guards.rs|crates/omnigraph/tests/memwal_enrollment_gate.rs|crates/omnigraph/tests/helpers/*) run_rustfs_ci=true ;; - crates/omnigraph/src/table_store.rs|crates/omnigraph/src/instrumentation.rs) run_rustfs_ci=true ;; + crates/omnigraph/src/db/omnigraph.rs|crates/omnigraph/src/db/omnigraph/stream_enrollment.rs|crates/omnigraph/src/db/omnigraph/stream_ingest.rs) run_rustfs_ci=true ;; + crates/omnigraph/tests/s3_storage.rs|crates/omnigraph/tests/write_cost_s3.rs|crates/omnigraph/tests/lance_surface_guards.rs|crates/omnigraph/tests/memwal_enrollment_gate.rs|crates/omnigraph/tests/memwal_stream.rs|crates/omnigraph/tests/memwal_stream_cost.rs|crates/omnigraph/tests/failpoints.rs|crates/omnigraph/tests/helpers/*) run_rustfs_ci=true ;; + crates/omnigraph/src/table_store.rs|crates/omnigraph/src/table_store/mem_wal.rs|crates/omnigraph/src/table_store/mem_wal/*|crates/omnigraph/src/instrumentation.rs) run_rustfs_ci=true ;; crates/omnigraph/src/runtime_cache.rs|crates/omnigraph/src/graph_index/*) run_rustfs_ci=true ;; crates/omnigraph-cluster/src/store.rs|crates/omnigraph-cluster/src/serve.rs) run_rustfs_ci=true ;; crates/omnigraph-cluster/tests/s3_cluster.rs) run_rustfs_ci=true ;; @@ -211,6 +212,7 @@ jobs: # omnigraph/failpoints, so each line rebuilds only its crate. run: | cargo test --locked -p omnigraph-engine --features failpoints --test failpoints + cargo test --locked -p omnigraph-engine --features failpoints --test memwal_stream set +e output=$(cargo test --locked -p omnigraph-engine --features failpoints --test memwal_stream_cost gate_r0_ -- --nocapture 2>&1); status=$? set -e @@ -243,6 +245,18 @@ jobs: || { echo "::error::required negative Gate R0 cell '${test_name}' did not execute as should-panic"; exit 1; } done + set +e + output=$(cargo test --locked -p omnigraph-engine --features failpoints --test memwal_stream_cost b2a_ -- --nocapture 2>&1); status=$? + set -e + echo "$output" + [ "$status" -eq 0 ] || exit "$status" + echo "$output" | grep -Eq "test result: ok\. 1 passed; 0 failed; 2 ignored" \ + || { echo "::error::B2a retained-history filter matched the wrong test set"; exit 1; } + echo "$output" | grep -Eq "test b2a_retained_history_small_depth_keeps_terms_and_old_roots_separate \.\.\. ok" \ + || { echo "::error::required B2a retained-history regression cell did not execute normally"; exit 1; } + echo "$output" | grep -Fq "B2a retained-history ci depth=" \ + || { echo "::error::B2a retained-history regression cell produced no evidence"; exit 1; } + # This is a remeasurement tripwire for the current single-exclusive- # fold shape, not a runtime allocator limit. Keep it outside the # gate_r0_ filter because the subprocess owner has a stable older name. @@ -512,13 +526,29 @@ jobs: # grep guards against the filter going vacuous (cargo passes # with 0 tests matched) if those tests are ever renamed. run: | + set +e output=$(cargo test --locked -p omnigraph-engine --features failpoints --test failpoints s3_ -- --nocapture 2>&1); status=$? + set -e echo "$output" [ "$status" -eq 0 ] || exit "$status" echo "$output" | grep -Eq "test result: ok\. [1-9][0-9]* passed" \ || { echo "::error::filter 's3_' matched no tests — vacuous pass"; exit 1; } + set +e + output=$(cargo test --locked -p omnigraph-engine --features failpoints --test memwal_stream s3_provider_shard_manifest_failure_retains_unreferenced_generation -- --exact --nocapture 2>&1); status=$? + set -e + echo "$output" + [ "$status" -eq 0 ] || exit "$status" + echo "$output" | grep -Fq "SKIP s3_provider_shard_manifest_failure_retains_unreferenced_generation" \ + && { echo "::error::RustFS B2a provider-failure cell skipped — vacuous pass"; exit 1; } + echo "$output" | grep -Eq "test result: ok\. 1 passed" \ + || { echo "::error::RustFS B2a provider-failure filter matched no test — vacuous pass"; exit 1; } + echo "$output" | grep -Eq "test s3_provider_shard_manifest_failure_retains_unreferenced_generation \.\.\. ok" \ + || { echo "::error::RustFS B2a provider-failure cell did not execute normally"; exit 1; } + + set +e output=$(cargo test --locked -p omnigraph-engine --features failpoints --test memwal_stream_cost gate_r0_widest_generation_closes_and_records_retain_all_growth_on_configured_rustfs -- --exact --nocapture 2>&1); status=$? + set -e echo "$output" [ "$status" -eq 0 ] || exit "$status" echo "$output" | grep -Eq "test result: ok\. 1 passed" \ diff --git a/AGENTS.md b/AGENTS.md index cb7f4128..7c35472d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -33,7 +33,7 @@ OmniGraph is a typed property-graph engine built as a coordination layer over ma - **Multi-modal querying**: vector ANN (`nearest`), full-text (`search`/`fuzzy`/`match_text`/`bm25`), Reciprocal Rank Fusion (`rrf`), and graph traversal (`Expand`, anti-join `not { … }`) in one runtime. - **Branches and commits across the whole graph**: Git-style — every successful publish appends to a commit DAG; merges are three-way at the row level. - **Atomic per-query writes**: `mutate_as` and `load` accumulate insert/update batches into an in-memory `MutationStaging.pending` per touched table. Strict insert and upsert both route through the sealed exact-`id`, filter-bearing adapter; bare Lance Append is test-only. Their RFC-022 adapter resolves or rejects relevant recovery intents before base capture, captures `(native branch id, exact graph_head, schema identity)`, then rechecks recovery and authority under schema → branch → table gates. Mutation/Load keeps one keyed transaction per table and rejects more than 8,192 rows or 32 MiB before recovery arm; mutation update scans stream into the remaining table budget after pending-key shadowing, with blob sizes checked before payload reads. It then arms an identity-bearing recovery-v9 sidecar containing exact Lance transaction identities + pre-minted lineage, commits each table with zero transparent conflict retries, confirms the achieved effects, and publishes under the same token. Unrelated retryable pre-effect authority movement may fully reprepare without changing logical mode. A proven effect-free strict conflict returns `KeyConflict` only after a fresh exact-ID probe; no exact match triggers bounded full strict-mode reprepare, while an effect-free upsert conflict also fully reprepares. Any earlier effect or ambiguity returns `RecoveryRequired` with the sidecar retained. Strict read-set conflicts return `ReadSetChanged`. Deletes stage through the same path. D₂ at parse time remains the constructive (insert/update) XOR destructive (delete) boundary. -- **RFC-026 private streaming core**: Phase A's main-only adapter enrolls one exact-`id` table into one empty unsharded Lance MemWAL under recovery-v10, then publishes its physical binding and `OPEN` lifecycle. Phase B1 adds a root-singleflight worker for one no-roll generation (8,192 rows / 32 MiB), watcher success plus the same writer's post-durability `check_fenced()` before clean acknowledgement, conservative replay/fold-only recovery, and one recovery-v11 strict exact-`id` fold whose sole `__manifest` CAS advances the table pointer, lifecycle witness/epoch floor, and graph lineage together. Cheap raw bounds and exact post-tombstone validation run before recovery; put ownership follows charge → shared admission → same-key queue → worker mode; cold replay installs exact fold-only accounting; exclusive admission spans seal, drain proof, table effect, and publication. Fold now charges the scanner's logical slices against the same 32-MiB generation cap and copies each scanner emission into dense owned arrays before retaining it, so the 8,192-row high-entropy near-cap closure cell folds and publishes successfully; its isolated fold RSS delta was 284,934,144 bytes (about 272 MiB), below the 384-MiB remeasurement tripwire. Post-invocation ambiguity is `AckUnknown`; unresolved recovery blocks progress. The topology remains main-only, unsharded, one resident writer and one live writer process, with no fresh reads or generation GC. The selected next profile is **unbounded retain-all**: OmniGraph imposes no retained-byte, object-count, file-count, or history quota and never deletes raw `_mem_wal` paths. Provider exhaustion is a loud storage failure and may halt admission, fold, or recovery progress; it never permits silent loss or publication of partial state. Row count, Arrow memory, deadlines, retries, and ambiguous outcomes remain bounded. RC.1's missing cross-open materialization-attempt receipt and complete physical-output envelope are therefore not blockers for this profile; they remain relevant to the future B2b managed-reclamation design. Compare-and-chain tokens, trusted attribution, lifecycle receipts, correction, authorization, and product parity remain required and inactive. The core remains crate-private behind one feature-gated, doc-hidden test seam; no schema-v9, SDK, CLI, HTTP, Cedar, or OpenAPI streaming contract ships from this result. +- **RFC-026 private streaming core**: Phase A's main-only adapter enrolls one exact-`id` table into one empty unsharded Lance MemWAL under recovery-v10, then publishes its physical binding and `OPEN` lifecycle. Phase B1 adds a root-singleflight worker for one no-roll generation (8,192 rows / 32 MiB), watcher success plus the same writer's post-durability `check_fenced()` before clean acknowledgement, conservative replay/fold-only recovery, and one recovery-v11 strict exact-`id` fold whose sole `__manifest` CAS advances the table pointer, lifecycle witness/epoch floor, and graph lineage together. Cheap raw bounds and exact post-tombstone validation run before recovery; put ownership follows charge → shared admission → same-key queue → worker mode; cold replay installs exact fold-only accounting; exclusive admission spans seal, drain proof, table effect, and publication. Fold now charges the scanner's logical slices against the same 32-MiB generation cap and copies each scanner emission into dense owned arrays before retaining it, so the 8,192-row high-entropy near-cap closure cell folds and publishes successfully; its isolated fold RSS delta was 284,934,144 bytes (about 272 MiB), below the 384-MiB remeasurement tripwire. Post-invocation ambiguity is `AckUnknown`; unresolved recovery blocks progress. The topology remains main-only, unsharded, one resident writer and one live writer process, with no fresh reads or generation GC. The private **B2a unbounded retain-all gate is implemented**: OmniGraph imposes no retained-byte, object-count, file-count, or history quota and never deletes a canonical durable `_mem_wal` object. Lance may clean only its losing `.binpb.tmp.` atomic-CAS staging. Complete and partial unreferenced generation residue stays non-authoritative and is never descended into, read, mutated, adopted, or deleted through recovery/reopen; local and configured-RustFS provider-failure cells prove loud typed outcomes. The 1/8/32/128 history instrument keeps warm-ack operation count flat while exposing growing serialized authority and combined fold/history cost; its LIST totals, wall time, and RSS are advisory, not quotas or SLOs. Provider exhaustion may halt admission, fold, or recovery progress; it never permits silent loss or publication of partial state. Row count, Arrow memory, deadlines, retries, and ambiguous outcomes remain bounded. RC.1's missing cross-open materialization-attempt receipt and complete physical-output envelope are therefore not blockers for this profile; they remain relevant to the future B2b managed-reclamation design. Compare-and-chain tokens, trusted attribution, lifecycle receipts, correction, authorization, and product parity remain required and inactive. The core remains crate-private behind one feature-gated, doc-hidden test seam; no schema-v9, SDK, CLI, HTTP, Cedar, or OpenAPI streaming contract ships from this result. - **HTTP server**: Axum + utoipa OpenAPI, bearer auth (SHA-256 hashed, optional AWS Secrets Manager). Cedar policy enforcement is engine-wide — every `_as` writer calls `Omnigraph::enforce(action, scope, actor)`, so HTTP, CLI, and embedded SDK consumers all hit the same gate. **Cluster-only boot** (RFC-011): the server always boots from a cluster directory (`--cluster `, RFC-005) and serves N graphs (N ≥ 1) under multi-graph routes (`/graphs/{graph_id}/...` + read-only `GET /graphs` enumeration); there are no single-graph flat routes and no positional-URI boot. Per-graph + server-level Cedar policies. Runtime add/remove (`POST /graphs`, `DELETE /graphs/{id}`) is not exposed — operators run `cluster apply` and restart. - **CLI** with two-surface config (RFC-007/008): the team-owned cluster directory (`cluster.yaml`) plus the per-operator `~/.omnigraph/config.yaml` (servers, clusters, credentials, actor, profiles, aliases, defaults). Graphs are addressed via `--store`/`--server`/`--cluster`/`--profile`/operator defaults (RFC-011). Multi-format output (json/jsonl/csv/kv/table). @@ -280,7 +280,7 @@ omnigraph policy explain --cluster ./company-brain --graph knowledge --actor act | Cleanup (`cleanup_old_versions`) | ✅ | `omnigraph cleanup` derives requested `--keep` / `--older-than` cutoffs from each table's available versions; Lance refs plus OmniGraph's live-lazy-branch and recovery floors may retain additional versions. It fails closed on unopenable pins, recovery intent, or uncovered main-table HEAD drift | | BTREE / inverted (FTS) / vector indexes | ✅ | `@index`/`@key` declares intent; the physical index is derived state that never fails a logical op. Built per column through one chokepoint (`build_indices_on_dataset_for_catalog`, type-dispatched by `node_prop_index_kind`: enum + orderable scalar → BTREE, free-text String → FTS, Vector → vector); idempotent; lazy across branches. **Schema apply and mutation/load build no indexes inline**: they publish only logical data/schema effects, leaving physical intent pending. `ensure_indices` first runs the roll-forward-only recovery barrier before base capture or planning, then materializes every declared-but-missing artifact for one table through one staged mixed CreateIndex transaction under its identity-bearing recovery-v9 authority/lineage/delta envelope; it continues to report untrainable Vector columns as pending. `Armed` is rollback-only, `EffectsConfirmed` rolls forward only while captured authority holds, and first-touch refs carry exact identity. `optimize` separately restores fragment coverage through its bounded identity-bearing v9 maintenance envelope. | | Strict insert / upsert ingestion | ✅ transaction conflict filters + uncommitted fragment staging | Internal schema v6 introduced the explicit logical mode and v8 preserves it. General strict insert and upsert use the sealed exact-`id`, forced-v2 MergeInsert adapter; strict insert exact-probes its pinned parent before minting `omnigraph.insert_absence=v1`, while an all-new upsert may mint the same optional certificate only from its completed effect statistics. Mutation/Load remains one transaction per table, capped before arm at 8,192 rows / 32 MiB. BranchMerge's proven all-new route accepts only a complete certificate chain plus final source/target native-incarnation checks; it stages bounded fragments with `InsertBuilder`, commits them as exact-`id` filtered `Update` transactions, and performs zero target preflights, target merge joins, or committed Appends. Missing or malformed proof falls back to the general ordered diff. Raw Lance writers are outside the supported graph-writer topology, and the certificate is an internal, non-cryptographic capability. The final five-pair production gate passed at 10K (3.875× median; 24,297,472-byte max paired RSS overhead) and 100K (~3.886×; 32,604,160 bytes). | -| MemWAL streaming foundation | ✅ MemWAL system index, durable WAL generations, epoch-fenced shard writer | Internal schema v8 carries exact identity-keyed physical binding, current-HEAD witness, `OPEN`/`DRAINING`/`SEALED` state, per-shard epoch floor, and stream-config v2. Phase A's recovery-v10 adapter enrolls one empty unsharded shard. Private B1 adds one root-singleflight, no-roll generation capped at 8,192 rows / 32 MiB; watcher success plus same-writer post-durability epoch check; detached ownership; fold-only replay; seal/drain proof; and recovery-v11 exact-transaction fold followed by one graph-visibility CAS. The fold charges logical slices and densifies each scanner emission, so the deterministic 8,192-row/near-32-MiB high-entropy generation now closes; the measured isolated fold RSS delta is 284,934,144 bytes (about 272 MiB), below the 384-MiB remeasurement tripwire. It performs no generation GC or fresh reads. The selected unbounded retain-all profile has no OmniGraph byte/object/file/history quota and no raw `_mem_wal` deletion; provider exhaustion fails loudly and may stop progress. Row/logical-Arrow admission, deadline, retry, and ambiguity bounds remain. B2b retains the future Lance-owned reclamation design. Common schema-v9/config-v3/state-v2/recovery-v12 enrollment/token/attribution/lifecycle/correction contracts remain inactive. There is no public stream schema, enrollment, put/fold/status API, Cedar action, CLI, HTTP route, or OpenAPI contract. | +| MemWAL streaming foundation | ✅ MemWAL system index, durable WAL generations, epoch-fenced shard writer | Internal schema v8 carries exact identity-keyed physical binding, current-HEAD witness, `OPEN`/`DRAINING`/`SEALED` state, per-shard epoch floor, and stream-config v2. Phase A's recovery-v10 adapter enrolls one empty unsharded shard. Private B1 adds one root-singleflight, no-roll generation capped at 8,192 rows / 32 MiB; watcher success plus same-writer post-durability epoch check; detached ownership; fold-only replay; seal/drain proof; and recovery-v11 exact-transaction fold followed by one graph-visibility CAS. The fold charges logical slices and densifies each scanner emission, so the deterministic 8,192-row/near-32-MiB high-entropy generation now closes; the measured isolated fold RSS delta is 284,934,144 bytes (about 272 MiB), below the 384-MiB remeasurement tripwire. It performs no generation GC or fresh reads. The private B2a gate implements unbounded retain-all: no OmniGraph byte/object/file/history quota; no canonical durable `_mem_wal` deletion; typed provider failure; and complete/partial orphan residue retained, non-authoritative, and untouched below its root through retry/reopen. Lance-owned losing manifest temp-file cleanup is allowed. The 1/8/32/128 local/RustFS instrument is advisory and exposes growing authority/history cost without creating a threshold. Row/logical-Arrow admission, deadline, retry, and ambiguity bounds remain. B2b retains the future Lance-owned reclamation design. Common schema-v9/config-v3/state-v2/recovery-v12 enrollment/token/attribution/lifecycle/correction contracts remain inactive. There is no public stream schema, enrollment, put/fold/status API, Cedar action, CLI, HTTP route, or OpenAPI contract. | | Vector search | ✅ | `nearest()` query op; embedding pipeline (Gemini / OpenAI clients); `@embed` in schema | | Full-text search | ✅ | `search/fuzzy/match_text/bm25` query ops | | Hybrid ranking | — | `rrf(...)` Reciprocal Rank Fusion in one runtime | diff --git a/crates/omnigraph/tests/forbidden_apis.rs b/crates/omnigraph/tests/forbidden_apis.rs index 8fddee25..aaffa627 100644 --- a/crates/omnigraph/tests/forbidden_apis.rs +++ b/crates/omnigraph/tests/forbidden_apis.rs @@ -1084,15 +1084,116 @@ fn collect_durable_use_renames(tree: &syn::UseTree, hits: &mut Vec) { } } +fn collect_retain_all_delete_use_renames(tree: &syn::UseTree, hits: &mut Vec) { + let tracked = retain_all_tracked_delete_names(RETAIN_ALL_DELETE_CALLS); + match tree { + syn::UseTree::Rename(rename) if tracked.contains(&rename.ident.to_string()) => { + hits.push(format!( + "delete gateway `{}` renamed to `{}`", + rename.ident, rename.rename + )); + } + syn::UseTree::Path(path) => { + collect_retain_all_delete_use_renames(&path.tree, hits); + } + syn::UseTree::Group(group) => { + for item in &group.items { + collect_retain_all_delete_use_renames(item, hits); + } + } + _ => {} + } +} + +fn expression_shape(expression: &Expr) -> String { + match expression { + Expr::Path(path) => path + .path + .segments + .iter() + .map(|segment| segment.ident.to_string()) + .collect::>() + .join("::"), + Expr::Reference(reference) => format!( + "&{}{}", + if reference.mutability.is_some() { + "mut " + } else { + "" + }, + expression_shape(&reference.expr) + ), + Expr::Field(field) => { + let member = match &field.member { + syn::Member::Named(identifier) => identifier.to_string(), + syn::Member::Unnamed(index) => index.index.to_string(), + }; + format!("{}.{}", expression_shape(&field.base), member) + } + Expr::Call(call) => format!("{}(..)", expression_shape(&call.func)), + Expr::MethodCall(call) => { + format!("{}.{}(..)", expression_shape(&call.receiver), call.method) + } + Expr::Paren(paren) => expression_shape(&paren.expr), + Expr::Group(group) => expression_shape(&group.expr), + Expr::Await(await_expression) => { + format!("{}.await", expression_shape(&await_expression.base)) + } + Expr::Index(index) => format!("{}[..]", expression_shape(&index.expr)), + Expr::Lit(_) => "".to_string(), + Expr::Macro(_) => "".to_string(), + _ => "".to_string(), + } +} + +fn argument_shape(arguments: &Punctuated) -> String { + arguments + .iter() + .map(expression_shape) + .collect::>() + .join(", ") +} + +fn call_shape(target: &Expr, arguments: &Punctuated) -> String { + format!( + "{} => {}", + expression_shape(target), + argument_shape(arguments) + ) +} + +fn impl_owner(ty: &Type) -> String { + type_final_ident(ty) + .map(|identifier| identifier.to_string()) + .unwrap_or_else(|| "".to_string()) +} + #[derive(Default)] struct CallInventory { counts: BTreeMap, + impl_stack: Vec, + function_stack: Vec, + call_shapes_by_function: BTreeMap<(String, String, String), usize>, macro_hits: Vec, + retain_all_hidden_route_hits: Vec, } impl CallInventory { fn record(&mut self, key: impl Into) { - *self.counts.entry(key.into()).or_default() += 1; + let key = key.into(); + *self.counts.entry(key).or_default() += 1; + } + + fn record_shape(&mut self, key: impl Into, shape: String) { + let function = self + .function_stack + .last() + .cloned() + .unwrap_or_else(|| "".to_string()); + *self + .call_shapes_by_function + .entry((function, key.into(), shape)) + .or_default() += 1; } } @@ -1108,21 +1209,32 @@ impl<'ast> Visit<'ast> for CallInventory { if cfg_requires_test(&node.attrs) { return; } + self.function_stack.push(node.sig.ident.to_string()); visit::visit_item_fn(self, node); + self.function_stack.pop(); } fn visit_item_impl(&mut self, node: &'ast syn::ItemImpl) { if cfg_requires_test(&node.attrs) { return; } + self.impl_stack.push(impl_owner(&node.self_ty)); visit::visit_item_impl(self, node); + self.impl_stack.pop(); } fn visit_impl_item_fn(&mut self, node: &'ast syn::ImplItemFn) { if cfg_requires_test(&node.attrs) { return; } + let function = self + .impl_stack + .last() + .map(|owner| format!("{owner}::{}", node.sig.ident)) + .unwrap_or_else(|| node.sig.ident.to_string()); + self.function_stack.push(function); visit::visit_impl_item_fn(self, node); + self.function_stack.pop(); } fn visit_item_macro(&mut self, node: &'ast syn::ItemMacro) { @@ -1134,6 +1246,7 @@ impl<'ast> Visit<'ast> for CallInventory { fn visit_item_use(&mut self, node: &'ast syn::ItemUse) { collect_durable_use_renames(&node.tree, &mut self.macro_hits); + collect_retain_all_delete_use_renames(&node.tree, &mut self.retain_all_hidden_route_hits); visit::visit_item_use(self, node); } @@ -1153,6 +1266,14 @@ impl<'ast> Visit<'ast> for CallInventory { fn visit_expr_method_call(&mut self, node: &'ast syn::ExprMethodCall) { let method = node.method.to_string(); self.record(method.clone()); + self.record_shape( + method.clone(), + format!( + "{} => {}", + expression_shape(&node.receiver), + argument_shape(&node.args) + ), + ); if method == "append" && node.args.len() == 2 { self.record(".raw_dataset_append("); } @@ -1170,6 +1291,7 @@ impl<'ast> Visit<'ast> for CallInventory { fn visit_expr_call(&mut self, node: &'ast syn::ExprCall) { if let Some(identifier) = final_path_ident(&node.func) { self.record(identifier.clone()); + self.record_shape(identifier.clone(), call_shape(&node.func, &node.args)); if identifier == "append" && node.args.len() == 3 { self.record(".raw_dataset_append("); } @@ -1203,6 +1325,12 @@ impl<'ast> Visit<'ast> for CallInventory { .last() .map(|segment| segment.ident.to_string()) .unwrap_or_else(|| "macro".into()); + for gateway in retain_all_tracked_delete_names(RETAIN_ALL_DELETE_CALLS) { + if tokens.contains(&format!("{gateway} (")) { + self.retain_all_hidden_route_hits + .push(format!("{macro_name}! may hide delete gateway `{gateway}`")); + } + } if macro_name == "include" { self.macro_hits .push("include! can hide unparsed durable calls".into()); @@ -1273,6 +1401,304 @@ fn call_inventory(ast: &syn::File) -> CallInventory { inventory } +/// Raw MemWAL storage is retained indefinitely in RFC-026's selected B2a +/// profile. These are the only production functions allowed to spell the +/// physical `_mem_wal` directory: both perform bounded, read-only validation. +/// A new owner is therefore an architectural change, not an incidental helper. +const RAW_MEM_WAL_PATH_OWNERS: &[(&str, &str, usize)] = &[ + ("table_store/mem_wal.rs", "raw_mem_wal_inventory", 2), + ( + "table_store/mem_wal/worker.rs", + "validate_b1_lifecycle_physical_state", + 1, + ), +]; + +/// Generic graph maintenance must reason from manifest/lifecycle authority, +/// not inspect or reinterpret retained MemWAL objects. StreamEnrollment and +/// StreamFold recovery are intentionally absent: their exact classifiers must +/// read MemWAL state, while the raw-path and no-reclamation guards below keep +/// those classifiers non-destructive. +const RETAIN_ALL_MAINTENANCE_FILES: &[&str] = &[ + "db/omnigraph/optimize.rs", + "db/omnigraph/repair.rs", + "db/omnigraph/schema_apply.rs", + "db/omnigraph/table_ops.rs", +]; + +const RETAIN_ALL_OMNIGRAPH_METHODS: &[&str] = &[ + "cleanup", + "cleanup_deleted_branch_tables", + "optimize", + "repair", +]; + +/// Raw object/prefix erasure is the only delegated route this B2a guard owns. +/// Lance version GC and branch-tree lifecycle remain covered by their existing +/// protocol registries and behavioral guards. +const RETAIN_ALL_RAW_DELETE_PRIMITIVES: &[&str] = &[ + "delete", + "delete_prefix", + "remove_dir", + "remove_dir_all", + "remove_file", +]; + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum RetainAllDeleteScope { + /// Accepts a caller-provided object/prefix. Registering the gateway also + /// enrolls its terminal function name so every caller edge is inventoried. + GenericGateway, + /// Deletes one protocol-derived schema, sidecar, staging, or lock object. + ExactControlObject, + /// Deletes an unregistered first-touch table root only after exact recovery + /// ownership and competing-sidecar proof. + FirstTouchRollback, + /// Destroys the entire graph after the governed cluster approval protocol. + WholeGraphDestroy, +} + +type RetainAllDeleteCallsite = ( + &'static str, + &'static str, + &'static str, + &'static str, + usize, + RetainAllDeleteScope, +); + +#[rustfmt::skip] +const RETAIN_ALL_DELETE_CALLS: &[RetainAllDeleteCallsite] = &[ + ("storage.rs", "ObjectStorageAdapter::delete", "delete", "self.store => &location", 1, RetainAllDeleteScope::GenericGateway), + ("storage.rs", "ObjectStorageAdapter::delete_prefix", "delete", "self.store => &location", 1, RetainAllDeleteScope::GenericGateway), + ("storage.rs", "ObjectStorageAdapter::delete_prefix", "remove_dir_all", "tokio::fs::remove_dir_all => &path", 1, RetainAllDeleteScope::GenericGateway), + ("instrumentation.rs", "CountingStorageAdapter::delete", "delete", "self.inner => uri", 1, RetainAllDeleteScope::GenericGateway), + ("instrumentation.rs", "CountingStorageAdapter::delete_prefix", "delete_prefix", "self.inner => prefix_uri", 1, RetainAllDeleteScope::GenericGateway), + ("db/omnigraph.rs", "best_effort_cleanup_init_artifacts", "delete", "storage => &uri", 1, RetainAllDeleteScope::ExactControlObject), + ("db/schema_state.rs", "cleanup_staging_files", "delete", "storage => &schema_source_staging_uri(..)", 1, RetainAllDeleteScope::ExactControlObject), + ("db/schema_state.rs", "cleanup_staging_files", "delete", "storage => &schema_ir_staging_uri(..)", 1, RetainAllDeleteScope::ExactControlObject), + ("db/schema_state.rs", "cleanup_staging_files", "delete", "storage => &schema_state_staging_uri(..)", 1, RetainAllDeleteScope::ExactControlObject), + ("db/manifest/recovery.rs", "delete_sidecar", "delete", "storage => &handle.sidecar_uri", 1, RetainAllDeleteScope::ExactControlObject), + ("db/manifest/recovery.rs", "delete_sidecar_by_operation_id", "delete", "storage => &sidecar_uri(..)", 1, RetainAllDeleteScope::ExactControlObject), + ("db/manifest/recovery.rs", "roll_back_schema_apply_v7", "delete_prefix", "storage => &pin.table_path", 2, RetainAllDeleteScope::FirstTouchRollback), + ("omnigraph-cluster/store.rs", "StateLockGuard::drop", "remove_file", "std::fs::remove_file => path", 1, RetainAllDeleteScope::ExactControlObject), + ("omnigraph-cluster/store.rs", "StateLockGuard::drop", "delete", "adapter => &uri", 2, RetainAllDeleteScope::ExactControlObject), + ("omnigraph-cluster/store.rs", "ClusterStore::force_unlock", "delete", "self.adapter => &lock_uri", 1, RetainAllDeleteScope::ExactControlObject), + ("omnigraph-cluster/store.rs", "ClusterStore::try_delete_object", "delete", "self.adapter => uri", 1, RetainAllDeleteScope::GenericGateway), + ("omnigraph-cluster/store.rs", "ClusterStore::delete_object", "try_delete_object", "self => uri", 1, RetainAllDeleteScope::GenericGateway), + ("omnigraph-cluster/store.rs", "ClusterStore::delete_graph_root", "delete_prefix", "self.adapter => graph_uri", 1, RetainAllDeleteScope::GenericGateway), + ("omnigraph-cluster/lib.rs", "apply_config_dir_with_options", "delete_graph_root", "backend => &graph_uri", 1, RetainAllDeleteScope::WholeGraphDestroy), + ("omnigraph-cluster/lib.rs", "apply_config_dir_with_options", "delete_object", "backend => &sidecar_path", 1, RetainAllDeleteScope::ExactControlObject), + ("omnigraph-cluster/lib.rs", "apply_config_dir_with_options", "delete_object", "backend => sidecar_uri", 1, RetainAllDeleteScope::ExactControlObject), + ("omnigraph-cluster/lib.rs", "apply_config_dir_with_options", "try_delete_object", "backend => &sidecar_path", 1, RetainAllDeleteScope::ExactControlObject), + ("omnigraph-cluster/lib.rs", "sync_config_dir", "delete_object", "backend => sidecar_uri", 1, RetainAllDeleteScope::ExactControlObject), +]; + +fn retain_all_tracked_delete_names(callsites: &[RetainAllDeleteCallsite]) -> BTreeSet { + RETAIN_ALL_RAW_DELETE_PRIMITIVES + .iter() + .map(|primitive| (*primitive).to_string()) + .chain( + callsites + .iter() + .filter(|callsite| callsite.5 == RetainAllDeleteScope::GenericGateway) + .map(|callsite| { + callsite + .1 + .rsplit("::") + .next() + .expect("qualified function owner") + .to_string() + }), + ) + .collect() +} + +fn normalized_symbol(value: &str) -> String { + value + .chars() + .filter(char::is_ascii_alphanumeric) + .flat_map(char::to_lowercase) + .collect() +} + +fn mentions_mem_wal(value: &str) -> bool { + normalized_symbol(value).contains("memwal") || value.contains("_mem_wal") +} + +fn is_reclamation_verb(value: &str) -> bool { + let normalized = normalized_symbol(value); + [ + "adopt", + "cleanup", + "compact", + "delete", + "drop", + "garbagecollect", + "prune", + "purge", + "reclaim", + "reclassify", + "remove", + "trim", + "vacuum", + ] + .iter() + .any(|verb| normalized.contains(verb)) + || normalized == "gc" + || normalized.ends_with("gc") +} + +fn is_mem_wal_reclamation_symbol(value: &str) -> bool { + mentions_mem_wal(value) && is_reclamation_verb(value) +} + +fn raw_mem_wal_marker_count(value: &str) -> usize { + value.matches("_mem_wal").count() +} + +#[derive(Default)] +struct RetainAllMemWalInventory { + function_stack: Vec, + raw_path_owners: BTreeMap, + mem_wal_uses: Vec<(String, String)>, + reclamation_symbols: Vec<(String, String)>, +} + +impl RetainAllMemWalInventory { + fn owner(&self) -> String { + self.function_stack + .last() + .cloned() + .unwrap_or_else(|| "".to_string()) + } + + fn record_raw_path(&mut self, count: usize) { + if count > 0 { + *self.raw_path_owners.entry(self.owner()).or_default() += count; + } + } + + fn record_reclamation_symbol(&mut self, symbol: impl Into) { + self.reclamation_symbols.push((self.owner(), symbol.into())); + } +} + +impl<'ast> Visit<'ast> for RetainAllMemWalInventory { + fn visit_item_mod(&mut self, node: &'ast syn::ItemMod) { + if cfg_requires_test(&node.attrs) { + return; + } + visit::visit_item_mod(self, node); + } + + fn visit_item_impl(&mut self, node: &'ast syn::ItemImpl) { + if cfg_requires_test(&node.attrs) { + return; + } + visit::visit_item_impl(self, node); + } + + fn visit_item_const(&mut self, node: &'ast syn::ItemConst) { + if cfg_requires_test(&node.attrs) { + return; + } + visit::visit_item_const(self, node); + } + + fn visit_item_static(&mut self, node: &'ast syn::ItemStatic) { + if cfg_requires_test(&node.attrs) { + return; + } + visit::visit_item_static(self, node); + } + + fn visit_item_macro(&mut self, node: &'ast syn::ItemMacro) { + if cfg_requires_test(&node.attrs) { + return; + } + visit::visit_item_macro(self, node); + } + + fn visit_item_fn(&mut self, node: &'ast syn::ItemFn) { + if cfg_requires_test(&node.attrs) { + return; + } + let function = node.sig.ident.to_string(); + if is_mem_wal_reclamation_symbol(&function) { + self.record_reclamation_symbol(format!("function `{function}`")); + } + self.function_stack.push(function); + visit::visit_item_fn(self, node); + self.function_stack.pop(); + } + + fn visit_impl_item_fn(&mut self, node: &'ast syn::ImplItemFn) { + if cfg_requires_test(&node.attrs) { + return; + } + let function = node.sig.ident.to_string(); + if is_mem_wal_reclamation_symbol(&function) { + self.record_reclamation_symbol(format!("method `{function}`")); + } + self.function_stack.push(function); + visit::visit_impl_item_fn(self, node); + self.function_stack.pop(); + } + + fn visit_ident(&mut self, node: &'ast syn::Ident) { + let identifier = node.to_string(); + if mentions_mem_wal(&identifier) { + self.mem_wal_uses.push((self.owner(), identifier.clone())); + } + if is_mem_wal_reclamation_symbol(&identifier) { + self.record_reclamation_symbol(format!("identifier `{identifier}`")); + } + visit::visit_ident(self, node); + } + + fn visit_path(&mut self, node: &'ast syn::Path) { + let path = node + .segments + .iter() + .map(|segment| segment.ident.to_string()) + .collect::>() + .join("::"); + if is_mem_wal_reclamation_symbol(&path) { + self.record_reclamation_symbol(format!("path `{path}`")); + } + visit::visit_path(self, node); + } + + fn visit_lit(&mut self, node: &'ast syn::Lit) { + if let syn::Lit::Str(value) = node { + let value = value.value(); + self.record_raw_path(raw_mem_wal_marker_count(&value)); + if mentions_mem_wal(&value) { + self.mem_wal_uses + .push((self.owner(), "literal `_mem_wal`".to_string())); + } + } + visit::visit_lit(self, node); + } + + fn visit_macro(&mut self, node: &'ast syn::Macro) { + let tokens = node.tokens.to_string(); + self.record_raw_path(raw_mem_wal_marker_count(&tokens)); + if mentions_mem_wal(&tokens) { + self.mem_wal_uses + .push((self.owner(), "macro token `_mem_wal`".to_string())); + } + visit::visit_macro(self, node); + } +} + +fn retain_all_mem_wal_inventory(ast: &syn::File) -> RetainAllMemWalInventory { + let mut inventory = RetainAllMemWalInventory::default(); + inventory.visit_file(ast); + inventory +} + fn protocol_scan_files(src: &Path) -> Vec { walk_rust_files(src) .into_iter() @@ -1280,6 +1706,34 @@ fn protocol_scan_files(src: &Path) -> Vec { .collect() } +fn retain_all_production_files(engine_src: &Path) -> Vec<(String, PathBuf)> { + let mut files = protocol_scan_files(engine_src) + .into_iter() + .map(|path| (relative_to_src(engine_src, &path), path)) + .collect::>(); + + // The cluster layer is the only sibling crate that owns a recursive graph + // root deletion. Scan it as part of the same retain-all boundary, while + // excluding its out-of-line `#[cfg(test)] mod tests` source (the attribute + // lives in lib.rs and is not visible when tests.rs is parsed alone). + let cluster_src = engine_src + .parent() + .and_then(Path::parent) + .expect("engine crate lives below the workspace crates directory") + .join("omnigraph-cluster/src"); + for path in walk_rust_files(&cluster_src) { + if path.file_name().and_then(|name| name.to_str()) == Some("tests.rs") { + continue; + } + files.push(( + format!("omnigraph-cluster/{}", relative_to_src(&cluster_src, &path)), + path, + )); + } + files.sort_by(|left, right| left.0.cmp(&right.0)); + files +} + fn is_omnigraph_type(ty: &Type) -> bool { is_named_type(ty, "Omnigraph") } @@ -1757,6 +2211,321 @@ fn graph_visible_write_chokepoints_are_registered() { ); } +/// RFC-026 B2a deliberately retains every raw MemWAL object. Generic graph +/// maintenance may block on stream lifecycle authority, and exact stream +/// recovery may classify its own bound effects, but neither is permission to +/// reinterpret, adopt, or delete `_mem_wal` storage. +#[test] +fn retain_all_mem_wal_has_no_production_reclamation_or_adoption_route() { + let src = engine_src_root(); + let expected_raw_owners = RAW_MEM_WAL_PATH_OWNERS + .iter() + .map(|(file, function, count)| (((*file).to_string(), (*function).to_string()), *count)) + .collect::>(); + let tracked_delete_names = retain_all_tracked_delete_names(RETAIN_ALL_DELETE_CALLS); + let expected_delete_calls = RETAIN_ALL_DELETE_CALLS + .iter() + .map(|callsite| { + ( + ( + callsite.0.to_string(), + callsite.1.to_string(), + callsite.2.to_string(), + callsite.3.to_string(), + ), + callsite.4, + ) + }) + .collect::>(); + assert_eq!( + expected_delete_calls.len(), + RETAIN_ALL_DELETE_CALLS.len(), + "retain-all delete callsite keys must be unique" + ); + let mut observed_raw_owners = BTreeMap::new(); + let mut observed_delete_calls = BTreeMap::new(); + let mut reclamation_symbols = Vec::new(); + let mut maintenance_mem_wal_uses = Vec::new(); + let mut hidden_delete_routes = Vec::new(); + let mut mem_wal_destructive_calls = Vec::new(); + + for (relative, file) in retain_all_production_files(&src) { + let contents = std::fs::read_to_string(&file) + .unwrap_or_else(|error| panic!("failed to read {}: {error}", file.display())); + let ast = parse_rust_source(&contents, &relative); + let inventory = retain_all_mem_wal_inventory(&ast); + let calls = call_inventory(&ast); + + for (function, count) in inventory.raw_path_owners { + observed_raw_owners.insert((relative.clone(), function), count); + } + reclamation_symbols.extend( + inventory + .reclamation_symbols + .into_iter() + .map(|(function, symbol)| format!("{relative}::{function}: {symbol}")), + ); + hidden_delete_routes.extend( + calls + .retain_all_hidden_route_hits + .iter() + .map(|hit| format!("{relative}: {hit}")), + ); + for ((function, primitive, arguments), count) in &calls.call_shapes_by_function { + if tracked_delete_names.contains(primitive) { + *observed_delete_calls + .entry(( + relative.clone(), + function.clone(), + primitive.clone(), + arguments.clone(), + )) + .or_default() += *count; + } + } + + if matches!( + relative.as_str(), + "table_store/mem_wal.rs" | "table_store/mem_wal/worker.rs" + ) { + for primitive in [ + "cleanup", + "cleanup_old_versions", + "compact", + "delete", + "delete_prefix", + "garbage_collect", + "prune", + "purge", + "reclaim", + "remove_dir", + "remove_dir_all", + "remove_file", + "trim", + "vacuum", + ] { + if let Some(count) = calls.counts.get(primitive) { + mem_wal_destructive_calls + .push(format!("{relative}: `{primitive}` called {count} time(s)")); + } + } + } + + if RETAIN_ALL_MAINTENANCE_FILES.contains(&relative.as_str()) { + maintenance_mem_wal_uses.extend( + inventory + .mem_wal_uses + .iter() + .map(|(function, usage)| format!("{relative}::{function}: {usage}")), + ); + } + if relative == "db/omnigraph.rs" { + maintenance_mem_wal_uses.extend( + inventory + .mem_wal_uses + .iter() + .filter(|(function, _)| { + RETAIN_ALL_OMNIGRAPH_METHODS.contains(&function.as_str()) + }) + .map(|(function, usage)| format!("{relative}::{function}: {usage}")), + ); + } + } + + assert_eq!( + observed_raw_owners, expected_raw_owners, + "raw `_mem_wal` path access must remain confined to the exact bounded read-only inventory and passive lifecycle validators" + ); + assert!( + reclamation_symbols.is_empty(), + "retain-all forbids any production MemWAL reclamation/adoption/reclassification route:\n {}", + reclamation_symbols.join("\n ") + ); + assert!( + maintenance_mem_wal_uses.is_empty(), + "cleanup/repair/schema maintenance must derive exclusions from manifest lifecycle authority and never inspect or reinterpret MemWAL objects:\n {}", + maintenance_mem_wal_uses.join("\n ") + ); + assert!( + hidden_delete_routes.is_empty(), + "retain-all delete routes may not hide behind aliases or macro tokens:\n {}", + hidden_delete_routes.join("\n ") + ); + assert_eq!( + observed_delete_calls, expected_delete_calls, + "retain-all raw-delete route inventory drifted. Runtime-built paths and ancestor-prefix deletes are unsafe even when no symbol spells `_mem_wal`; every sink and caller-provided-path gateway requires an owner/receiver/argument disposition" + ); + assert!( + mem_wal_destructive_calls.is_empty(), + "the private MemWAL adapter may validate, acknowledge, and fold, but retain-all forbids destructive/reclamation primitives:\n {}", + mem_wal_destructive_calls.join("\n ") + ); + + let mem_wal_source = std::fs::read_to_string(src.join("table_store/mem_wal.rs")) + .expect("read MemWAL adapter source"); + let mem_wal_ast = parse_rust_source(&mem_wal_source, "table_store/mem_wal.rs"); + for helper in ["raw_mem_wal_inventory", "classify_raw_inventory"] { + let visibility = mem_wal_ast.items.iter().find_map(|item| match item { + Item::Fn(function) if function.sig.ident == helper => Some(&function.vis), + _ => None, + }); + assert!( + matches!(visibility, Some(Visibility::Inherited)), + "raw MemWAL helper `{helper}` must remain module-private so maintenance/recovery cannot turn inventory into an adoption or deletion API" + ); + } +} + +#[test] +fn retain_all_mem_wal_scanner_rejects_production_delete_but_skips_test_fixture() { + let fixture_only = parse_rust_source( + r#" + #[cfg(test)] + mod tests { + fn destructive_fixture(storage: &Storage) { + let mem_wal_root = "_mem_wal"; + storage.delete_prefix(mem_wal_root); + } + } + + #[cfg(test)] + const TEST_MEM_WAL_ROOT: &str = "_mem_wal"; + + #[cfg(test)] + static TEST_MEM_WAL_PATH: &str = "_mem_wal"; + + #[cfg(test)] + macro_rules! reclaim_mem_wal_fixture { + () => { + let mem_wal_root = "_mem_wal"; + mem_wal::reclaim(mem_wal_root); + }; + } + "#, + "retain-all cfg(test) self-test", + ); + let fixture_inventory = retain_all_mem_wal_inventory(&fixture_only); + assert!(fixture_inventory.raw_path_owners.is_empty()); + assert!(fixture_inventory.reclamation_symbols.is_empty()); + + let production = parse_rust_source( + r#" + fn destructive_production(storage: &Storage) { + let mem_wal_root = "_mem_wal"; + storage.delete_prefix(mem_wal_root); + mem_wal::reclaim(); + } + "#, + "retain-all production self-test", + ); + let production_inventory = retain_all_mem_wal_inventory(&production); + assert_eq!( + production_inventory.raw_path_owners, + BTreeMap::from([("destructive_production".to_string(), 1)]) + ); + assert!( + production_inventory + .reclamation_symbols + .iter() + .any(|(_, symbol)| symbol.contains("mem_wal::reclaim")) + ); +} + +#[test] +fn retain_all_delete_scanner_catches_opaque_delegated_and_hidden_routes() { + let ast = parse_rust_source( + r#" + use tokio::fs::remove_dir_all as erase_tree; + + macro_rules! hidden_delete { + ($storage:expr, $path:expr) => { $storage.delete_prefix($path) }; + } + + fn direct_maintenance(storage: &Storage, table_root: &str) { + storage.delete_prefix(table_root); + } + + fn remove_tree(storage: &Storage, path: &str) { + storage.delete_prefix(path); + } + + fn delegated_maintenance(storage: &Storage, runtime_path: &str) { + remove_tree(storage, runtime_path); + } + + #[cfg(test)] + mod fixtures { + fn teardown(storage: &Storage, root: &str) { + storage.delete_prefix(root); + } + } + "#, + "retain-all opaque destructive-route self-test", + ); + let inventory = call_inventory(&ast); + const SYNTHETIC_GATEWAYS: &[RetainAllDeleteCallsite] = &[( + "synthetic.rs", + "remove_tree", + "delete_prefix", + "storage => path", + 1, + RetainAllDeleteScope::GenericGateway, + )]; + let tracked = retain_all_tracked_delete_names(SYNTHETIC_GATEWAYS); + let observed = inventory + .call_shapes_by_function + .iter() + .filter(|((_, primitive, _), _)| tracked.contains(primitive)) + .map(|(key, count)| (key.clone(), *count)) + .collect::>(); + + // Neither route needs to spell `_mem_wal`: a table/root prefix is already + // ancestor-dangerous. Classifying `remove_tree` as a generic gateway also + // causes the production filter to inventory its caller edge. + assert_eq!( + observed.get(&( + "direct_maintenance".to_string(), + "delete_prefix".to_string(), + "storage => table_root".to_string(), + )), + Some(&1) + ); + assert_eq!( + observed.get(&( + "remove_tree".to_string(), + "delete_prefix".to_string(), + "storage => path".to_string(), + )), + Some(&1) + ); + assert_eq!( + observed.get(&( + "delegated_maintenance".to_string(), + "remove_tree".to_string(), + "remove_tree => storage, runtime_path".to_string(), + )), + Some(&1) + ); + assert!( + inventory + .call_shapes_by_function + .keys() + .all(|(function, _, _)| function != "teardown") + ); + assert!( + inventory + .retain_all_hidden_route_hits + .iter() + .any(|hit| hit.contains("remove_dir_all") && hit.contains("erase_tree")) + ); + assert!( + inventory + .retain_all_hidden_route_hits + .iter() + .any(|hit| hit.contains("delete_prefix")) + ); +} + #[test] fn structural_call_scanner_counts_method_and_ufcs_not_text() { let ast = parse_rust_source( diff --git a/crates/omnigraph/tests/helpers/memwal.rs b/crates/omnigraph/tests/helpers/memwal.rs new file mode 100644 index 00000000..fde59667 --- /dev/null +++ b/crates/omnigraph/tests/helpers/memwal.rs @@ -0,0 +1,354 @@ +//! Canonical fail-closed inventory for the currently listed Lance MemWAL +//! namespace used by RFC-026 evidence tests. +//! +//! This is intentionally one shared persisted-layout truth. It classifies only +//! the pinned Lance RC.1 path grammar and validates shard-manifest authority; +//! unknown or malformed paths remain visible as `Unknown` so callers fail +//! closed. Ordinary LIST still cannot expose incomplete multipart uploads, +//! superseded provider versions, delete markers, local staging files, or +//! provider-billed bytes. + +use std::collections::{BTreeMap, BTreeSet}; +use std::sync::Arc; + +use futures::TryStreamExt; +use lance::dataset::mem_wal::ShardManifestStore; +use lance_index::mem_wal::{ShardId, ShardManifest}; +use object_store::path::Path as ObjectPath; + +#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)] +pub enum MemWalObjectKind { + Wal, + ShardManifestVersion, + ShardManifestHint, + GenerationData, + GenerationManifest, + GenerationTransaction, + GenerationDeletion, + GenerationPkSidecar, + GenerationBloom, + GenerationUserIndex, + Unknown, +} + +#[derive(Debug, Clone, Default)] +pub struct CurrentMemWalInventory { + pub objects: BTreeMap, + pub generation_roots: BTreeSet, + pub referenced_generation_roots: BTreeSet, +} + +impl CurrentMemWalInventory { + pub fn object_count(&self, kind: MemWalObjectKind) -> usize { + self.objects + .values() + .filter(|(candidate, _)| *candidate == kind) + .count() + } + + pub fn bytes(&self, kind: MemWalObjectKind) -> u64 { + self.objects + .values() + .filter(|(candidate, _)| *candidate == kind) + .map(|(_, bytes)| *bytes) + .sum() + } + + pub fn immutable_object_bytes(&self) -> u64 { + self.objects + .values() + .filter(|(kind, _)| *kind != MemWalObjectKind::ShardManifestHint) + .map(|(_, bytes)| *bytes) + .sum() + } + + pub fn unknown_paths(&self) -> Vec<&str> { + self.objects + .iter() + .filter_map(|(path, (kind, _))| { + (*kind == MemWalObjectKind::Unknown).then_some(path.as_str()) + }) + .collect() + } + + pub fn generation_subtree_objects(&self) -> BTreeMap { + self.objects + .iter() + .filter(|(_, (kind, _))| { + matches!( + kind, + MemWalObjectKind::GenerationData + | MemWalObjectKind::GenerationManifest + | MemWalObjectKind::GenerationTransaction + | MemWalObjectKind::GenerationDeletion + | MemWalObjectKind::GenerationPkSidecar + | MemWalObjectKind::GenerationBloom + | MemWalObjectKind::GenerationUserIndex + ) + }) + .map(|(path, value)| (path.clone(), *value)) + .collect() + } + + pub fn assert_path_class_size_retained_by(&self, later: &Self) { + for (path, (kind, bytes)) in &self.objects { + if *kind == MemWalObjectKind::ShardManifestHint { + continue; + } + assert_eq!( + later.objects.get(path), + Some(&(*kind, *bytes)), + "retain-all evidence lost or changed the class/size of listed object {path}" + ); + } + } +} + +pub fn is_generation_root(component: &str) -> bool { + let Some((hash, generation)) = component.split_once("_gen_") else { + return false; + }; + let Ok(generation_number) = generation.parse::() else { + return false; + }; + hash.len() == 8 + && hash + .bytes() + .all(|byte| byte.is_ascii_digit() || (b'a'..=b'f').contains(&byte)) + && generation_number > 0 + && generation_number.to_string() == generation +} + +pub fn parse_positive_bit_reversed_filename(filename: &str, extension: &str) -> Option { + let Some(stem) = filename.strip_suffix(&format!(".{extension}")) else { + return None; + }; + if stem.len() != 64 || !stem.bytes().all(|byte| matches!(byte, b'0' | b'1')) { + return None; + } + let reversed = u64::from_str_radix(stem, 2).ok()?; + let value = reversed.reverse_bits(); + (value > 0).then_some(value) +} + +fn parse_canonical_shard_id(value: &str) -> Option { + let shard_id = ShardId::parse_str(value).ok()?; + (shard_id.as_hyphenated().to_string() == value).then_some(shard_id) +} + +fn join_object_path(base: &str, suffix: &str) -> String { + if base.is_empty() { + suffix.to_string() + } else { + format!("{base}/{suffix}") + } +} + +pub fn classify_mem_wal_object( + components: &[&str], + mem_wal_index: usize, +) -> (MemWalObjectKind, Option) { + let tail = &components[mem_wal_index + 1..]; + if tail.len() < 3 || parse_canonical_shard_id(tail[0]).is_none() { + return (MemWalObjectKind::Unknown, None); + } + + match tail[1] { + "wal" + if tail.len() == 3 + && parse_positive_bit_reversed_filename(tail[2], "arrow").is_some() => + { + (MemWalObjectKind::Wal, None) + } + "manifest" if tail.len() == 3 && tail[2] == "version_hint.json" => { + (MemWalObjectKind::ShardManifestHint, None) + } + "manifest" + if tail.len() == 3 + && parse_positive_bit_reversed_filename(tail[2], "binpb").is_some() => + { + (MemWalObjectKind::ShardManifestVersion, None) + } + generation if is_generation_root(generation) => { + let root = components[..mem_wal_index + 3].join("/"); + let kind = match tail.get(2).copied() { + Some("bloom_filter.bin") if tail.len() == 3 => MemWalObjectKind::GenerationBloom, + Some("data") if tail.len() >= 4 => MemWalObjectKind::GenerationData, + Some("_versions") if tail.len() >= 4 => MemWalObjectKind::GenerationManifest, + Some("_transactions") if tail.len() >= 4 => MemWalObjectKind::GenerationTransaction, + Some("_deletions") if tail.len() >= 4 => MemWalObjectKind::GenerationDeletion, + Some("_pk_index") if tail.len() >= 4 => MemWalObjectKind::GenerationPkSidecar, + Some("_indices") if tail.len() >= 4 => MemWalObjectKind::GenerationUserIndex, + _ => MemWalObjectKind::Unknown, + }; + (kind, Some(root)) + } + _ => (MemWalObjectKind::Unknown, None), + } +} + +pub fn validated_referenced_generation_roots( + table_base: &str, + directory_shard_id: ShardId, + manifest: &ShardManifest, + listed_generation_roots: &BTreeSet, + listed_manifest_versions: &BTreeSet, +) -> BTreeSet { + assert_eq!( + manifest.shard_id, directory_shard_id, + "Gate R0 must fail closed when decoded shard authority disagrees with its directory" + ); + assert!( + manifest.version > 0, + "Gate R0 must fail closed on a zero shard-manifest version" + ); + let latest_listed_version = listed_manifest_versions + .last() + .copied() + .expect("Gate R0 must fail closed when a shard has no listed manifest version"); + assert_eq!( + latest_listed_version, manifest.version, + "Gate R0 must fail closed when the latest shard-manifest filename disagrees with its body" + ); + let expected_manifest_version_count = usize::try_from(manifest.version) + .expect("Gate R0 must fail closed when shard-manifest authority exceeds usize"); + assert_eq!( + listed_manifest_versions.len(), + expected_manifest_version_count, + "Gate R0 must fail closed on a gapped or noncanonical shard-manifest version chain" + ); + assert!( + manifest.current_generation > 0, + "Gate R0 must fail closed on zero next-generation allocation authority" + ); + + let shard_prefix = join_object_path( + table_base, + &format!("_mem_wal/{}", directory_shard_id.as_hyphenated()), + ); + let mut generations = BTreeSet::new(); + let mut paths = BTreeSet::new(); + let mut roots = BTreeSet::new(); + for flushed in &manifest.flushed_generations { + assert!( + flushed.generation > 0 && generations.insert(flushed.generation), + "Gate R0 must fail closed on a zero or duplicate flushed generation" + ); + assert!( + paths.insert(flushed.path.clone()), + "Gate R0 must fail closed on a duplicate flushed-generation path" + ); + assert!( + is_generation_root(&flushed.path), + "Gate R0 must fail closed on a noncanonical flushed-generation path" + ); + let (_, path_generation) = flushed + .path + .split_once("_gen_") + .expect("validated generation root must contain _gen_"); + assert_eq!( + path_generation.parse::().unwrap(), + flushed.generation, + "Gate R0 must fail closed when generation authority disagrees with its path" + ); + assert!( + manifest.current_generation > flushed.generation, + "Gate R0 must fail closed when next-generation allocation authority has not advanced past a flushed generation" + ); + let root = join_object_path(&shard_prefix, &flushed.path); + assert!( + listed_generation_roots.contains(&root), + "Gate R0 must fail closed when shard authority references a missing listed generation" + ); + assert!( + roots.insert(root), + "Gate R0 must fail closed on duplicate generation authority" + ); + } + roots +} + +/// Strict current-object inventory for the part of retained storage that stock +/// Lance exposes through ordinary LIST. +pub async fn current_mem_wal_inventory(uri: &str) -> CurrentMemWalInventory { + let (store, root_path) = lance_io::object_store::ObjectStore::from_uri(uri) + .await + .expect("MemWAL fixture URI must resolve"); + let listed = store + .inner + .list(Some(&root_path)) + .try_collect::>() + .await + .expect("MemWAL inventory must fail closed on a listing error"); + + let mut inventory = CurrentMemWalInventory::default(); + let mut shards = BTreeSet::new(); + let mut manifest_versions = BTreeMap::<(String, ShardId), BTreeSet>::new(); + for metadata in listed { + let path = metadata.location.as_ref().to_string(); + let components = path.split('/').collect::>(); + let Some(mem_wal_index) = components + .iter() + .position(|component| *component == "_mem_wal") + else { + continue; + }; + let (kind, generation_root) = classify_mem_wal_object(&components, mem_wal_index); + if let Some(root) = generation_root { + inventory.generation_roots.insert(root); + } + let bytes = u64::try_from(metadata.size).expect("object size must fit u64"); + assert!( + inventory + .objects + .insert(path.clone(), (kind, bytes)) + .is_none(), + "object listing returned duplicate path {path}" + ); + + let tail = &components[mem_wal_index + 1..]; + if let Some(shard) = tail + .first() + .and_then(|value| parse_canonical_shard_id(value)) + { + let table_base = components[..mem_wal_index].join("/"); + shards.insert((table_base.clone(), shard)); + if kind == MemWalObjectKind::ShardManifestVersion { + let version = parse_positive_bit_reversed_filename(tail[2], "binpb") + .expect("classified shard-manifest version must decode"); + assert!( + manifest_versions + .entry((table_base, shard)) + .or_default() + .insert(version), + "object listing returned duplicate decoded shard-manifest version" + ); + } + } + } + + for (table_base, shard_id) in shards { + let table_path = + ObjectPath::parse(&table_base).expect("listed table base must remain a valid path"); + let manifest_store = ShardManifestStore::new(Arc::clone(&store), &table_path, shard_id, 2); + let manifest = manifest_store + .read_latest() + .await + .expect("MemWAL inventory must fail closed on an unreadable shard manifest") + .expect("Gate R0 must fail closed when a listed shard has no latest manifest"); + let listed_manifest_versions = manifest_versions + .get(&(table_base.clone(), shard_id)) + .expect("Gate R0 must fail closed when a shard has no listed manifest version"); + inventory + .referenced_generation_roots + .extend(validated_referenced_generation_roots( + &table_base, + shard_id, + &manifest, + &inventory.generation_roots, + listed_manifest_versions, + )); + } + + inventory +} diff --git a/crates/omnigraph/tests/helpers/mod.rs b/crates/omnigraph/tests/helpers/mod.rs index e5c179bf..e1aa6df5 100644 --- a/crates/omnigraph/tests/helpers/mod.rs +++ b/crates/omnigraph/tests/helpers/mod.rs @@ -3,6 +3,8 @@ pub mod cost; #[cfg(feature = "failpoints")] pub mod failpoint; +#[cfg(feature = "failpoints")] +pub mod memwal; pub mod recovery; use arrow_array::{Array, RecordBatch, StringArray}; diff --git a/crates/omnigraph/tests/memwal_stream.rs b/crates/omnigraph/tests/memwal_stream.rs index 5bb7a4dc..35b5b07b 100644 --- a/crates/omnigraph/tests/memwal_stream.rs +++ b/crates/omnigraph/tests/memwal_stream.rs @@ -7,20 +7,36 @@ mod helpers; +use std::collections::BTreeSet; +use std::fmt; use std::sync::Arc; +use std::sync::Mutex; use std::sync::atomic::{AtomicUsize, Ordering}; use std::time::Duration; use arrow_array::{Array, ArrayRef, Int32Array, RecordBatch, StringArray}; use arrow_schema::{DataType, Field, Schema}; +use async_trait::async_trait; use fail::FailScenario; +use futures::StreamExt; +use futures::stream::BoxStream; use lance::Dataset; use lance::dataset::mem_wal::{DatasetMemWalExt, ShardWriterConfig}; +use lance::io::WrappingObjectStore; +use object_store::path::Path as ObjectPath; +use object_store::{ + CopyOptions, GetOptions, GetResult, ListResult, MultipartUpload, ObjectMeta, ObjectStore, + PutMultipartOptions, PutOptions, PutPayload, PutResult, Result as ObjectStoreResult, + UploadPart, +}; use omnigraph::db::{Omnigraph, ReadTarget}; use omnigraph::error::OmniError; use omnigraph::failpoints::{ScopedFailPoint, names}; +use omnigraph::instrumentation::{QueryIoProbes, with_query_io_probes}; use serial_test::serial; +use helpers::memwal::CurrentMemWalInventory; + const STREAM_SCHEMA: &str = "node Person { score: I32 }\n"; const TWO_TABLE_STREAM_SCHEMA: &str = r#" node Person { score: I32 } @@ -45,6 +61,411 @@ query insert_company($score: I32) { } "#; +const PROVIDER_FAILURE_MESSAGE: &str = "injected RFC-026 provider exhaustion"; +const PROVIDER_PUT_TIMEOUT: Duration = Duration::from_secs(45); +const PROVIDER_FOLD_TIMEOUT: Duration = Duration::from_secs(90); + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum ProviderFailureTarget { + PartialGeneration, + ShardManifest, + WalEntry, +} + +impl ProviderFailureTarget { + fn matches(self, location: &ObjectPath) -> bool { + let path = location.to_string(); + let is_mem_wal = path.split('/').any(|part| part == "_mem_wal"); + if !is_mem_wal { + return false; + } + match self { + Self::PartialGeneration => { + generation_root_in_path(&path).is_some() && path.ends_with("/bloom_filter.bin") + } + Self::ShardManifest => { + path.contains("/manifest/") && !path.ends_with("version_hint.json") + } + Self::WalEntry => path.contains("/wal/") && path.contains(".arrow"), + } + } +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum ProviderAccessKind { + Put, + MultipartStart, + MultipartPart, + MultipartComplete, + MultipartAbort, + Get, + Delete, + List, + ListResult, + ListWithOffset, + ListWithDelimiter, + CopyFrom, + CopyTo, +} + +#[derive(Debug, Clone, PartialEq, Eq)] +struct ProviderAccess { + kind: ProviderAccessKind, + path: String, +} + +#[derive(Debug, Default)] +struct ProviderFailureState { + active: Option, + failed: Vec<(ProviderFailureTarget, String)>, + accesses: Vec, + successful_generation_writes: usize, +} + +/// Fails real object-store writes made by Lance's detached MemWAL workers. +/// +/// `QueryIoProbes` installs this on the table handle before the worker is +/// opened, and OmniGraph propagates that task-local wrapper into detached put +/// and fold owners. This therefore exercises the provider boundary rather than +/// translating an OmniGraph failpoint into a provider-shaped error. +#[derive(Debug, Clone, Default)] +struct ProviderWriteFailure(Arc>); + +impl ProviderWriteFailure { + fn arm(&self, target: ProviderFailureTarget) { + let mut state = self.0.lock().unwrap(); + state.active = Some(target); + state.successful_generation_writes = 0; + } + + fn disarm(&self) { + self.0.lock().unwrap().active = None; + } + + fn failed_paths(&self, target: ProviderFailureTarget) -> Vec { + self.0 + .lock() + .unwrap() + .failed + .iter() + .filter(|(actual, _)| *actual == target) + .map(|(_, path)| path.clone()) + .collect() + } + + fn access_checkpoint(&self) -> usize { + self.0.lock().unwrap().accesses.len() + } + + fn accesses_since(&self, checkpoint: usize) -> Vec { + self.0.lock().unwrap().accesses[checkpoint..].to_vec() + } + + fn successful_generation_writes(&self) -> usize { + self.0.lock().unwrap().successful_generation_writes + } + + fn record(&self, kind: ProviderAccessKind, location: &ObjectPath) { + self.0.lock().unwrap().accesses.push(ProviderAccess { + kind, + path: location.to_string(), + }); + } + + fn record_successful_write(&self, location: &ObjectPath) { + if generation_root_in_path(&location.to_string()).is_some() { + self.0.lock().unwrap().successful_generation_writes += 1; + } + } + + fn record_write_and_maybe_fail( + &self, + kind: ProviderAccessKind, + location: &ObjectPath, + ) -> Option { + let mut state = self.0.lock().unwrap(); + state.accesses.push(ProviderAccess { + kind, + path: location.to_string(), + }); + let target = state.active?; + if !target.matches(location) { + return None; + } + state.failed.push((target, location.to_string())); + Some(object_store::Error::Generic { + store: "rfc026-provider-failure", + source: Box::new(std::io::Error::other(PROVIDER_FAILURE_MESSAGE)), + }) + } + + fn assert_no_access_under_roots_since( + &self, + checkpoint: usize, + roots: &BTreeSet, + boundary: &str, + ) { + let offending = self + .accesses_since(checkpoint) + .into_iter() + .filter(|access| { + roots + .iter() + .any(|root| path_is_at_or_under(&access.path, root)) + }) + .collect::>(); + assert!( + offending.is_empty(), + "{boundary} must not descend into, read, write, copy, or delete at/below retained unreferenced roots {roots:?}; observed {offending:?}" + ); + } +} + +impl WrappingObjectStore for ProviderWriteFailure { + fn wrap(&self, _store_prefix: &str, target: Arc) -> Arc { + Arc::new(ProviderFailingStore { + target, + failure: self.clone(), + }) + } +} + +#[derive(Debug)] +struct ProviderFailingStore { + target: Arc, + failure: ProviderWriteFailure, +} + +impl fmt::Display for ProviderFailingStore { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!(f, "ProviderFailingStore({})", self.target) + } +} + +#[derive(Debug)] +struct ProviderTrackedMultipart { + target: Box, + failure: ProviderWriteFailure, + location: ObjectPath, +} + +#[async_trait] +impl MultipartUpload for ProviderTrackedMultipart { + fn put_part(&mut self, data: PutPayload) -> UploadPart { + self.failure + .record(ProviderAccessKind::MultipartPart, &self.location); + self.target.put_part(data) + } + + async fn complete(&mut self) -> ObjectStoreResult { + self.failure + .record(ProviderAccessKind::MultipartComplete, &self.location); + let result = self.target.complete().await; + if result.is_ok() { + self.failure.record_successful_write(&self.location); + } + result + } + + async fn abort(&mut self) -> ObjectStoreResult<()> { + self.failure + .record(ProviderAccessKind::MultipartAbort, &self.location); + self.target.abort().await + } +} + +#[async_trait] +impl ObjectStore for ProviderFailingStore { + async fn put_opts( + &self, + location: &ObjectPath, + payload: PutPayload, + opts: PutOptions, + ) -> ObjectStoreResult { + if let Some(error) = self + .failure + .record_write_and_maybe_fail(ProviderAccessKind::Put, location) + { + return Err(error); + } + let result = self.target.put_opts(location, payload, opts).await; + if result.is_ok() { + self.failure.record_successful_write(location); + } + result + } + + async fn put_multipart_opts( + &self, + location: &ObjectPath, + opts: PutMultipartOptions, + ) -> ObjectStoreResult> { + if let Some(error) = self + .failure + .record_write_and_maybe_fail(ProviderAccessKind::MultipartStart, location) + { + return Err(error); + } + let target = self.target.put_multipart_opts(location, opts).await?; + Ok(Box::new(ProviderTrackedMultipart { + target, + failure: self.failure.clone(), + location: location.clone(), + })) + } + + async fn get_opts( + &self, + location: &ObjectPath, + options: GetOptions, + ) -> ObjectStoreResult { + self.failure.record(ProviderAccessKind::Get, location); + self.target.get_opts(location, options).await + } + + fn delete_stream( + &self, + locations: BoxStream<'static, ObjectStoreResult>, + ) -> BoxStream<'static, ObjectStoreResult> { + let failure = self.failure.clone(); + self.target.delete_stream( + locations + .map(move |location| { + if let Ok(location) = &location { + failure.record(ProviderAccessKind::Delete, location); + } + location + }) + .boxed(), + ) + } + + fn list( + &self, + prefix: Option<&ObjectPath>, + ) -> BoxStream<'static, ObjectStoreResult> { + self.failure.record( + ProviderAccessKind::List, + &prefix.cloned().unwrap_or_default(), + ); + let failure = self.failure.clone(); + self.target + .list(prefix) + .map(move |result| { + if let Ok(metadata) = &result { + failure.record(ProviderAccessKind::ListResult, &metadata.location); + } + result + }) + .boxed() + } + + fn list_with_offset( + &self, + prefix: Option<&ObjectPath>, + offset: &ObjectPath, + ) -> BoxStream<'static, ObjectStoreResult> { + self.failure.record( + ProviderAccessKind::ListWithOffset, + &prefix.cloned().unwrap_or_default(), + ); + self.failure + .record(ProviderAccessKind::ListWithOffset, offset); + let failure = self.failure.clone(); + self.target + .list_with_offset(prefix, offset) + .map(move |result| { + if let Ok(metadata) = &result { + failure.record(ProviderAccessKind::ListResult, &metadata.location); + } + result + }) + .boxed() + } + + async fn list_with_delimiter( + &self, + prefix: Option<&ObjectPath>, + ) -> ObjectStoreResult { + self.failure.record( + ProviderAccessKind::ListWithDelimiter, + &prefix.cloned().unwrap_or_default(), + ); + let result = self.target.list_with_delimiter(prefix).await?; + for metadata in &result.objects { + self.failure + .record(ProviderAccessKind::ListResult, &metadata.location); + } + Ok(result) + } + + async fn copy_opts( + &self, + from: &ObjectPath, + to: &ObjectPath, + options: CopyOptions, + ) -> ObjectStoreResult<()> { + self.failure.record(ProviderAccessKind::CopyFrom, from); + if let Some(error) = self + .failure + .record_write_and_maybe_fail(ProviderAccessKind::CopyTo, to) + { + return Err(error); + } + let result = self.target.copy_opts(from, to, options).await; + if result.is_ok() { + self.failure.record_successful_write(to); + } + result + } +} + +async fn mem_wal_inventory(uri: &str) -> CurrentMemWalInventory { + let inventory = helpers::memwal::current_mem_wal_inventory(uri).await; + assert!( + inventory.unknown_paths().is_empty(), + "provider-failure inventory must fail closed on unknown paths: {:?}", + inventory.unknown_paths() + ); + inventory +} + +fn generation_root_in_path(path: &str) -> Option { + let parts = path.split('/').collect::>(); + let index = parts + .iter() + .position(|part| helpers::memwal::is_generation_root(part))?; + Some(parts[..=index].join("/")) +} + +fn path_is_at_or_under(path: &str, root: &str) -> bool { + path == root + || path + .strip_prefix(root) + .is_some_and(|suffix| suffix.starts_with('/')) +} + +fn assert_inventory_retained( + before: &CurrentMemWalInventory, + after: &CurrentMemWalInventory, + boundary: &str, +) { + before.assert_path_class_size_retained_by(after); + assert!( + before.generation_roots.is_subset(&after.generation_roots), + "{boundary} must retain every previously listed generation root" + ); +} + +async fn enroll_stream_at(uri: &str) { + let db = Omnigraph::init(uri, STREAM_SCHEMA) + .await + .expect("provider-failure fixture must initialize"); + db.failpoint_enroll_stream_table_for_test(TABLE) + .await + .expect("provider-failure fixture must enroll its one stream table"); +} + async fn init_enrolled() -> (tempfile::TempDir, Arc) { init_enrolled_with_schema(STREAM_SCHEMA).await } @@ -1524,3 +1945,391 @@ async fn ack_unknown_retry_can_overwrite_a_newer_same_key_without_reconciling_th "retrying ambiguous X after durable Y demonstrates the documented overwrite hazard; it does not prove X was reconciled" ); } + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +#[serial] +async fn provider_write_failures_preserve_acknowledged_rows_and_type_outcomes_local() { + let _scenario = FailScenario::setup(); + let dir = tempfile::tempdir().unwrap(); + let uri = dir.path().to_str().unwrap().to_string(); + enroll_stream_at(&uri).await; + + let failure = ProviderWriteFailure::default(); + let probes = QueryIoProbes { + table_wrapper: Some(Arc::new(failure.clone())), + ..Default::default() + }; + with_query_io_probes(probes, async { + let db = Arc::new( + Omnigraph::open(&uri) + .await + .expect("the enrolled fixture must reopen under the provider wrapper"), + ); + let version_before = db + .snapshot_of(ReadTarget::branch("main")) + .await + .unwrap() + .version(); + + // A cold writer must first durably claim its epoch in the shard + // manifest. Failing that provider write happens before put_no_wait, so + // this refusal must remain effect-free and must not become AckUnknown. + failure.arm(ProviderFailureTarget::ShardManifest); + let rejected = physical_batch(&db, &[("claim-refused".to_string(), 1)]).await; + let error = tokio::time::timeout( + PROVIDER_PUT_TIMEOUT, + db.failpoint_stream_b1_for_test(TABLE, Some(rejected), 0), + ) + .await + .expect("the failed cold claim must settle") + .expect_err("provider exhaustion must refuse the cold claim"); + failure.disarm(); + assert!(matches!(error, OmniError::Lance(_)), "{error:?}"); + assert!( + !matches!(error, OmniError::AckUnknown { .. }), + "a failed pre-invocation claim cannot be acknowledgement-ambiguous" + ); + assert!( + error.to_string().contains(PROVIDER_FAILURE_MESSAGE), + "the typed substrate error must retain its provider cause: {error:?}" + ); + assert!( + !failure + .failed_paths(ProviderFailureTarget::ShardManifest) + .is_empty(), + "the wrapper must have reached the detached cold-claim manifest write" + ); + assert!(visible_rows(&db).await.is_empty()); + assert_eq!( + db.snapshot_of(ReadTarget::branch("main")) + .await + .unwrap() + .version(), + version_before, + "provider refusal before put invocation cannot advance graph visibility" + ); + + // Establish one clean durability acknowledgement, then fail the next + // real WAL write. Lance has already accepted the second invocation into + // its asynchronous writer machinery, so OmniGraph must return + // AckUnknown while preserving the first durable row for replay. + let acknowledged = physical_batch(&db, &[("acknowledged".to_string(), 2)]).await; + db.failpoint_stream_b1_for_test(TABLE, Some(acknowledged), 1) + .await + .expect("the first row must cross the complete durability boundary"); + + failure.arm(ProviderFailureTarget::WalEntry); + let ambiguous = physical_batch(&db, &[("wal-refused".to_string(), 3)]).await; + let error = tokio::time::timeout( + PROVIDER_PUT_TIMEOUT, + db.failpoint_stream_b1_for_test(TABLE, Some(ambiguous), 2), + ) + .await + .expect("the failed WAL persistence retries must settle") + .expect_err("a failed post-invocation WAL write cannot acknowledge cleanly"); + failure.disarm(); + assert!(matches!(error, OmniError::AckUnknown { .. }), "{error:?}"); + assert!( + error.to_string().contains(PROVIDER_FAILURE_MESSAGE), + "AckUnknown must retain the provider failure cause: {error:?}" + ); + assert!( + !failure + .failed_paths(ProviderFailureTarget::WalEntry) + .is_empty(), + "the wrapper must have reached the detached WAL persistence task" + ); + assert!( + visible_rows(&db).await.is_empty(), + "neither a clean WAL ack nor a failed WAL invocation is graph publication" + ); + + tokio::time::timeout( + PROVIDER_FOLD_TIMEOUT, + db.failpoint_stream_b1_for_test(TABLE, None, 0), + ) + .await + .expect("provider recovery fold must settle") + .expect("the earlier durable WAL row must remain replayable"); + assert_eq!( + visible_rows(&db).await, + vec![("acknowledged".to_string(), 2)], + "provider exhaustion must neither lose a prior clean ack nor invent durability for the failed WAL write" + ); + assert_eq!( + db.snapshot_of(ReadTarget::branch("main")) + .await + .unwrap() + .version(), + version_before + 1, + "only the recovery fold's manifest CAS may make the acknowledged row visible" + ); + }) + .await; +} + +async fn provider_generation_failure_retains_unreferenced_generation_at( + uri: &str, + target: ProviderFailureTarget, +) { + enroll_stream_at(uri).await; + let failure = ProviderWriteFailure::default(); + let probes = QueryIoProbes { + table_wrapper: Some(Arc::new(failure.clone())), + ..Default::default() + }; + + with_query_io_probes(probes, async { + let db = Arc::new( + Omnigraph::open(uri) + .await + .expect("the enrolled fixture must reopen under the provider wrapper"), + ); + let version_before = db + .snapshot_of(ReadTarget::branch("main")) + .await + .unwrap() + .version(); + let row = physical_batch(&db, &[("retained-after-failure".to_string(), 41)]).await; + db.failpoint_stream_b1_for_test(TABLE, Some(row), 0) + .await + .expect("the row must be durably acknowledged before the fold failure"); + let before_failure = mem_wal_inventory(uri).await; + assert!( + before_failure.generation_roots.is_empty(), + "the fixture must begin with WAL only and no flushed generation" + ); + + // The complete-output cell fails the shard-manifest publication after + // every generation object exists. The partial-output cell fails the + // later bloom write after the generation's Lance data objects exist but + // before the MemWAL generation envelope is complete. Neither path has + // shard-manifest authority for its randomized root. + failure.arm(target); + let error = tokio::time::timeout( + PROVIDER_FOLD_TIMEOUT, + db.failpoint_stream_b1_for_test(TABLE, None, 0), + ) + .await + .expect("the failed generation publication must settle") + .expect_err("a provider failure after the physical cut must block the fold"); + failure.disarm(); + match error { + OmniError::RecoveryRequired { + operation_id, + reason, + } => { + assert!(operation_id.starts_with("stream-cut:"), "{operation_id}"); + assert!(reason.contains(PROVIDER_FAILURE_MESSAGE), "{reason}"); + } + other => panic!("post-cut provider failure must require recovery: {other:?}"), + } + assert!( + !failure.failed_paths(target).is_empty(), + "the wrapper must reach the selected provider failure after generation output" + ); + if target == ProviderFailureTarget::PartialGeneration { + assert!( + failure.successful_generation_writes() > 0, + "partial-generation injection must follow at least one successful generation object write" + ); + assert!( + failure + .failed_paths(target) + .iter() + .all(|path| path.ends_with("/bloom_filter.bin")), + "the partial-generation cell must fail the deterministic post-data bloom boundary" + ); + } + assert!(visible_rows(&db).await.is_empty()); + assert_eq!( + db.snapshot_of(ReadTarget::branch("main")) + .await + .unwrap() + .version(), + version_before, + "a flushed generation is not graph-visible without the graph manifest CAS" + ); + + let after_failure = mem_wal_inventory(uri).await; + assert_inventory_retained( + &before_failure, + &after_failure, + "failed generation publication", + ); + let failed_roots = after_failure.generation_roots.clone(); + assert!( + !failed_roots.is_empty(), + "generation output must exist before its injected shard-manifest failure" + ); + let referenced_after_failure = after_failure.referenced_generation_roots.clone(); + assert!( + failed_roots.is_disjoint(&referenced_after_failure), + "a provider-refused generation publication must not authorize its randomized output: failed={failed_roots:?}, referenced={referenced_after_failure:?}" + ); + if target == ProviderFailureTarget::PartialGeneration { + for root in &failed_roots { + assert!( + !after_failure + .objects + .contains_key(&format!("{root}/bloom_filter.bin")), + "the failed bloom write must leave a physically incomplete generation envelope" + ); + assert!( + !after_failure + .objects + .keys() + .any(|path| path_is_at_or_under(path, &format!("{root}/_pk_index"))), + "failure before the PK sidecar must leave a physically incomplete generation envelope" + ); + } + } + // From this point the orphan roots are completely identified. The + // independent inventory reads above intentionally bypass this engine + // wrapper; every subsequent engine access remains in this log. + let orphan_access_checkpoint = failure.access_checkpoint(); + + // The retained WAL/cut owns progress. A new row cannot open a side + // generation around it; recovery must fold the exact durable prefix. + let blocked = physical_batch(&db, &[("must-not-pass-recovery".to_string(), 42)]).await; + let blocked_error = tokio::time::timeout( + PROVIDER_PUT_TIMEOUT, + db.failpoint_stream_b1_for_test(TABLE, Some(blocked), 1), + ) + .await + .expect("fold-only admission refusal must settle") + .expect_err("unresolved retained state must block fresh admission"); + assert!( + matches!(blocked_error, OmniError::FoldRequired { .. }), + "retained provider residue must reopen fold-only: {blocked_error:?}" + ); + failure.assert_no_access_under_roots_since( + orphan_access_checkpoint, + &failed_roots, + "blocked admission", + ); + + tokio::time::timeout( + PROVIDER_FOLD_TIMEOUT, + db.failpoint_stream_b1_for_test(TABLE, None, 0), + ) + .await + .expect("the recovery fold must settle after provider service resumes") + .expect("the retained WAL prefix must fold through a fresh generation"); + assert_eq!( + visible_rows(&db).await, + vec![("retained-after-failure".to_string(), 41)], + "the unreferenced residue must be inert while the authoritative retry publishes exactly once" + ); + + let after_retry = mem_wal_inventory(uri).await; + assert_inventory_retained( + &after_failure, + &after_retry, + "successful recovery retry", + ); + let retry_roots = after_retry.generation_roots.clone(); + let fresh_retry_roots = retry_roots + .difference(&failed_roots) + .cloned() + .collect::>(); + assert_eq!( + fresh_retry_roots.len(), + 1, + "one recovery retry must materialize exactly one fresh randomized generation: failed={failed_roots:?}, retry={retry_roots:?}" + ); + let referenced_after_retry = after_retry.referenced_generation_roots.clone(); + assert!( + failed_roots.is_disjoint(&referenced_after_retry), + "recovery must not retroactively adopt provider-refused output" + ); + assert!( + fresh_retry_roots.is_subset(&referenced_after_retry), + "the exact fresh retry root must become shard-manifest-authoritative: fresh={fresh_retry_roots:?}, referenced={referenced_after_retry:?}" + ); + failure.assert_no_access_under_roots_since( + orphan_access_checkpoint, + &failed_roots, + "authoritative recovery retry", + ); + drop(db); + + let reopened = Arc::new( + Omnigraph::open(uri) + .await + .expect("the recovered graph must survive a cold reopen"), + ); + assert_eq!( + visible_rows(&reopened).await, + vec![("retained-after-failure".to_string(), 41)], + "cold recovery must ignore retained unreferenced output and preserve one visible row" + ); + let after_reopen = mem_wal_inventory(uri).await; + assert_inventory_retained(&after_failure, &after_reopen, "cold graph reopen"); + assert!( + failed_roots.is_subset(&after_reopen.generation_roots), + "retain-all permits no raw `_mem_wal` deletion during reopen" + ); + let referenced_after_reopen = after_reopen.referenced_generation_roots.clone(); + assert!( + failed_roots.is_disjoint(&referenced_after_reopen), + "cold reopen must keep every provider-refused root non-authoritative" + ); + failure.assert_no_access_under_roots_since( + orphan_access_checkpoint, + &failed_roots, + "recovery plus cold reopen", + ); + }) + .await; +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +#[serial] +async fn provider_shard_manifest_failure_retains_unreferenced_generation_local() { + let _scenario = FailScenario::setup(); + let dir = tempfile::tempdir().unwrap(); + provider_generation_failure_retains_unreferenced_generation_at( + dir.path().to_str().unwrap(), + ProviderFailureTarget::ShardManifest, + ) + .await; +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +#[serial] +async fn provider_partial_generation_failure_retains_inert_unreferenced_residue_local() { + let _scenario = FailScenario::setup(); + let dir = tempfile::tempdir().unwrap(); + provider_generation_failure_retains_unreferenced_generation_at( + dir.path().to_str().unwrap(), + ProviderFailureTarget::PartialGeneration, + ) + .await; +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +#[serial] +async fn s3_provider_shard_manifest_failure_retains_unreferenced_generation() { + let _scenario = FailScenario::setup(); + let Some(uri) = helpers::s3_test_graph_uri("memwal-b2a-provider") else { + eprintln!("SKIP s3_provider_shard_manifest_failure_retains_unreferenced_generation"); + return; + }; + + provider_generation_failure_retains_unreferenced_generation_at( + &uri, + ProviderFailureTarget::ShardManifest, + ) + .await; + + // This destroys the test's unique isolated graph root after all retention + // assertions. It is fixture teardown, not a production MemWAL GC path. + let (store, root) = lance_io::object_store::ObjectStore::from_uri(&uri) + .await + .expect("configured S3/RustFS fixture must resolve for cleanup"); + store + .remove_dir_all(root) + .await + .expect("configured S3/RustFS fixture cleanup must succeed"); +} diff --git a/crates/omnigraph/tests/memwal_stream_cost.rs b/crates/omnigraph/tests/memwal_stream_cost.rs index 78d6a3e0..5997ed53 100644 --- a/crates/omnigraph/tests/memwal_stream_cost.rs +++ b/crates/omnigraph/tests/memwal_stream_cost.rs @@ -7,6 +7,11 @@ //! metadata, and the shared graph-manifest publisher are different costs with //! different scaling variables. Combining them into one latency number would //! hide exactly the regressions this instrument exists to expose. +//! +//! B2a's retained-history sweep keeps wall-clock numbers diagnostic. The +//! real private B1 path grows retained shard metadata and the folded base table +//! together, so the sweep reports each observable term but does not attribute a +//! slope to retained objects alone or turn local timings into a product claim. mod helpers; @@ -19,12 +24,11 @@ use arrow_array::{ArrayRef, BooleanArray, Int32Array, RecordBatch, StringArray}; use arrow_schema::{DataType, Field, Schema}; use fail::FailScenario; use futures::TryStreamExt; +use lance::dataset::mem_wal::schema_with_tombstone; use lance::dataset::mem_wal::write::StoredBatch; -use lance::dataset::mem_wal::{ShardManifestStore, schema_with_tombstone}; use lance_core::utils::bloomfilter::sbbf::Sbbf; use lance_index::mem_wal::{FlushedGeneration, ShardId, ShardManifest, ShardStatus}; use lance_io::utils::tracking_store::{IOTracker, IoStats}; -use object_store::path::Path as ObjectPath; use omnigraph::db::{Omnigraph, ReadTarget}; use omnigraph::error::OmniError; use omnigraph::failpoints::{ScopedFailPoint, names}; @@ -33,6 +37,7 @@ use omnigraph::storage::storage_for_uri; use serial_test::serial; use helpers::cost::with_raw_io_trackers; +use helpers::memwal::{CurrentMemWalInventory, MemWalObjectKind}; const STREAM_SCHEMA: &str = r#" node Person { score: I32 } @@ -57,175 +62,11 @@ const NO_AUTO_ROLL_BATCHES: usize = 8_193; const RSS_CHILD_ENV: &str = "OMNIGRAPH_MEMWAL_COST_CHILD"; const SURVEYED_LANCE_REV: &str = "cec0b7dffe2d85c7e66dbe9d1f3891c297903a1d"; -#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)] -enum MemWalObjectKind { - Wal, - ShardManifestVersion, - ShardManifestHint, - GenerationData, - GenerationManifest, - GenerationTransaction, - GenerationDeletion, - GenerationPkSidecar, - GenerationBloom, - GenerationUserIndex, - Unknown, -} - -#[derive(Debug, Clone, Default)] -struct CurrentMemWalInventory { - objects: BTreeMap, - generation_roots: BTreeSet, - referenced_generation_roots: BTreeSet, -} - -impl CurrentMemWalInventory { - fn object_count(&self, kind: MemWalObjectKind) -> usize { - self.objects - .values() - .filter(|(candidate, _)| *candidate == kind) - .count() - } - - fn bytes(&self, kind: MemWalObjectKind) -> u64 { - self.objects - .values() - .filter(|(candidate, _)| *candidate == kind) - .map(|(_, bytes)| *bytes) - .sum() - } - - fn immutable_object_bytes(&self) -> u64 { - self.objects - .values() - .filter(|(kind, _)| *kind != MemWalObjectKind::ShardManifestHint) - .map(|(_, bytes)| *bytes) - .sum() - } - - fn unknown_paths(&self) -> Vec<&str> { - self.objects - .iter() - .filter_map(|(path, (kind, _))| { - (*kind == MemWalObjectKind::Unknown).then_some(path.as_str()) - }) - .collect() - } - - fn generation_subtree_objects(&self) -> BTreeMap { - self.objects - .iter() - .filter(|(_, (kind, _))| { - matches!( - kind, - MemWalObjectKind::GenerationData - | MemWalObjectKind::GenerationManifest - | MemWalObjectKind::GenerationTransaction - | MemWalObjectKind::GenerationDeletion - | MemWalObjectKind::GenerationPkSidecar - | MemWalObjectKind::GenerationBloom - | MemWalObjectKind::GenerationUserIndex - ) - }) - .map(|(path, value)| (path.clone(), *value)) - .collect() - } - - fn assert_path_class_size_retained_by(&self, later: &Self) { - for (path, (kind, bytes)) in &self.objects { - if *kind == MemWalObjectKind::ShardManifestHint { - continue; - } - assert_eq!( - later.objects.get(path), - Some(&(*kind, *bytes)), - "retain-all evidence lost or changed the class/size of listed object {path}" - ); - } - } -} - -fn is_generation_root(component: &str) -> bool { - let Some((hash, generation)) = component.split_once("_gen_") else { - return false; - }; - let Ok(generation_number) = generation.parse::() else { - return false; - }; - hash.len() == 8 - && hash - .bytes() - .all(|byte| byte.is_ascii_digit() || (b'a'..=b'f').contains(&byte)) - && generation_number > 0 - && generation_number.to_string() == generation -} - -fn parse_positive_bit_reversed_filename(filename: &str, extension: &str) -> Option { - let Some(stem) = filename.strip_suffix(&format!(".{extension}")) else { - return None; - }; - if stem.len() != 64 || !stem.bytes().all(|byte| matches!(byte, b'0' | b'1')) { - return None; - } - let reversed = u64::from_str_radix(stem, 2).ok()?; - let value = reversed.reverse_bits(); - (value > 0).then_some(value) -} - -fn parse_canonical_shard_id(value: &str) -> Option { - let shard_id = ShardId::parse_str(value).ok()?; - (shard_id.as_hyphenated().to_string() == value).then_some(shard_id) -} - -fn join_object_path(base: &str, suffix: &str) -> String { - if base.is_empty() { - suffix.to_string() - } else { - format!("{base}/{suffix}") - } -} - fn classify_mem_wal_object( components: &[&str], mem_wal_index: usize, ) -> (MemWalObjectKind, Option) { - let tail = &components[mem_wal_index + 1..]; - if tail.len() < 3 || parse_canonical_shard_id(tail[0]).is_none() { - return (MemWalObjectKind::Unknown, None); - } - - match tail[1] { - "wal" - if tail.len() == 3 - && parse_positive_bit_reversed_filename(tail[2], "arrow").is_some() => - { - (MemWalObjectKind::Wal, None) - } - "manifest" if tail.len() == 3 && tail[2] == "version_hint.json" => { - (MemWalObjectKind::ShardManifestHint, None) - } - "manifest" - if tail.len() == 3 - && parse_positive_bit_reversed_filename(tail[2], "binpb").is_some() => - { - (MemWalObjectKind::ShardManifestVersion, None) - } - generation if is_generation_root(generation) => { - let root = components[..mem_wal_index + 3].join("/"); - let kind = match tail.get(2).copied() { - Some("bloom_filter.bin") if tail.len() == 3 => MemWalObjectKind::GenerationBloom, - Some("data") if tail.len() >= 4 => MemWalObjectKind::GenerationData, - Some("_versions") if tail.len() >= 4 => MemWalObjectKind::GenerationManifest, - Some("_transactions") if tail.len() >= 4 => MemWalObjectKind::GenerationTransaction, - Some("_deletions") if tail.len() >= 4 => MemWalObjectKind::GenerationDeletion, - Some("_pk_index") if tail.len() >= 4 => MemWalObjectKind::GenerationPkSidecar, - Some("_indices") if tail.len() >= 4 => MemWalObjectKind::GenerationUserIndex, - _ => MemWalObjectKind::Unknown, - }; - (kind, Some(root)) - } - _ => (MemWalObjectKind::Unknown, None), - } + helpers::memwal::classify_mem_wal_object(components, mem_wal_index) } fn validated_referenced_generation_roots( @@ -235,165 +76,17 @@ fn validated_referenced_generation_roots( listed_generation_roots: &BTreeSet, listed_manifest_versions: &BTreeSet, ) -> BTreeSet { - assert_eq!( - manifest.shard_id, directory_shard_id, - "Gate R0 must fail closed when decoded shard authority disagrees with its directory" - ); - assert!( - manifest.version > 0, - "Gate R0 must fail closed on a zero shard-manifest version" - ); - let latest_listed_version = listed_manifest_versions - .last() - .copied() - .expect("Gate R0 must fail closed when a shard has no listed manifest version"); - assert_eq!( - latest_listed_version, manifest.version, - "Gate R0 must fail closed when the latest shard-manifest filename disagrees with its body" - ); - let expected_manifest_version_count = usize::try_from(manifest.version) - .expect("Gate R0 must fail closed when shard-manifest authority exceeds usize"); - assert_eq!( - listed_manifest_versions.len(), - expected_manifest_version_count, - "Gate R0 must fail closed on a gapped or noncanonical shard-manifest version chain" - ); - assert!( - manifest.current_generation > 0, - "Gate R0 must fail closed on zero next-generation allocation authority" - ); - - let shard_prefix = join_object_path( + helpers::memwal::validated_referenced_generation_roots( table_base, - &format!("_mem_wal/{}", directory_shard_id.as_hyphenated()), - ); - let mut generations = BTreeSet::new(); - let mut paths = BTreeSet::new(); - let mut roots = BTreeSet::new(); - for flushed in &manifest.flushed_generations { - assert!( - flushed.generation > 0 && generations.insert(flushed.generation), - "Gate R0 must fail closed on a zero or duplicate flushed generation" - ); - assert!( - paths.insert(flushed.path.clone()), - "Gate R0 must fail closed on a duplicate flushed-generation path" - ); - assert!( - is_generation_root(&flushed.path), - "Gate R0 must fail closed on a noncanonical flushed-generation path" - ); - let (_, path_generation) = flushed - .path - .split_once("_gen_") - .expect("validated generation root must contain _gen_"); - assert_eq!( - path_generation.parse::().unwrap(), - flushed.generation, - "Gate R0 must fail closed when generation authority disagrees with its path" - ); - assert!( - manifest.current_generation > flushed.generation, - "Gate R0 must fail closed when next-generation allocation authority has not advanced past a flushed generation" - ); - let root = join_object_path(&shard_prefix, &flushed.path); - assert!( - listed_generation_roots.contains(&root), - "Gate R0 must fail closed when shard authority references a missing listed generation" - ); - assert!( - roots.insert(root), - "Gate R0 must fail closed on duplicate generation authority" - ); - } - roots + directory_shard_id, + manifest, + listed_generation_roots, + listed_manifest_versions, + ) } -/// Strict current-object inventory for the part of retained storage that stock -/// Lance exposes through ordinary LIST. This deliberately does not pretend to -/// see incomplete multipart uploads, superseded provider versions, delete -/// markers, local staged temp files, or billed storage. async fn current_mem_wal_inventory(uri: &str) -> CurrentMemWalInventory { - let (store, root_path) = lance_io::object_store::ObjectStore::from_uri(uri) - .await - .expect("Gate R0 fixture URI must resolve"); - let listed = store - .inner - .list(Some(&root_path)) - .try_collect::>() - .await - .expect("Gate R0 inventory must fail closed on a listing error"); - - let mut inventory = CurrentMemWalInventory::default(); - let mut shards = BTreeSet::new(); - let mut manifest_versions = BTreeMap::<(String, ShardId), BTreeSet>::new(); - for metadata in listed { - let path = metadata.location.as_ref().to_string(); - let components = path.split('/').collect::>(); - let Some(mem_wal_index) = components - .iter() - .position(|component| *component == "_mem_wal") - else { - continue; - }; - let (kind, generation_root) = classify_mem_wal_object(&components, mem_wal_index); - if let Some(root) = generation_root { - inventory.generation_roots.insert(root); - } - let bytes = u64::try_from(metadata.size).expect("object size must fit u64"); - assert!( - inventory - .objects - .insert(path.clone(), (kind, bytes)) - .is_none(), - "object listing returned duplicate path {path}" - ); - - let tail = &components[mem_wal_index + 1..]; - if let Some(shard) = tail - .first() - .and_then(|value| parse_canonical_shard_id(value)) - { - let table_base = components[..mem_wal_index].join("/"); - shards.insert((table_base.clone(), shard)); - if kind == MemWalObjectKind::ShardManifestVersion { - let version = parse_positive_bit_reversed_filename(tail[2], "binpb") - .expect("classified shard-manifest version must decode"); - assert!( - manifest_versions - .entry((table_base, shard)) - .or_default() - .insert(version), - "object listing returned duplicate decoded shard-manifest version" - ); - } - } - } - - for (table_base, shard_id) in shards { - let table_path = - ObjectPath::parse(&table_base).expect("listed table base must remain a valid path"); - let manifest_store = ShardManifestStore::new(Arc::clone(&store), &table_path, shard_id, 2); - let manifest = manifest_store - .read_latest() - .await - .expect("Gate R0 must fail closed on an unreadable shard manifest") - .expect("Gate R0 must fail closed when a listed shard has no latest manifest"); - let listed_manifest_versions = manifest_versions - .get(&(table_base.clone(), shard_id)) - .expect("Gate R0 must fail closed when a shard has no listed manifest version"); - inventory - .referenced_generation_roots - .extend(validated_referenced_generation_roots( - &table_base, - shard_id, - &manifest, - &inventory.generation_roots, - listed_manifest_versions, - )); - } - - inventory + helpers::memwal::current_mem_wal_inventory(uri).await } async fn current_recovery_sidecars(uri: &str) -> Vec { @@ -430,6 +123,41 @@ struct PathIo { generation_data_writes: u64, pk_sidecar_reads: u64, pk_sidecar_writes: u64, + memwal_atomic_temp_delete_requests: u64, + memwal_canonical_delete_requests: u64, +} + +fn is_lance_atomic_memwal_temp_path(path: &str) -> bool { + fn canonical_lower_uuid(value: &str) -> bool { + value.len() == 36 + && value.bytes().enumerate().all(|(index, byte)| match index { + 8 | 13 | 18 | 23 => byte == b'-', + _ => byte.is_ascii_digit() || (b'a'..=b'f').contains(&byte), + }) + } + + let components = path.split('/').collect::>(); + let Some(mem_wal) = components + .iter() + .position(|component| *component == "_mem_wal") + else { + return false; + }; + let tail = &components[mem_wal + 1..]; + if tail.len() != 3 || tail[1] != "manifest" { + return false; + } + let Ok(shard) = ShardId::parse_str(tail[0]) else { + return false; + }; + if shard.as_hyphenated().to_string() != tail[0] { + return false; + } + let Some((manifest_name, temporary_id)) = tail[2].split_once(".tmp.") else { + return false; + }; + helpers::memwal::parse_positive_bit_reversed_filename(manifest_name, "binpb").is_some() + && canonical_lower_uuid(temporary_id) } impl PathIo { @@ -486,6 +214,47 @@ impl PathIo { if write && pk_sidecar { out.pk_sidecar_writes += 1; } + if request.method == "delete" && memwal { + if is_lance_atomic_memwal_temp_path(path) { + out.memwal_atomic_temp_delete_requests += 1; + } else { + out.memwal_canonical_delete_requests += 1; + } + } + } + out + } +} + +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] +struct RetainedRootIo { + reads: u64, + writes: u64, + deletes: u64, +} + +fn path_is_under_any_root(path: &str, roots: &BTreeSet) -> bool { + roots.iter().any(|root| { + path == root + || path + .strip_prefix(root) + .is_some_and(|suffix| suffix.starts_with('/')) + }) +} + +impl RetainedRootIo { + fn from_stats(stats: &IoStats, roots: &BTreeSet) -> Self { + let mut out = Self::default(); + for request in &stats.requests { + let path = request.path.as_ref(); + if !path_is_under_any_root(path, roots) { + continue; + } + match request.method { + "delete" => out.deletes += 1, + "put_opts" | "put_part" | "copy" | "rename" => out.writes += 1, + _ => out.reads += 1, + } } out } @@ -578,6 +347,39 @@ struct RetainedMetadataSample { inventory: CurrentMemWalInventory, } +#[derive(Debug)] +struct RetentionTermSample { + elapsed_us: u128, + table: PathIo, + manifest: PathIo, + adapter: AdapterIo, + retained_root_io: RetainedRootIo, +} + +#[derive(Debug)] +struct RetentionDepthSample { + retained_generations_before: u64, + retained_generations_after: u64, + before: CurrentMemWalInventory, + after_ack: CurrentMemWalInventory, + after_replay: CurrentMemWalInventory, + after_fold: CurrentMemWalInventory, + warm_ack: RetentionTermSample, + cold_reopen_replay: RetentionTermSample, + fold: RetentionTermSample, + visibility_probe: RetentionTermSample, + manifest_version_before: u64, + manifest_version_after_ack: u64, + manifest_version_after_replay: u64, + manifest_version_after_fold: u64, + table_version_before: u64, + table_version_after_ack: u64, + table_version_after_replay: u64, + table_version_after_fold: u64, + visible_rows_after_fold: usize, + whole_process_peak_rss_bytes: Option, +} + #[derive(Debug)] struct WidestRetainedGrowthSample { before: CurrentMemWalInventory, @@ -1069,6 +871,495 @@ fn generation_data_file_bytes(root: &Path) -> u64 { total } +fn retention_term_sample( + started: Instant, + table_tracker: &IOTracker, + manifest_tracker: &IOTracker, + adapter_counts: &StorageReadCounts, + adapter_before: AdapterIo, + retained_roots: &BTreeSet, +) -> RetentionTermSample { + let elapsed_us = started.elapsed().as_micros(); + let table_stats = table_tracker.incremental_stats(); + let manifest_stats = manifest_tracker.incremental_stats(); + RetentionTermSample { + elapsed_us, + table: PathIo::from_stats(&table_stats), + manifest: PathIo::from_stats(&manifest_stats), + adapter: AdapterIo::snapshot(adapter_counts).since(adapter_before), + retained_root_io: RetainedRootIo::from_stats(&table_stats, retained_roots), + } +} + +fn reset_retention_term_trackers( + table_tracker: &IOTracker, + manifest_tracker: &IOTracker, + adapter_counts: &StorageReadCounts, +) -> AdapterIo { + let _ = table_tracker.incremental_stats(); + let _ = manifest_tracker.incremental_stats(); + AdapterIo::snapshot(adapter_counts) +} + +#[cfg(unix)] +fn current_process_peak_rss_bytes() -> Option { + let mut rusage: libc::rusage = unsafe { std::mem::zeroed() }; + let result = unsafe { libc::getrusage(libc::RUSAGE_SELF, &mut rusage) }; + assert_eq!(result, 0, "getrusage(RUSAGE_SELF) failed"); + Some(normalized_peak_rss_bytes(&rusage)) +} + +#[cfg(not(unix))] +fn current_process_peak_rss_bytes() -> Option { + None +} + +async fn retained_history_scaling_samples_at( + uri: &str, + depths: &[u64], +) -> Vec { + assert!(!depths.is_empty()); + assert!( + depths + .windows(2) + .all(|pair| pair[0] > 0 && pair[0] < pair[1]) + && depths.last().copied().unwrap() > 0, + "retained-history sweep depths must be positive and strictly increasing" + ); + + init_enrolled(uri, STREAM_SCHEMA).await; + let table_tracker = IOTracker::default(); + let manifest_tracker = IOTracker::default(); + + with_raw_io_trackers(&table_tracker, &manifest_tracker, async { + let (adapter, adapter_counts) = CountingStorageAdapter::new(storage_for_uri(uri).unwrap()); + let mut db = Arc::new( + Omnigraph::open_with_storage(uri, Arc::clone(&adapter)) + .await + .unwrap(), + ); + let schema = table_schema(&db).await; + let mut retained_generations = 0u64; + let mut visible_rows = 0usize; + let mut ordinal = 0u64; + let mut samples = Vec::new(); + + for &depth in depths { + while retained_generations < depth { + let batch = score_batch( + Arc::clone(&schema), + &format!("retention-fill-{retained_generations}"), + 1, + ); + db.failpoint_stream_b1_for_test(TABLE, Some(batch), ordinal) + .await + .unwrap(); + ordinal += 1; + db.failpoint_stream_b1_for_test(TABLE, None, 0) + .await + .unwrap(); + retained_generations += 1; + visible_rows += 1; + } + + let before = current_mem_wal_inventory(uri).await; + assert_eq!( + before.generation_roots.len(), + usize::try_from(depth).unwrap(), + "fixture must reach the requested retained-generation depth exactly" + ); + let retained_roots = before.generation_roots.clone(); + let before_snapshot = db.snapshot_of(ReadTarget::branch("main")).await.unwrap(); + let manifest_version_before = before_snapshot.version(); + let table_version_before = before_snapshot.entry(TABLE).unwrap().table_version; + + // Establish one resident writer, then measure only the second put. + // This keeps warm acknowledgement separate from cold claim/replay. + let warmup = score_batch(Arc::clone(&schema), &format!("retention-{depth}-warmup"), 1); + db.failpoint_stream_b1_for_test(TABLE, Some(warmup), ordinal) + .await + .unwrap(); + ordinal += 1; + let adapter_before = + reset_retention_term_trackers(&table_tracker, &manifest_tracker, &adapter_counts); + let started = Instant::now(); + let measured = score_batch( + Arc::clone(&schema), + &format!("retention-{depth}-measured"), + 1, + ); + db.failpoint_stream_b1_for_test(TABLE, Some(measured), ordinal) + .await + .unwrap(); + ordinal += 1; + let warm_ack = retention_term_sample( + started, + &table_tracker, + &manifest_tracker, + &adapter_counts, + adapter_before, + &retained_roots, + ); + let after_ack = current_mem_wal_inventory(uri).await; + let after_ack_snapshot = db.snapshot_of(ReadTarget::branch("main")).await.unwrap(); + let manifest_version_after_ack = after_ack_snapshot.version(); + let table_version_after_ack = after_ack_snapshot.entry(TABLE).unwrap().table_version; + + // Add one definitely durable but acknowledgement-ambiguous entry to + // retire the resident writer. The measured cold interval then + // includes graph reopen, shard claim, and authoritative WAL replay, + // while stopping before generation materialization. + let ambiguous = score_batch( + Arc::clone(&schema), + &format!("retention-{depth}-ambiguous"), + 1, + ); + let ambiguous_retired = helpers::failpoint::Rendezvous::park_first( + names::STREAM_B1_AFTER_RETIREMENT_RELEASE, + ); + let error = { + let _after_durable = + ScopedFailPoint::new(names::STREAM_B1_AFTER_WATCHER_SUCCESS, "return"); + db.failpoint_stream_b1_for_test(TABLE, Some(ambiguous), ordinal) + .await + .expect_err("the injected post-durability boundary must retire the writer") + }; + assert!(matches!(error, OmniError::AckUnknown { .. }), "{error:?}"); + ordinal += 1; + ambiguous_retired.wait_until_reached().await; + ambiguous_retired.release(); + drop(db); + + let adapter_before = + reset_retention_term_trackers(&table_tracker, &manifest_tracker, &adapter_counts); + let started = Instant::now(); + let replay_before_abort = + helpers::failpoint::Rendezvous::park_first(names::STREAM_B1_BEFORE_ABORT); + let replay_retired = helpers::failpoint::Rendezvous::park_first( + names::STREAM_B1_AFTER_RETIREMENT_RELEASE, + ); + db = Arc::new( + Omnigraph::open_with_storage(uri, Arc::clone(&adapter)) + .await + .unwrap(), + ); + let error = { + let _before_seal = + ScopedFailPoint::new(names::STREAM_B1_BEFORE_FORCE_SEAL, "return"); + db.failpoint_stream_b1_for_test(TABLE, None, 0) + .await + .expect_err("cold replay measurement must stop before generation output") + }; + assert!( + error + .to_string() + .contains(names::STREAM_B1_BEFORE_FORCE_SEAL), + "{error:?}" + ); + replay_before_abort.wait_until_reached().await; + let cold_reopen_replay = retention_term_sample( + started, + &table_tracker, + &manifest_tracker, + &adapter_counts, + adapter_before, + &retained_roots, + ); + replay_before_abort.release(); + replay_retired.wait_until_reached().await; + replay_retired.release(); + let after_replay = current_mem_wal_inventory(uri).await; + let after_replay_snapshot = db.snapshot_of(ReadTarget::branch("main")).await.unwrap(); + let manifest_version_after_replay = after_replay_snapshot.version(); + let table_version_after_replay = + after_replay_snapshot.entry(TABLE).unwrap().table_version; + + let adapter_before = + reset_retention_term_trackers(&table_tracker, &manifest_tracker, &adapter_counts); + let started = Instant::now(); + db.failpoint_stream_b1_for_test(TABLE, None, 0) + .await + .expect("the replayed generation must fold and publish"); + let fold = retention_term_sample( + started, + &table_tracker, + &manifest_tracker, + &adapter_counts, + adapter_before, + &retained_roots, + ); + + let adapter_before = + reset_retention_term_trackers(&table_tracker, &manifest_tracker, &adapter_counts); + let started = Instant::now(); + let after_fold_snapshot = db.snapshot_of(ReadTarget::branch("main")).await.unwrap(); + let visibility_probe = retention_term_sample( + started, + &table_tracker, + &manifest_tracker, + &adapter_counts, + adapter_before, + &retained_roots, + ); + let manifest_version_after_fold = after_fold_snapshot.version(); + let table_version_after_fold = after_fold_snapshot.entry(TABLE).unwrap().table_version; + let after_fold = current_mem_wal_inventory(uri).await; + assert!( + current_recovery_sidecars(uri).await.is_empty(), + "successful retained-history fold must retire its recovery sidecar" + ); + + retained_generations += 1; + visible_rows += 3; + let visible_rows_after_fold = helpers::count_rows(&db, TABLE).await; + samples.push(RetentionDepthSample { + retained_generations_before: depth, + retained_generations_after: retained_generations, + before, + after_ack, + after_replay, + after_fold, + warm_ack, + cold_reopen_replay, + fold, + visibility_probe, + manifest_version_before, + manifest_version_after_ack, + manifest_version_after_replay, + manifest_version_after_fold, + table_version_before, + table_version_after_ack, + table_version_after_replay, + table_version_after_fold, + visible_rows_after_fold, + whole_process_peak_rss_bytes: current_process_peak_rss_bytes(), + }); + assert_eq!(visible_rows_after_fold, visible_rows); + } + samples + }) + .await +} + +async fn retained_history_scaling_samples_local(depths: &[u64]) -> Vec { + let dir = tempfile::tempdir().unwrap(); + retained_history_scaling_samples_at(dir.path().to_str().unwrap(), depths).await +} + +fn assert_retained_history_scaling_sample(sample: &RetentionDepthSample) { + let before_depth = usize::try_from(sample.retained_generations_before).unwrap(); + let after_depth = usize::try_from(sample.retained_generations_after).unwrap(); + assert_eq!(after_depth, before_depth + 1); + for inventory in [ + &sample.before, + &sample.after_ack, + &sample.after_replay, + &sample.after_fold, + ] { + assert!( + inventory.unknown_paths().is_empty(), + "retained-history census missed paths: {:?}", + inventory.unknown_paths() + ); + assert_eq!( + inventory.generation_roots, inventory.referenced_generation_roots, + "success-only retained-history sweep must not adopt or invent orphan roots" + ); + } + assert_eq!(sample.before.generation_roots.len(), before_depth); + assert_eq!(sample.after_ack.generation_roots.len(), before_depth); + assert_eq!(sample.after_replay.generation_roots.len(), before_depth); + assert_eq!(sample.after_fold.generation_roots.len(), after_depth); + sample + .before + .assert_path_class_size_retained_by(&sample.after_ack); + sample + .after_ack + .assert_path_class_size_retained_by(&sample.after_replay); + sample + .after_replay + .assert_path_class_size_retained_by(&sample.after_fold); + assert!( + sample.after_fold.immutable_object_bytes() > sample.before.immutable_object_bytes(), + "one additional retained generation must increase advisory current-listed bytes" + ); + + for (name, term) in [ + ("warm ack", &sample.warm_ack), + ("cold reopen/replay", &sample.cold_reopen_replay), + ("fold", &sample.fold), + ("visibility probe", &sample.visibility_probe), + ] { + assert_eq!( + term.retained_root_io, + RetainedRootIo::default(), + "{name} touched an older retained generation root: {term:#?}" + ); + assert_eq!( + term.table.memwal_canonical_delete_requests, 0, + "{name} requested deletion of a canonical MemWAL object: {term:#?}" + ); + } + assert!(sample.warm_ack.table.wal_writes > 0, "{sample:#?}"); + assert_eq!(sample.warm_ack.table.generation_writes, 0, "{sample:#?}"); + assert_eq!(sample.warm_ack.manifest.total_writes, 0, "{sample:#?}"); + assert!(sample.cold_reopen_replay.table.wal_reads > 0, "{sample:#?}"); + assert_eq!( + sample.cold_reopen_replay.table.generation_writes, 0, + "{sample:#?}" + ); + assert!(sample.fold.table.generation_reads > 0, "{sample:#?}"); + assert!(sample.fold.table.generation_writes > 0, "{sample:#?}"); + assert!(sample.fold.manifest.total_writes > 0, "{sample:#?}"); + assert_eq!(sample.visibility_probe.table.total_writes, 0, "{sample:#?}"); + assert_eq!( + sample.visibility_probe.manifest.total_writes, 0, + "{sample:#?}" + ); + + assert_eq!( + sample.manifest_version_after_ack, sample.manifest_version_before, + "acknowledgement must remain graph-invisible" + ); + assert_eq!( + sample.manifest_version_after_replay, sample.manifest_version_before, + "cold replay must remain graph-invisible" + ); + assert_eq!( + sample.manifest_version_after_fold, + sample.manifest_version_before + 1, + "one fold must have exactly one graph visibility point" + ); + assert_eq!( + sample.table_version_after_ack, sample.table_version_before, + "acknowledgement must not advance the base table" + ); + assert_eq!( + sample.table_version_after_replay, sample.table_version_before, + "cold replay must not advance the base table" + ); + assert_eq!( + sample.table_version_after_fold, + sample.table_version_before + 1, + "one fold must commit exactly one Lance table effect" + ); +} + +fn print_retained_history_scaling_sample(sample: &RetentionDepthSample, scale: &str) { + let print_term = |name: &str, term: &RetentionTermSample| { + eprintln!( + "B2a retained-history {scale} depth={} term={name} elapsed_us={} table_reads={} table_read_bytes={} table_writes={} table_written_bytes={} graph_manifest_reads={} graph_manifest_read_bytes={} graph_manifest_writes={} graph_manifest_written_bytes={} adapter_ops={} retained_root_reads={} retained_root_writes={} retained_root_deletes={} memwal_atomic_temp_delete_requests={} memwal_canonical_delete_requests={}", + sample.retained_generations_before, + term.elapsed_us, + term.table.total_reads, + term.table.total_read_bytes, + term.table.total_writes, + term.table.total_written_bytes, + term.manifest.total_reads, + term.manifest.total_read_bytes, + term.manifest.total_writes, + term.manifest.total_written_bytes, + term.adapter.operations(), + term.retained_root_io.reads, + term.retained_root_io.writes, + term.retained_root_io.deletes, + term.table.memwal_atomic_temp_delete_requests, + term.table.memwal_canonical_delete_requests, + ); + }; + print_term("warm_ack", &sample.warm_ack); + print_term("cold_reopen_replay", &sample.cold_reopen_replay); + print_term("fold", &sample.fold); + print_term("visibility_probe", &sample.visibility_probe); + eprintln!( + "B2a retained-history {scale} depth={} advisory_current_listed_before_objects={} advisory_current_listed_before_immutable_bytes={} advisory_current_listed_after_ack_objects={} advisory_current_listed_after_ack_immutable_bytes={} advisory_current_listed_after_replay_objects={} advisory_current_listed_after_replay_immutable_bytes={} advisory_current_listed_after_fold_objects={} advisory_current_listed_after_fold_immutable_bytes={} retained_roots_before={} retained_roots_after={} visible_rows={} whole_process_peak_rss_bytes={:?}", + sample.retained_generations_before, + sample.before.objects.len(), + sample.before.immutable_object_bytes(), + sample.after_ack.objects.len(), + sample.after_ack.immutable_object_bytes(), + sample.after_replay.objects.len(), + sample.after_replay.immutable_object_bytes(), + sample.after_fold.objects.len(), + sample.after_fold.immutable_object_bytes(), + sample.before.generation_roots.len(), + sample.after_fold.generation_roots.len(), + sample.visible_rows_after_fold, + sample.whole_process_peak_rss_bytes, + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +#[serial] +async fn b2a_retained_history_small_depth_keeps_terms_and_old_roots_separate() { + let _scenario = FailScenario::setup(); + let samples = retained_history_scaling_samples_local(&[1, 8]).await; + for sample in &samples { + print_retained_history_scaling_sample(sample, "ci"); + assert_retained_history_scaling_sample(sample); + } + + let shallow = &samples[0]; + let deep = &samples[1]; + assert_eq!( + deep.warm_ack.table.total_reads, shallow.warm_ack.table.total_reads, + "warm acknowledgement read-operation count must not scale with retained history" + ); + assert_eq!( + deep.warm_ack.table.total_writes, shallow.warm_ack.table.total_writes, + "warm acknowledgement write-operation count must not scale with retained history" + ); + assert_eq!( + deep.cold_reopen_replay.table.wal_reads, shallow.cold_reopen_replay.table.wal_reads, + "the fixed three-entry replay tail must keep one deterministic WAL-read shape" + ); + assert!( + deep.cold_reopen_replay.table.shard_manifest_reads + <= shallow.cold_reopen_replay.table.shard_manifest_reads + 2, + "cold shard-authority read-operation count grew with retained history: {samples:#?}" + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +#[serial] +#[ignore = "on-demand RFC-026 B2a retain-all decision-scale instrument"] +async fn b2a_retained_history_decision_scale_sweeps_to_128_generations() { + let _scenario = FailScenario::setup(); + let samples = retained_history_scaling_samples_local(&[1, 8, 32, 128]).await; + for sample in &samples { + print_retained_history_scaling_sample(sample, "decision"); + assert_retained_history_scaling_sample(sample); + } +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +#[serial] +#[ignore = "on-demand RFC-026 B2a configured-object-store decision-scale instrument"] +async fn b2a_retained_history_decision_scale_sweeps_to_128_generations_on_configured_rustfs() { + let _scenario = FailScenario::setup(); + let Some(uri) = helpers::s3_test_graph_uri("memwal-b2a-retained-history") else { + eprintln!( + "SKIP b2a_retained_history_decision_scale_sweeps_to_128_generations_on_configured_rustfs" + ); + return; + }; + let samples = retained_history_scaling_samples_at(&uri, &[1, 8, 32, 128]).await; + for sample in &samples { + print_retained_history_scaling_sample(sample, "rustfs-decision"); + assert_retained_history_scaling_sample(sample); + } + + // This destroys the benchmark's unique isolated graph root only after all + // retention assertions. It is fixture teardown, not production MemWAL GC. + let (store, root) = lance_io::object_store::ObjectStore::from_uri(&uri) + .await + .expect("configured S3/RustFS benchmark URI must resolve for cleanup"); + store + .remove_dir_all(root) + .await + .expect("configured S3/RustFS benchmark fixture cleanup must succeed"); +} + #[tokio::test] #[serial] async fn gate_r0_retain_all_current_object_growth_is_swept_explicitly() { @@ -1291,6 +1582,30 @@ fn gate_r0_current_inventory_path_classifier_is_canonical_and_fail_closed() { "malformed path must fail closed: {path}" ); } + + let retained_root = format!("table/_mem_wal/{shard}/abcdef12_gen_1"); + let retained_roots = BTreeSet::from([retained_root.clone()]); + assert!(path_is_under_any_root(&retained_root, &retained_roots)); + assert!(path_is_under_any_root( + &format!("{retained_root}/data/file.lance"), + &retained_roots + )); + assert!(!path_is_under_any_root( + &format!("{retained_root}-other/data/file.lance"), + &retained_roots + )); + assert!(is_lance_atomic_memwal_temp_path(&format!( + "table/_mem_wal/{shard}/manifest/{bits}.binpb.tmp.01234567-89ab-cdef-0123-456789abcdef" + ))); + assert!(!is_lance_atomic_memwal_temp_path(&format!( + "table/_mem_wal/{shard}/manifest/{bits}.binpb" + ))); + assert!(!is_lance_atomic_memwal_temp_path(&format!( + "table/_mem_wal/{shard}/wal/{bits}.arrow.tmp.01234567-89ab-cdef-0123-456789abcdef" + ))); + assert!(!is_lance_atomic_memwal_temp_path(&format!( + "table/_mem_wal/{shard}/manifest/{bits}.binpb.tmp.NOT-A-UUID" + ))); } #[test] diff --git a/docs/dev/canon.md b/docs/dev/canon.md index 587bb2f6..0179ed24 100644 --- a/docs/dev/canon.md +++ b/docs/dev/canon.md @@ -3,7 +3,7 @@ **Audience:** maintainers, contributors, and coding agents — internal **Type:** narrative reference ("the book"), read top-to-bottom **Status:** living document -**Surveyed:** OmniGraph 0.8.1 development (`main`), with RFC-026 Phase A and the private Phase B1 core present, near-cap fold closure green, and unbounded retain-all selected; Lance 9.0.0-rc.1 (git rev `cec0b7df`); internal manifest schema v8 +**Surveyed:** OmniGraph 0.8.1 development (`main`), with RFC-026 Phase A and the private Phase B1 core present, near-cap fold closure green, and the private B2a unbounded retain-all gate implemented; Lance 9.0.0-rc.1 (git rev `cec0b7df`); internal manifest schema v8 --- @@ -513,19 +513,20 @@ witness-update adapter existed. Native branch create/delete was the narrow exception: it did not advance a table HEAD, so it could proceed at `SEALED`; it still refused `OPEN` or `DRAINING`. Schema v8 preserves those exclusions and adds only private B1's exact one-generation fold as a guarded HEAD/witness -transition. RFC-026's common B2 inventory specifies the next contract: -compare-and-chain +transition. Private B2a adds no new format or transition: it implements the +unbounded retain-all storage/correctness gate over that v8 core, with no +canonical durable MemWAL deletion. RFC-026's B2-common inventory specifies the +next contract: compare-and-chain write/predecessor tokens, trusted hidden row attribution, a manifest-selected current-token participant, durable `OPEN -> DRAINING -> SEALED -> OPEN`, -bounded `REPLACE`/`WITHDRAW` correction, and unbounded retain-all with no -OmniGraph MemWAL deletion. Managed reclamation is a deferred Lance-owned -optimization, not an activation gate. None of the common contracts is -implemented. Schema +and bounded `REPLACE`/`WITHDRAW` correction. Managed reclamation is a deferred +Lance-owned optimization, not an activation gate. None of those B2-common +contracts is implemented. Schema v9/config-v3/state-v2/recovery-v12 and every public row surface stay inactive until their crash, storage, and cross-version evidence passes. Phase D owns future automatic operation-scoped drain and atomic witness advancement/rebind. Resume rechecks the bounded -profile's no-named-branch topology; an incompatible branch operation stays +narrow main-only topology; an incompatible branch operation stays `SEALED`. --- @@ -1011,13 +1012,22 @@ physical rows/vectors without external embedding or unspecified derivation. RFC-026 remains Draft: there is no production enrollment, row put/ack/fold, drain/resume, or fresh-read surface. -**Retain-all decision and the inactive public contract:** The first profile is -unbounded retain-all on stock Lance. OmniGraph performs no MemWAL GC and claims -no retained-byte/file-count bound; provider exhaustion is an accepted loud -operational risk. RC.1's missing durable cross-open materialization-attempt +**Implemented private B2a retain-all gate and the inactive public contract:** +The first profile is unbounded retain-all on stock Lance. OmniGraph performs no +MemWAL GC, never deletes a canonical durable MemWAL object, and claims no +retained-byte/file-count bound; provider exhaustion is an accepted loud +operational risk. Lance may remove only a losing manifest-CAS temporary staging +object. Complete and partial orphan output remains non-authoritative and is not +descended into, read, mutated, adopted, or deleted through retry/reopen, though +parent shard discovery may observe its prefix. The local/configured-RustFS +provider matrix pins typed failure and recovery, while the 1/8/32/128 instrument +separates acknowledgement, replay, fold, visibility, table, graph-manifest, +adapter, advisory object, and RSS terms. Warm-ack operation shape stays flat +while serialized authority and combined retained-history work grow; LIST bytes, +wall times, and RSS are diagnostics rather than quotas or SLOs. RC.1's missing durable cross-open materialization-attempt receipt and complete physical-output envelope remain documented limits on any future bounded-storage claim, not blockers for this profile. No schema or -product state was activated by the decision. +product surface was activated by B2a. The common B2 inventory defines explicit enrollment, compare-and-chain tokens, trusted visible-winner/current-withdrawal @@ -1040,8 +1050,8 @@ physical access shape)**; checkpoint-pinned retention **(RFC-025, research-blocked after Gate 0 rejected the current compacted registry-access shape)**; public MemWAL row admission and later lifecycle/read phases **(RFC-026, draft; -private one-generation Phase B1 implemented and closure-green; unbounded -retain-all selected on stock RC.1; managed reclamation is optional later work; +private one-generation Phase B1 implemented and closure-green; private B2a +unbounded retain-all gate implemented on stock RC.1; managed reclamation is optional later work; public strict activation still requires the common token/attribution, correction, lifecycle, cross-version, and product-parity evidence)**; lineage-based merge deltas @@ -1061,7 +1071,7 @@ resource budgets. | **R4: Manifest authority access grows with commit count.** Current-state resolution folds history; a selective index does not by itself bound the complete physical read. | Medium | `optimize` compacts internal tables (keeps periodically-optimized shipped paths flat where separately cost-gated). RFC-024 Gate A rejected durable heads because representative RustFS latest-manifest reads/bytes grow despite flat exact-BTREE row/range work. RFC-025 Gate 0 independently rejected checkpoint-registry activation: at local 10→1,000 on RC.1, uncompacted reconciled work and the eight-fragment tail stay flat, but compacted list/cleanup scan bytes grow 17,012→38,000 cold and 12,336→15,064 warm; exact-show bytes and operation counts also grow. Both RFCs are research-blocked; v8 retains the journal fold and internal-table *cleanup* remains deferred behind the resurrection watermark. | | **R5: Schema identity corruption or alias/identity drift.** Internal schema v5 introduced stable IDs/incarnation as durable authority; v6, v7, and v8 preserve them. | Medium | Open/init validate the SchemaIR domain and exact bidirectional IR↔manifest identity/path/alias contract; every active recovery envelope carries the identity pair; zero, duplicate, missing, or mismatched identity fails closed. | | **R6: Merge cost at divergence** — full-width classification and history-growing manifest folds. | Medium | Coherent coordinator scans plus retained probe handles reduced the pre-slice measured depth-5/depth-80 baseline from 59/651 manifest reads to 40/410 and cap the common fast-forward route at three internal opens and three scans, but the uncompacted-history slope remains. `merge_cost.rs` keeps both facts visible; O(delta) merge is blocked on a real deletion-delta source **(RFC-027)**; fragment adoption is **(draft RFC-0001)**. | -| **R7: No public streaming row path** — production writes are still capped by the `graph_head` CAS rate; high-frequency small writes remain wasteful outside the private evidence seam. | Medium | MemWAL is the strategic substrate. Phase A makes its opaque initializer + separate shard effect recoverable for one main-only/unsharded/single-live-writer-process empty enrollment, durably records lifecycle authority, and excludes competing local effects. Private B1 implements one root-scoped, no-rollover generation with watcher-plus-post-fence acknowledgement, conservative replay, and one recovery-v11 fold. Its legal high-entropy near-cap shape now closes through logical-slice accounting plus dense `take`; the measured one-exclusive-fold RSS delta is 284,934,144 bytes (~272 MiB), with a 384-MiB CI remeasurement tripwire. The selected first profile retains every MemWAL object without an OmniGraph byte/file bound and accepts loud provider exhaustion. Managed reclamation and a whole-root history budget are optional later work. Public activation remains closed on the common token/attribution/lifecycle/correction, private-format, cross-version, and product-parity contracts—not on a storage receipt. A public exact enrollment receipt plus reversible admission seal gates broader overlapping-process topology. | +| **R7: No public streaming row path** — production writes are still capped by the `graph_head` CAS rate; high-frequency small writes remain wasteful outside the private evidence seam. | Medium | MemWAL is the strategic substrate. Phase A makes its opaque initializer + separate shard effect recoverable for one main-only/unsharded/single-live-writer-process empty enrollment, durably records lifecycle authority, and excludes competing local effects. Private B1 implements one root-scoped, no-rollover generation with watcher-plus-post-fence acknowledgement, conservative replay, and one recovery-v11 fold. Its legal high-entropy near-cap shape now closes through logical-slice accounting plus dense `take`; the measured one-exclusive-fold RSS delta is 284,934,144 bytes (~272 MiB), with a 384-MiB CI remeasurement tripwire. Private B2a implements the selected unbounded retain-all storage/correctness gate: no canonical durable MemWAL deletion, inert complete/partial orphan roots, typed provider failures, and advisory 1/8/32/128 local/RustFS evidence. It accepts loud provider exhaustion and growing combined retained-history cost without claiming a quota or SLO. Managed reclamation and a whole-root history budget are optional later work. Public activation remains closed on the common token/attribution/lifecycle/correction, private-format, cross-version, and product-parity contracts—not on a storage receipt. A public exact enrollment receipt plus reversible admission seal gates broader overlapping-process topology. | | **R8: Some operations lack enforced memory/time budgets.** | Medium | Known gap, narrowed and accepted for RFC-023. Its direct-substrate instrument rejected the first whole-delta fenced adopt (~447 MB peak at 100K × 256 versus ~74 MB Append), and the first corrected production 10K series failed at 30.0× / 108,625,920 bytes overhead; both negative results remain evidence. Mutation/Load now refuses a keyed table above 8,192 rows / 32 MiB before arm, while BranchMerge uses a recovery-enrolled chain with the same per-chunk bounds and a 1,024-transaction ceiling. The inductive certificate route removes the general diff, temporary delta, target preflight, and target join without weakening that chain. Final five-pair production medians passed at 31/8 ms (3.875×) for 10K and 136/35 ms (~3.886×) for 100K; maximum signed paired RSS overheads were 24,297,472 and 32,604,160 bytes. Inclusive row/transaction ceilings, byte refusal (including materialized blobs), operation-wide validation retention, exact source/target incarnation revalidation, second-generation certificate composition, and both between-chunk recovery directions are pinned; other operations still need explicit bounds. | | **R9: Local-FS conditional-write emulation** (`write_text_if_match` check-then-act gap). | Low | All current callers sit behind the cluster lock protocol; S3 uses true conditional puts; close before admitting any lock-free caller. | | **R10: Doc/spec drift as the system grows** — this document included. | Low | Maintenance contract (same-PR doc updates, `check-agents-md.sh` link CI, "don't lie" stale markers); this canon defers to area docs by construction. | @@ -1128,7 +1138,7 @@ The plan of record is the RFC-022…028 family (all under | [0023 — Key-conflict fencing](../rfcs/0023-key-conflict-fencing.md) | Substrate-native keyed-write fencing via Lance's unenforced-PK filter; fleet/format activation barrier | **Implemented** (2026-07-15) | | [0024 — Durable table heads](../rfcs/0024-durable-table-heads.md) | Materialized head-row research; the first exact-BTREE candidate bounded scan work but failed the full latest-manifest/object-byte cost gate | **Research blocked** | | [0025 — Checkpoint-pinned retention](../rfcs/0025-checkpoint-retention.md) | Named checkpoints as authoritative retention roots, materialized as Lance tags; current in-manifest registry lookup rejected by Gate 0 | **Research-blocked** | -| [0026 — MemWAL streaming ingest](../rfcs/0026-memwal-streaming-ingest.md) | Durability-first streaming writes: one-generation private watcher-plus-post-fence acknowledgement, then graph-atomic fold; legal near-cap closure is green; the selected first profile retains every MemWAL object with no OmniGraph byte/file limit; common token/attribution/lifecycle/correction contracts remain inactive | **Draft; Phase A/B1 private core implemented; unbounded retain-all selected; public inactive** | +| [0026 — MemWAL streaming ingest](../rfcs/0026-memwal-streaming-ingest.md) | Durability-first streaming writes: one-generation private watcher-plus-post-fence acknowledgement, then graph-atomic fold; legal near-cap closure is green; private B2a retains every canonical durable MemWAL object with no OmniGraph byte/file limit and pins inert orphan/provider/history behavior; common token/attribution/lifecycle/correction contracts remain inactive | **Draft; Phase A/B1 private core and B2a retain-all gate implemented; public inactive** | | [0027 — Lineage merge deltas](../rfcs/0027-lineage-merge-deltas.md) | O(delta) merge classification from row-version lineage | Research-blocked | Deliberately split, not one mega-format: identity, key fencing, head rows, diff --git a/docs/dev/ci.md b/docs/dev/ci.md index 67de4605..aa90cbea 100644 --- a/docs/dev/ci.md +++ b/docs/dev/ci.md @@ -2,13 +2,13 @@ `.github/workflows/`: -- **ci.yml**: text-only changes skip; otherwise `cargo test --workspace --locked` on ubuntu-latest with protobuf compiler, the engine/cluster failpoint suites, and RFC-026 Gate R0's local retention/closure cells. The Gate-R0 step requires a nonzero pass count, the measured local evidence line, every load-bearing classifier/authority/source/retry/sweep/widest cell by exact name and polarity, and the full-fold RSS remeasurement cell with its 384-MiB tripwire. A rename, partial filter, inverted test, or empty match therefore fails. The tripwire detects implementation-shape drift; it is not a runtime allocator limit. OpenAPI-drift check that auto-commits the regenerated `openapi.json` for same-repository PRs. Also runs the AGENTS.md cross-link integrity check (`scripts/check-agents-md.sh`). +- **ci.yml**: text-only changes skip; otherwise `cargo test --workspace --locked` on ubuntu-latest with protobuf compiler, the engine/cluster failpoint suites, the full RFC-026 private `memwal_stream` target, Gate R0's local retention/closure cells, and B2a's small 1/8 retained-history regression. The Gate-R0 step requires a nonzero pass count, the measured local evidence line, every load-bearing classifier/authority/source/retry/sweep/widest cell by exact name and polarity, and the full-fold RSS remeasurement cell with its 384-MiB tripwire. B2a likewise requires exactly one passing small-depth cell, exactly two ignored decision-scale cells, the exact test name, and its evidence prefix. A rename, partial filter, inverted test, or empty match therefore fails. The tripwire detects implementation-shape drift; it is not a runtime allocator limit; B2a LIST/timing/RSS output is advisory rather than an SLO or quota. OpenAPI-drift check that auto-commits the regenerated `openapi.json` for same-repository PRs. Also runs the AGENTS.md cross-link integrity check (`scripts/check-agents-md.sh`). - **`Test Workspace` does not run on pull requests.** The job is gated `if: github.event_name != 'pull_request'`, so the full workspace + failpoints suite runs only on push to `main` (post-merge), on `v*` tags, and on manual `workflow_dispatch`. This was a deliberate PR-latency trade-off — it was the slowest gate (~15min warm, up to the 75min cold ceiling). `RustFS S3 Integration` `needs: test`, so it is push-/dispatch-only for the same reason. The fast PR gates remain: `Classify Changes`, `Check AGENTS.md Links`, and `Test omnigraph-server --features aws`. `Test Workspace` is correspondingly **not** in the required-check list (`.github/branch-protection.json`); see [branch-protection.md](branch-protection.md). - **Consequences to internalize:** (1) a regression that the suite would catch now lands on `main` and turns the post-merge run red, rather than being blocked pre-merge — `main` can briefly break, so run `cargo test --workspace --locked` locally before merging anything non-trivial, or trigger this workflow on your branch via the Actions "Run workflow" button. (2) `openapi.json` is no longer auto-regenerated on PRs (that step is inside the `test` job); for server/API changes, regenerate it locally with `OMNIGRAPH_UPDATE_OPENAPI=1 cargo test -p omnigraph-server --test openapi` and commit it, or the strict drift check fails the post-merge `main` run. - **Applying this policy:** removing `Test Workspace` from the JSON is inert until an admin runs `./scripts/apply-branch-protection.sh`. **Run it immediately after this change merges** — until then GitHub still requires a `Test Workspace` context that no longer reports on PRs, which leaves every open PR permanently pending (the job-never-reports trap). - **AWS feature build job**: `cargo build/test -p omnigraph-server --features aws` on ubuntu-latest. - **Windows binary build job**: `cargo build --release --locked -p omnigraph-cli -p omnigraph-server` on windows-latest with smoke checks for `omnigraph.exe version`, `omnigraph-server.exe --help`, and PowerShell installer syntax. -- **RustFS S3 integration**: spins up RustFS in Docker and runs the bucket-gated S3 suites against it. **Sharded across parallel runners** (`strategy.matrix.shard`: `engine` = `s3_storage` plus the exact `lance_surface_guards::public_physical_ref_token_rejects_s3_same_version_aba` guard, `server` = server `s3`, `cluster` = `s3_cluster`, `cli` = `local_cli_s3_end_to_end_init_load_read_flow`, `failpoints` = `failpoints s3_` plus RFC-026 Gate R0's exact configured-RustFS widest-generation retain-all closure cell), one suite per shard with `fail-fast: false` and a per-shard `rust-cache` key. The physical-ref and Gate-R0 steps check that their exact filters matched so a renamed test cannot pass vacuously; Gate R0 additionally requires the measured `rustfs` evidence line, so its explicit unconfigured skip fails CI. This job carries **correctness and load-bearing decision** suites only; RFC-013's `write_cost_s3` and RFC-024's `durable_head_lookup_cost` S3 matrices remain on-demand **cost** instruments. Gate R0's configured-object-store cell preserves closure and monotonic-retention evidence, not a finite storage claim. The tests run in seconds; the wall-clock is the per-shard `cargo test` **compile** of the engine tree, so on a cold cache (any `Cargo.lock` change) six serial steps summed past the old 75-min timeout — sharding makes wall-clock the slowest single shard (~40m cold, ~25m warm). `needs: test`, so like `Test Workspace` it is push-/dispatch-only. Not a required check. +- **RustFS S3 integration**: spins up RustFS in Docker and runs the bucket-gated S3 suites against it. **Sharded across parallel runners** (`strategy.matrix.shard`: `engine` = `s3_storage` plus the exact `lance_surface_guards::public_physical_ref_token_rejects_s3_same_version_aba` guard, `server` = server `s3`, `cluster` = `s3_cluster`, `cli` = `local_cli_s3_end_to_end_init_load_read_flow`, `failpoints` = `failpoints s3_` plus RFC-026 B2a's exact provider shard-manifest-failure/orphan-retention cell and Gate R0's exact configured-RustFS widest-generation closure cell), one suite per shard with `fail-fast: false` and a per-shard `rust-cache` key. The physical-ref, B2a, and Gate-R0 steps check that their exact filters matched so a renamed test cannot pass vacuously; the B2a and Gate-R0 commands reject their explicit unconfigured skips, and Gate R0 additionally requires the measured `rustfs` evidence line. This job carries **correctness and load-bearing decision** suites only; RFC-013's `write_cost_s3`, RFC-024's `durable_head_lookup_cost`, and B2a's ignored configured-RustFS 1/8/32/128 cost sweep remain on-demand **cost** instruments. Gate R0 preserves closure/monotonic-retention evidence, while B2a pins typed provider failure plus non-authoritative retained orphan behavior; neither creates a finite storage claim. The tests run in seconds; the wall-clock is the per-shard `cargo test` **compile** of the engine tree, so on a cold cache (any `Cargo.lock` change) six serial steps summed past the old 75-min timeout — sharding makes wall-clock the slowest single shard (~40m cold, ~25m warm). `needs: test`, so like `Test Workspace` it is push-/dispatch-only. Not a required check. - **release-edge.yml**: on every push to main, retags `edge`, builds Linux x86_64 / Linux arm64 / macOS arm64 archives and Windows x86_64 zip + sha256, publishes a rolling prerelease, then smoke-tests the Windows PowerShell installer against `edge`. - **release.yml**: on `v*` tags, builds the Linux x86_64 / Linux arm64 / macOS arm64 archives and Windows x86_64 zip release matrix, updates the Homebrew tap (`scripts/update-homebrew-formula.sh`) by pushing the regenerated formula to `ModernRelay/homebrew-tap`, and smoke-tests the Windows PowerShell installer against the tag. - **package.yml**: manual ECR image build; emits two image tags per commit (``, `-aws`) via CodeBuild. diff --git a/docs/dev/index.md b/docs/dev/index.md index 6a995a3a..a0d925b4 100644 --- a/docs/dev/index.md +++ b/docs/dev/index.md @@ -91,7 +91,7 @@ as durable disposition history after closure, so RFC backlinks stay valid. | Area | Read | |---|---| -| RFC-022–028 split architecture review — RFC-022/023/028 implemented; RFC-024/025/027 research-blocked; RFC-026 Phase A/B1 private core implemented with widest-shape closure green, unbounded retain-all selected, and common public/B2b contracts inactive; dependency corrections and acceptance evidence remain tracked | [rfc-022-027-architecture-review.md](rfc-022-027-architecture-review.md) | +| RFC-022–028 split architecture review — RFC-022/023/028 implemented; RFC-024/025/027 research-blocked; RFC-026 Phase A/B1 private core and private B2a unbounded retain-all gate implemented with widest-shape closure green, while common public/B2b contracts remain inactive; dependency corrections and acceptance evidence remain tracked | [rfc-022-027-architecture-review.md](rfc-022-027-architecture-review.md) | ## Active Implementation Plans diff --git a/docs/dev/invariants.md b/docs/dev/invariants.md index 98067c5e..4d26fd8e 100644 --- a/docs/dev/invariants.md +++ b/docs/dev/invariants.md @@ -160,8 +160,8 @@ converge the physical state. swallowed. Metrics and capabilities must be exposed before planning or operator behavior depends on them. These bounds apply to row and memory admission, deadlines, retries, and ambiguous outcomes; they do not require - an OmniGraph storage quota. RFC-026's selected unbounded retain-all profile - has no byte, object, file, or history ceiling. Exhaustion at the storage + an OmniGraph storage quota. RFC-026's implemented private B2a unbounded + retain-all profile has no byte, object, file, or history ceiling. Exhaustion at the storage provider is surfaced and may halt progress without weakening durability, recovery, or manifest-only visibility. @@ -192,7 +192,7 @@ converge the physical state. | Constructive mutations | In-memory `MutationStaging`, one end-of-query table commit per touched table, then one manifest publish | [writes.md](writes.md), [execution.md](execution.md) | | Keyed writes | Every current v8 node/edge table declares exact non-null physical `id` as Lance's unenforced PK from creation, using the v6-introduced, v7/v8-preserved exact-`id` fence. General production strict insert and upsert use the sealed exact-`id`, forced-v2 MergeInsert adapter; strict insert exact-probes its pinned parent before minting `omnigraph.insert_absence=v1`, while an all-new upsert may mint it only when completed effect statistics prove one attempt inserted every source row with zero updates, deletes, or skipped duplicates. Certificate admission is optional: BranchMerge uses the shortcut only when every transaction in the complete contiguous source interval carries v1 and its persisted operation is the full pure-insert `Update` shape (exact parent, no removed or updated fragments, nonempty new fragments with `physical_rows`, no field or generation rewrites, `RewriteRows`, exact-`id` filter, full nested schema preorder, and matching physical-row total). It then rechecks both source and target native ref incarnations and passes owned batches through an opaque capability whose one production mint site is structurally guarded. The proven publisher stages immutable fragments with `InsertBuilder`, replaces the uncommitted Append operation with that filter-bearing `Update`, and performs zero target preflights, target merge joins, or committed Appends. Missing, cleaned, unknown, or malformed proof uses the general ordered diff. Mutation/Load remains one keyed transaction per table and rejects more than 8,192 rows or 32 MiB before arm; BranchMerge keeps its bounded v4 chain and exact-recovery limits. Raw Lance graph writers are unsupported, and the certificate is an internal non-cryptographic capability rather than an authenticity mechanism. The final production cost gate passed at 10K (3.875× median; 24,297,472-byte max paired RSS overhead) and 100K (~3.886×; 32,604,160 bytes) | [RFC-023](../rfcs/0023-key-conflict-fencing.md), [writes.md](writes.md), [execution.md](execution.md) | | Deletes | Staged like inserts/updates (`stage_delete` via Lance 7.0 `DeleteBuilder::execute_uncommitted`, MR-A) — no inline HEAD advance; mixed insert/update/delete in one query rejected by D2 as a deliberate boundary (constructive XOR destructive per query; compose via separate mutations or a branch) | [query-language.md](../user/queries/index.md), [writes.md](writes.md) | -| Streaming ingest | RFC-026 adopts Lance MemWAL as the strategic substrate and remains Draft; public row streaming is inactive. Historical Phase A schema v7 implemented exact bounded enrollment recovery, identity-keyed lifecycle authority, process-local admission/exclusion, current-HEAD witnesses, compatible-open validation, partial-format refusal, and strict v6↔v7 rebuild for the main-only, one-unsharded-shard, one-live-writer-process profile. Current schema v8/config-v2/recovery-v11 adds the private B1 core behind one feature-gated, doc-hidden engine seam: one normalized physical batch enters one hard-capped 8,192-row/32-MiB generation; watcher success plus the same writer's post-durability `check_fenced()` is required for a clean acknowledgement; replayed or flushed-unmerged residue is fold-only; and one exact strict fold publishes table pointer, lifecycle witness, and lineage only at the `__manifest` CAS. Fold scanning now charges each logical slice against the generation cap and copies every scanner emission into dense owned arrays before retaining it. The deterministic 8,192-row high-entropy near-cap cell therefore closes and publishes; its isolated fold RSS delta measured 284,934,144 bytes (about 272 MiB), below the 384-MiB remeasurement tripwire. The selected unbounded retain-all profile imposes no OmniGraph retained-byte, object-count, file-count, or history quota and performs no raw `_mem_wal` deletion. Provider exhaustion is loud and may block admission, fold, or recovery; row/memory/deadline/retry/ambiguity bounds remain. RC.1's absent cross-open materialization-attempt receipt and complete physical-output envelope are no longer activation blockers for this profile, but remain inputs to the future B2b managed-reclamation route. Compare-and-chain tokens, attribution, lifecycle receipts, correction, authorization, and product parity remain unimplemented. There is no production first-use caller, schema/SDK/HTTP/CLI/OpenAPI surface, operator drain/resume, correction, GC, fresh read, or schema v9 | [RFC-026](../rfcs/0026-memwal-streaming-ingest.md), [writes.md](writes.md) | +| Streaming ingest | RFC-026 adopts Lance MemWAL as the strategic substrate and remains Draft; public row streaming is inactive. Historical Phase A schema v7 implemented exact bounded enrollment recovery, identity-keyed lifecycle authority, process-local admission/exclusion, current-HEAD witnesses, compatible-open validation, partial-format refusal, and strict v6↔v7 rebuild for the main-only, one-unsharded-shard, one-live-writer-process profile. Current schema v8/config-v2/recovery-v11 adds the private B1 core behind one feature-gated, doc-hidden engine seam: one normalized physical batch enters one hard-capped 8,192-row/32-MiB generation; watcher success plus the same writer's post-durability `check_fenced()` is required for a clean acknowledgement; replayed or flushed-unmerged residue is fold-only; and one exact strict fold publishes table pointer, lifecycle witness, and lineage only at the `__manifest` CAS. Fold scanning charges each logical slice against the generation cap and copies every scanner emission into dense owned arrays before retaining it. The deterministic 8,192-row high-entropy near-cap cell therefore closes and publishes; its isolated fold RSS delta measured 284,934,144 bytes (about 272 MiB), below the 384-MiB remeasurement tripwire. The private B2a gate now implements unbounded retain-all: no retained-byte, object-count, file-count, or history quota; no deletion of a canonical durable `_mem_wal` object; typed provider failures; and complete or partial unreferenced generation residue kept non-authoritative and untouched below its root through retry and reopen. Parent shard discovery may observe the orphan prefix, and Lance may remove only a losing manifest-CAS `.binpb.tmp.` staging object; neither is adoption or retained-history reclamation. The 1/8/32/128 local/RustFS instrument separates warm acknowledgement, cold replay, fold, visibility, and retained-history terms; its LIST totals, wall times, and RSS are advisory, not quotas or SLOs. Provider exhaustion is loud and may block admission, fold, or recovery; row/memory/deadline/retry/ambiguity bounds remain. RC.1's absent cross-open materialization-attempt receipt and complete physical-output envelope are not activation blockers for this profile, but remain inputs to the future B2b managed-reclamation route. Compare-and-chain tokens, attribution, lifecycle receipts, correction, authorization, and product parity remain unimplemented. There is no production first-use caller, schema/SDK/HTTP/CLI/OpenAPI surface, operator drain/resume, correction, GC, fresh read, or schema v9 | [RFC-026](../rfcs/0026-memwal-streaming-ingest.md), [writes.md](writes.md) | | Branch create/delete | `__manifest` `BranchContents` is the single logical authority. Lance create is physically two-phase, so OmniGraph prevalidates names, enforces path-prefix-disjoint live graph names, reclaims an absent-ref clone-only tree, and uses a bounded completion classifier; delete removes authority before tree cleanup, so an absent ref is success and derived tree reclaim may converge later. Neither control emits graph lineage. Under schema/source-target/all-table gates, each control uses one operation-local post-gate manifest/namespace capture rather than refreshing the handle-local coordinator around table-gate acquisition; successful ref movement explicitly invalidates derived read caches. Per-table forks are derived state, reclaimed best-effort with `cleanup` as backstop. A target-scoped unresolved sidecar may be made unreachable by deletion and is then audit-discarded by recovery; graph-global SchemaApply still blocks | [branches-commits.md](../user/branching/index.md), [maintenance.md](../user/operations/maintenance.md), [writes.md](writes.md) | | Cleanup retention | Explicit cleanup derives exact `keep` cutoffs from Lance's available version list, caps each main-table GC cutoff at the oldest exact main version inherited by any live lazy graph branch, and refuses uncovered main HEAD drift. Lance protects native per-table branch refs itself. The graph-wide live-reference preflight fails closed before the first table GC, after which individual table failures remain fault-isolated | [maintenance.md](../user/operations/maintenance.md), [writes.md](writes.md) | | Optimize visibility | One operation-local accepted catalog and fresh main snapshot are planned under schema → main → all-table gates. Every productive table shares one identity-bearing v9 recovery envelope; bounded-parallel physical effects become visible through at most one monotonic manifest/lineage CAS. Complete crash residuals roll forward together and partial residuals compensate before visibility. Optimize's payload remains a bounded maintenance adapter rather than an exact caller-minted Lance transaction proof, so it is supported within the single-writer-process recovery boundary. Exact provenance is deferred until Lance exposes a stable public caller-controlled maintenance transaction API and OmniGraph has distributed recovery fencing | [maintenance.md](../user/operations/maintenance.md), [writes.md](writes.md), [RFC-022](../rfcs/0022-unified-write-path.md) | @@ -228,7 +228,8 @@ incomplete would let a legacy writer prepare from unresolved physical state. Do not hide these behind invariant wording. Either move them forward or keep them explicit. -- **Private B1 closure is repaired; public row streaming is not active:** RC.1's public +- **Private B1 closure and B2a retain-all gate are implemented; public row + streaming is not active:** RC.1's public initializer commits the singleton MemWAL index internally and shard claim is a separate durable effect. RFC-026 no longer treats an upstream release as the only calendar path: a production-neutral harness proved exact @@ -277,18 +278,26 @@ them explicit. the 384-MiB remeasurement tripwire; crossing that tripwire requires remeasurement rather than a silent admission change. - The selected next profile is unbounded retain-all. A flush may write a fresh + The private B2a profile implements unbounded retain-all. A flush may write a fresh randomized generation subtree before the shard-manifest CAS, and a cold replay may therefore leave additional materialization roots. OmniGraph deliberately imposes no retained-byte, object-count, file-count, or history quota, adds no cross-open attempt ledger for quota enforcement, and never - deletes raw `_mem_wal` paths. The strict current-object census remains useful + deletes a canonical durable `_mem_wal` object. Lance may remove only a losing + shard-manifest-CAS `.binpb.tmp.` staging object, which never became + shard authority or retained history. Complete and partial unreferenced + generation residue stays non-authoritative: parent shard discovery may list + its prefix, but OmniGraph never descends into, reads, mutates, adopts, or + deletes the subtree during retry or reopen. The strict current-object census remains useful as retention and path-shape evidence, not as a claim about multipart uploads, provider versions, billed bytes, or a finite storage envelope. Provider exhaustion is a loud storage failure and may halt admission, folding, or recovery until capacity is restored; it never permits an acknowledgement to disappear or a partial effect to become graph-visible. Row and Arrow-memory - admission, deadlines, retries, and ambiguity handling remain bounded. + admission, deadlines, retries, and ambiguity handling remain bounded. The + local/configured-RustFS 1/8/32/128 instrument keeps these cost terms separate + and is advisory; it does not create a quota, latency SLO, or history-flat + claim. RC.1's missing durable materialization-attempt receipt and reserve-first complete-output envelope are consequently not blockers for unbounded diff --git a/docs/dev/lance.md b/docs/dev/lance.md index 8d42ec1d..432222bc 100644 --- a/docs/dev/lance.md +++ b/docs/dev/lance.md @@ -171,7 +171,7 @@ for schemas that avoid the newly reserved names documented below. RC.1 requires Rust 1.91 or newer; OmniGraph continues to track stable Rust and this audit ran on Rust 1.95. -#### Post-audit implementation notes: 2026-07-18–20 +#### Post-audit implementation notes: 2026-07-18–21 RFC-026 Gate E0 subsequently passed and Phase A activated OmniGraph internal schema v7. V7 adds only the bounded streaming foundation: exact recovery-v10 @@ -226,8 +226,11 @@ requires empty frozen refs and the exact generation/cursor in the latest shard manifest. And RC.1 writes the randomized generation dataset and sidecars before that manifest CAS, so a crash may leave a complete or partial unreferenced `{hash}_gen_N/` subtree. B1 treats only that recognized subtree shape as -retained derived orphan output—never adopted, scanned, or deleted—and leaves -it in place under the selected unbounded retain-all profile. A later B2b +retained derived orphan output. Private B2a proves complete and partial +unreferenced output remains non-authoritative and is never descended into, +read, mutated, adopted, or deleted through retry/reopen. Parent shard discovery +may still observe the common prefix. The subtree remains in place under the +selected unbounded retain-all profile. A later B2b managed profile would need a separately proved Lance-owned GC contract. Seal/drain/abort are background-owned with deadline-bounded caller waits: an error or stalled handler keeps the @@ -250,10 +253,17 @@ drain's captured floor. Replay preserves possible residue but cannot resolve which caller attempt produced it. Configuration identity binds only correctness/topology/no-rollover fields; explicit runtime policy and injected Session/store capabilities remain separate. Private B1 activates graph schema -v8 and stream-config v2 rather than adopting v7/config-v1 in place. The selected -next profile is B2a **unbounded retain-all**: OmniGraph deletes no raw -`_mem_wal` path, imposes no retained-byte/object/file/history quota, and accepts -loud provider exhaustion. Explicit enrollment, trusted attribution, +v8 and stream-config v2 rather than adopting v7/config-v1 in place. The private +B2a **unbounded retain-all** gate is implemented: OmniGraph deletes no canonical +durable `_mem_wal` object, imposes no retained-byte/object/file/history quota, +and accepts loud provider exhaustion. Lance may remove only its losing shard- +manifest-CAS `.binpb.tmp.` staging object, which never became authority. +The provider matrix pins typed local/configured-RustFS failure and inert orphan +behavior. The 1/8/32/128 local/RustFS instrument keeps warm acknowledgement, +cold replay, fold, visibility, table, graph-manifest, adapter, advisory object, +and RSS terms separate. Warm-ack operation shape is flat while serialized +authority and combined retained-history work grow; timings, LIST totals, and +RSS are diagnostics, not quotas or SLOs. Explicit enrollment, trusted attribution, compare-and-chain tokens, manifest-selected current-token authority, bounded correction, revisioned lifecycle/management receipts, authorization, and product parity remain common contracts. A graph-history budget is not one of @@ -378,6 +388,16 @@ profile deliberately does not promise. They become relevant again if a later RFC proposes bounded retention. No schema v9 or public surface is authorized by the source audit or closure repair alone. +Private B2a closes the narrower stock-RC.1 storage/correctness gate. A shared +fail-closed classifier validates canonical current MemWAL paths and decoded +authority. Injected local and configured-RustFS failures prove effect-free claim +errors, post-invocation acknowledgement ambiguity, complete post-cut orphan +output, and partial post-data orphan output remain typed and recoverable without +subtree adoption or deletion. The 1/8/32/128 local/RustFS sweep shows the warm- +ack operation shape is flat but serialized authority bytes and combined +retained-history work grow. Those observations are intentionally advisory; +they neither revise the unbounded contract nor claim an isolated MemWAL slope. + The no-roll profile uses fixed portable capacities (`8,193` rows/batches and 1-GiB byte/unflushed thresholds), not architecture-dependent `usize::MAX` metadata. After reopen, the 32-MiB contract is validated from public diff --git a/docs/dev/rfc-022-027-architecture-review.md b/docs/dev/rfc-022-027-architecture-review.md index 4af96cb7..23096653 100644 --- a/docs/dev/rfc-022-027-architecture-review.md +++ b/docs/dev/rfc-022-027-architecture-review.md @@ -2,8 +2,8 @@ **Status:** RFC-022, RFC-023, and RFC-028 implemented; RFC-024, RFC-025, and RFC-027 research-blocked; RFC-026 Phase A implemented, private B1's legal -near-cap closure repaired and green, unbounded retain-all selected for B2a, -B2b/common contracts inactive, and no public stream surface exists +near-cap closure repaired and green, private B2a unbounded retain-all gate +implemented, B2b/common contracts inactive, and no public stream surface exists **Date:** 2026-07-11 **Last updated:** 2026-07-21 **Audience:** RFC authors, engine/storage maintainers, and release reviewers @@ -47,10 +47,16 @@ bounded retain-all claim on stock RC.1. Random generation roots have no MemWAL-persisted/enforced durable cross-open materialization-attempt cap/receipt; RC.1 exposes no admission-grade reserve-first complete physical-output envelope; and current LIST cannot prove all provider-retained -or billed bytes. The selected B2a profile deliberately makes none of those -finite-storage claims: retain every `_mem_wal` object, perform no raw WAL GC, -impose no retained-byte, object-count, file-count, or history quota, and surface -provider exhaustion loudly. The missing materialization-attempt receipt and +or billed bytes. The implemented private B2a gate deliberately makes none of +those finite-storage claims: retain every canonical durable `_mem_wal` object, +perform no raw WAL GC, impose no retained-byte, object-count, file-count, or +history quota, and surface provider exhaustion loudly. Lance may remove only a +losing manifest-CAS temporary staging object. Complete and partial orphan output +remains non-authoritative and untouched below its root through retry/reopen; +parent shard discovery may observe its prefix. Local/configured-RustFS provider +cells and a 1/8/32/128 retained-history instrument pin that boundary. Warm-ack +operation shape stays flat while serialized authority and combined history work +grow; LIST bytes, timings, and RSS remain advisory. The missing materialization-attempt receipt and complete envelope therefore do not block B2a. No test-only attempt ledger or `GraphHistoryBudget` is on its immediate path. B2b remains optional future Lance-owned managed reclamation. Common explicit enrollment, @@ -485,8 +491,12 @@ and [§11–13](../rfcs/0026-memwal-streaming-ingest.md#11-format-activation-and **Status:** Format separation is specified. Gate R0's legal near-cap closure failure was repaired on 2026-07-21 with logical-slice accounting and dense scanner-output copies; the closure and RSS cells are green. The selected B2a -profile is unbounded retain-all with no raw `_mem_wal` GC and no retained-byte, -object-count, file-count, or history quota. Gate R0's historical no-go applies +profile's private gate is implemented as unbounded retain-all with no canonical +durable `_mem_wal` deletion and no retained-byte, object-count, file-count, or +history quota. Structural guards, typed local/configured-RustFS provider +failures, complete/partial inert orphan evidence, and the 1/8/32/128 advisory +history instrument pin that boundary. Lance losing manifest-CAS temp staging is +not retained authority. Gate R0's historical no-go applies only to claiming a finite storage bound on stock RC.1. Public activation remains closed on the common contracts below, not on the missing materialization-attempt receipt or physical-output envelope. RC.1 also lacks the ideal caller-owned @@ -523,7 +533,9 @@ empty-enrollment contract: implemented v8 is the first data-bearing stream format, uses stream-config v2 and recovery schema v11, refuses v7/config-v1, and carries genuine v7↔v8 rebuild/refusal evidence. -B2 is a third strict strand, not an in-place reinterpretation of v8. The common +B2 is a third strict strand, not an in-place reinterpretation of v8. Private +B2a adds no schema change; it closes the stock-v8 storage/correctness gate for +the selected retain-all profile. The common B2 inventory assigns internal schema v9, stream-config v3, stream-state protocol v2, and recovery-v12 to trusted hidden row metadata, the manifest-selected current-token participant, persistent revisioned @@ -961,7 +973,8 @@ managed-reclamation route; implementation-blocking only if that profile is pursued. Gate R0's 2026-07-20 result remains a historical no-go for a finite or otherwise bounded retain-all claim on stock RC.1. It does not reject the selected unbounded B2a profile, whose legal near-cap B1 closure was repaired on -2026-07-21. Neither result reopens the MemWAL substrate choice or activates a +2026-07-21 and whose private no-delete/provider-failure/history-cost gate is now +implemented. Neither result reopens the MemWAL substrate choice or activates a public surface. Two production-neutral guards pin the current RC.1 boundary: @@ -993,15 +1006,19 @@ output. The retained one/four/eight-fold current-object census and referenced-cut retry prove useful invariants but cannot establish provider versions, incomplete multipart, or billed bytes. -The selected B2a profile accepts that limitation explicitly. It retains all raw -`_mem_wal` state, performs no OmniGraph WAL deletion, sets no retained-byte, +The implemented private B2a profile accepts that limitation explicitly. It +retains every canonical durable `_mem_wal` object, performs no OmniGraph WAL deletion, sets no retained-byte, object-count, file-count, or history quota, and treats provider exhaustion as a loud failure that may stop admission, fold, or recovery progress without weakening acknowledgement durability or manifest-only visibility. Because it makes no finite-storage claim, B2a does not need a test-only materialization attempt ledger, a complete reserve-first physical-output envelope, or a `GraphHistoryBudget`. Any later production state still binds to the existing -manifest/recovery authority. +manifest/recovery authority. Complete and partial unreferenced generation output +remains non-authoritative and untouched below its root through retry/reopen; +parent discovery may observe the prefix, and Lance may remove only a losing +manifest-CAS temporary staging object. The checked-in provider matrix and +1/8/32/128 advisory instrument pin that exact boundary. **Optional future B2b disposition:** if managed reclamation is pursued, author the capability inside Lance as an opaque inspect/plan/execute operation with @@ -1223,10 +1240,12 @@ The review does not require all RFCs to land together. A safe order is: schema-v8/config-v2 put/watcher/post-fence/replay/recovery-v11 evidence and its repaired legal near-cap closure: logical-slice accounting plus dense scanner-output copies now fold and publish the 8,192-row high-entropy cell, - with the isolated fold RSS delta below the remeasurement tripwire. Proceed - with the selected unbounded B2a retain-all profile: never delete raw - `_mem_wal` state, impose no retained-byte/object/file/history quota, and - surface provider exhaustion loudly. Do not add a materialization-attempt + with the isolated fold RSS delta below the remeasurement tripwire. Preserve + the implemented private B2a unbounded retain-all gate: never delete a + canonical durable `_mem_wal` object, keep complete/partial orphan subtrees + inert through retry/reopen, impose no retained-byte/object/file/history + quota, surface provider exhaustion loudly, and retain the local/RustFS + provider plus 1/8/32/128 advisory evidence. Do not add a materialization-attempt ledger, reserve-first complete physical-output envelope, storage watermark, or `GraphHistoryBudget` to that immediate path. Implement and prove the common explicit-enrollment, compare-and-chain-token, trusted-attribution, diff --git a/docs/dev/testing.md b/docs/dev/testing.md index c2b63c15..1809764e 100644 --- a/docs/dev/testing.md +++ b/docs/dev/testing.md @@ -24,10 +24,11 @@ The engine's `tests/` is the principal coverage surface; most graph-shaped behav | `merge_fast_forward.rs` | Branch-adopt cost + correctness under RFC-023. The one-batch and 8,193-row fixtures prove that a complete v1 insertion-absence history chain publishes one/two bounded exact-`id` filtered `Update` transactions with zero target strict-insert preflights, target MergeInsert joins, committed Appends, ordered-cursor scans, or whole-delta staged combines. `pure_insert_fast_forward_retains_value_constraint_validation` proves the certificate skips only redundant key work, not logical row constraints; its all-new Upsert source is certified from completed effect statistics. `proven_fast_forward_certificate_composes_across_merge_generation` proves the publisher re-mints v1 and a second merge consumes that output as the next proof-chain link. A missing intermediate transaction proves cleaned history is an optimization miss: the merge enters the general ordered diff, preserves exact rows, and leaves no recovery residue. `lazy_target_ref_only_fast_forward_uses_pin_after_main_advances` distinguishes a valid old lazy graph pin from drift when the inherited main ref advances. A nested `main → feature → experiment` cell prevents a deeper valid `BranchIdentifier` from becoming a false read-set conflict. Every general-route base/source/target `OrderedTableCursor` scan applies both Lance `batch_size(8,192)` and `batch_size_bytes(32 MiB)`. Validation streams projected `id`/`src`/`dst`/scalar batches, charges exact Arrow memory before retention, and shares one 32 MiB operation-wide budget across candidate tables; `branch_merge_validation_delta_is_aggregate_bounded_pre_arm` crosses it with two individually valid ~18 MiB deltas while proving zero HEAD/manifest/lineage/sidecar movement. Deletes use exact escaped-filter chunks with the same row/byte and retained-plan bounds. Production-helper unit cells pin chain/delete/recovery limits. The subprocess scenario owns the final production latency/RSS evidence; these integration tests own route semantics, not timings | | `writes.rs` | Direct-publish writes: cancellation, RFC-022 non-strict full-attempt reprepare from fresh branch authority, strict stale-write conflicts, multi-statement atomicity, MR-794 staged-write rewire (D₂ rejection, insert+update coalesce, multi-append coalesce, partial-failure recovery, load RI/cardinality recovery); RFC-023 pins the inclusive 8,192-row keyed input ceiling, the same exact/+1 boundary on streamed mutation-update matches, no-effect state for both refusals, and oversized stored-Blob rejection before payload read. Crate-internal pending-scan cells pin inclusive/+1 32 MiB accounting plus pending-key shadow-before-charge. The lance#7444 row-id-overlap regression (`filtered_read_after_merge_update_and_delete_keeps_row_ids_consistent` — merge-load → same-key merge-load → delete → keyed point lookup, green only under the vendored lance-table patch — plus its append-only control) | | `src/table_store/staged_tests.rs` | Crate-internal staged primitives. RFC-023 pins one exact target preflight for general StrictInsert, durable v1 mint/commit/reopen/history persistence, exact-`id` filter emission, typed `KeyConflict`, and missing/wrong PK refusal. `all_new_upsert_certifies_insert_absence_and_persists_it_in_history` proves an all-new completed Upsert receives the optional certificate, a mixed/update Upsert does not, unrelated transaction properties survive, and UUID rebinding does not erase it. Proven-insert cells show the opaque path performs zero strict preflights; stages with `InsertBuilder` but commits the full pure-insert `Update` shape (exact parent and `id` filter, `RewriteRows`, no updates/removals, full nested schema preorder, physical rows); persists/re-admits its own output for proof composition; leaves new fragments outside old index coverage; and fails same-key races loudly in proven/proven and proven/general orders. The in-source `exec/merge.rs` certificate unit table rejects missing/unknown properties, wrong parent/filter/full-preorder/mode/offsets, rewrite/removal shapes, missing `physical_rows`, and Append. Source-interval cells pin exact selection, lazy retained-parent splitting, coalescing, and pinned Lance's approximate raw-emission boundary while every normalized/writer chunk remains hard-capped. Generic `stage_append`/`stage_merge_insert` remain primitive tests only. The file also owns index staging and `commit_staged{,_exact}` | -| `forbidden_apis.rs` | Defense-in-depth syntax-tree/source guard over the whole engine. The primary boundary is Rust visibility: raw storage/coordinator/handle-cache modules are crate-private; public `Snapshot::open` returns `SnapshotTable`; and `SnapshotScanner` executes reads without exposing Lance's raw scanner or physical plan. The guard pins those visibility/return-type boundaries, classifies public async inherent `Omnigraph` methods plus loader conveniences, classifies every crate-visible async method on `GraphCoordinator` / `ManifestCoordinator`, and exact-counts registered method/UFCS durable-call shapes including recovery. RFC-023 rejects production graph call sites of generic `stage_append{,_stream}` and `proven_insert_capability_has_one_production_mint_site` pins `ProvenInsertChunk::from_verified_history` to the complete-history classifier in `exec/merge.rs`, preventing the no-preflight capability from becoming a reusable bypass. At the RFC-026 Phase-A checkpoint the guard registered only the exact v10 enrollment gateway and feature-gated test seam, counted its sidecar/index/shard durability primitives, and kept every row-put/ack/fold surface absent; the Phase-B1 owner below now exact-counts only the approved crate-private put/fold durable-call sites while retaining the absence of public schema, SDK, HTTP, CLI, and OpenAPI side doors. It also counts selected raw `SnapshotHandle` / Dataset shapes, rejects renamed-owner/macro/include/path-lookalike forms, skips structurally test-only code, and pins retired escape hatches absent. This is intentionally not a Rust macro-expander or general alias analysis; `// forbidden-api-allow: ` exempts reviewed inline-Lance lines only | +| `forbidden_apis.rs` | Defense-in-depth syntax-tree/source guard over the whole engine. The primary boundary is Rust visibility: raw storage/coordinator/handle-cache modules are crate-private; public `Snapshot::open` returns `SnapshotTable`; and `SnapshotScanner` executes reads without exposing Lance's raw scanner or physical plan. The guard pins those visibility/return-type boundaries, classifies public async inherent `Omnigraph` methods plus loader conveniences, classifies every crate-visible async method on `GraphCoordinator` / `ManifestCoordinator`, and exact-counts registered method/UFCS durable-call shapes including recovery. RFC-023 rejects production graph call sites of generic `stage_append{,_stream}` and `proven_insert_capability_has_one_production_mint_site` pins `ProvenInsertChunk::from_verified_history` to the complete-history classifier in `exec/merge.rs`, preventing the no-preflight capability from becoming a reusable bypass. At the RFC-026 Phase-A checkpoint the guard registered only the exact v10 enrollment gateway and feature-gated test seam, counted its sidecar/index/shard durability primitives, and kept every row-put/ack/fold surface absent; the Phase-B1 owner below now exact-counts only the approved crate-private put/fold durable-call sites while retaining the absence of public schema, SDK, HTTP, CLI, and OpenAPI side doors. B2a inventories the only production `_mem_wal` literal owners, forbids MemWAL reclamation/adoption symbols and destructive primitives in the adapter, keeps generic maintenance unaware of MemWAL, and keeps raw inventory/classifier helpers private. This remains defense in depth rather than macro expansion, alias, or data-flow analysis; the visibility boundary and behavior tests are still primary. It also counts selected raw `SnapshotHandle` / Dataset shapes, rejects renamed-owner/macro/include/path-lookalike forms, skips structurally test-only code, and pins retired escape hatches absent. `// forbidden-api-allow: ` exempts reviewed inline-Lance lines only | | `lance_surface_guards.rs` | Pins the Lance API surfaces omnigraph depends on (named runtime + compile-only guards; see [lance.md](lance.md)) — the first smoke check on any Lance version bump. `cached_and_zero_cache_sessions_share_store_registry_not_metadata_cache` proves a cached data Session and zero-cache control Session reuse one live `ObjectStoreRegistry` client while their metadata caches remain isolated. `_compile_uncommitted_full_table_vector_index_shape` pins the public `IndexMetadata` shape suitable for `Operation::CreateIndex`; `compact_files_succeeds_on_blob_columns` pins blob-v2 compaction; Guard 9 pins clone-only branch reclaim semantics. RFC-023's `unenforced_pk_filter_shape_is_route_dependent` explicitly forces v2 versus indexed routes and pins the `Some(populated)` / `Some(empty)` / `None` key-filter shapes; `unenforced_pk_conflict_matrix_is_directional` pins the directional filtered/unfiltered and filtered/Append matrix. RFC-024's compile guard pins the public `BranchIdentifier` + current table version + current `Transaction.uuid` + `ManifestLocation.e_tag` current-HEAD witness; the local/shared-`Session` guard proves unchanged-reopen stability, ordinary-commit movement, and same-version ABA, while RustFS covers object-store ABA. RFC-025 adds exact main/named-branch tag-target, sparse cleanup pin/unpin, and branch-tree-deletion guards. RFC-026 pins doc-hidden `has_successor_version`, initializer/readback/shard-writer/durability/fencing, flush/drain, replay watermark, scanner, and merged-generation shapes; runtime Gate E0 classification belongs to `memwal_enrollment_gate.rs`, while v7 Phase-A and v8 B1 publication/recovery belong to the manifest/failpoint suites. B2b adds `cleanup_old_versions_does_not_reclaim_mem_wal_objects` and `mem_wal_deleted_fence_slot_allows_stale_writer_success_on_pinned_lance`: the first proves generic cleanup leaves the present MemWAL fixture unchanged and the second proves deleting the successor's empty fence sentinel is unsafe. The pinned source audit, not those two tests alone, establishes that stock RC.1 exposes no owned MemWAL reclamation API. The RC.1 compiler guard pins the five surveyed public Lance virtual system-column constants to early `.pg` rejection. These guards prove substrate shapes/tokens and negative ownership boundaries; they do not by themselves prove heads/checkpoint activation, the v8 publisher, or a safe reclamation implementation | | `memwal_enrollment_gate.rs` | RFC-026's green production-neutral Gate E0 harness, isolated from the production manifest and graph writer. Fourteen substantive local cells plus one explicit unconfigured-S3 skip cover exact no-effect / `N + 1` index / pre-minted empty-shard classification, buried-effect refusal, marker survival, strict inventory/error handling, and the broad fail-closed matrix. The rejected first instrument used `checkout_latest` plus `IOTracker`, which missed local `read_dir`. The accepted exact-version classifier pins doc-hidden `has_successor_version`; its `AttemptTracker` records failed/`NotFound` attempts before forwarding and proves the identical complete six-attempt shape at baseline versions 8/80: four successful manifest HEADs, one `NotFound` manifest HEAD, one successful manifest GET, zero lists. A Unix execute-only `_versions` tripwire proves exact probing works when latest enumeration fails and an unreadable exact HEAD errors. The configured RustFS exact cell passes non-vacuously with the same zero-list shape and owns the positive lost-result/index/empty-shard/reopen sequence plus foreign shard, malformed/loose root, durable WAL, persisted cursor, and corrupt-manifest negatives. S3 ABA remains in `lance_surface_guards.rs`; CI rejects skipped E0/ABA cells. This file never mutates production manifest/schema state or deletes ambiguous artifacts; Phase A consumes its classifier through the private adapter | -| `memwal_stream_cost.rs` | Feature-gated RFC-026 B1 and Gate-R0 decision instrument. It separately measures warm already-claimed durability acknowledgement, cold replay, selected-generation fold scanning, retained merged metadata, the uncompacted graph-manifest term, legal no-roll estimates, and paired peak RSS. Gate R0 adds a revision-pinned source-audit tripwire, strict current-object classification/reference census, listed path/class/size retain-all comparisons at one/four/eight folds, referenced-cut retry reuse, and deterministic high-entropy near-cap local/configured-RustFS cells. The near-cap cell now positively proves that an admitted 8,192-row/near-32-MiB generation acknowledges without graph visibility, then folds and publishes exactly once after logical-slice charging plus dense per-scanner-batch take. The isolated fold RSS delta measured 284,934,144 bytes (about 272 MiB), below a 384-MiB remeasurement tripwire; that tripwire is not a runtime allocator limit. Current LIST evidence explicitly excludes multipart/provider versions/local staging/billed bytes and is retention evidence, not a quota. A green test proves private closure and retain-all behavior; it does not activate a public API. | +| `memwal_stream.rs` | Feature-gated RFC-026 private B1 behavior plus B2a provider-failure evidence. B1 owns bounded put/ack/replay/fold, authority, cancellation, crash, and manifest-only visibility. B2a injects a recording/failing store at the real Lance table-store boundary and covers effect-free cold-claim failure, post-invocation WAL ambiguity, complete post-cut unreferenced generation output, and partial generation output after the data object. Exact local and configured-RustFS cells prove typed `Lance` / `AckUnknown` / `RecoveryRequired` outcomes, blocked fresh progress, one fresh authoritative retry root, retained non-authoritative residue, and zero access below the orphan root through retry and cold reopen. Parent shard discovery may observe the common prefix; it may not descend into or adopt the subtree. | +| `memwal_stream_cost.rs` | Feature-gated RFC-026 B1, Gate-R0, and B2a decision instrument. It separately measures warm already-claimed durability acknowledgement, cold replay, selected-generation fold scanning, visibility, retained merged metadata, the uncompacted graph-manifest term, legal no-roll estimates, and paired peak RSS. Gate R0 adds a revision-pinned source-audit tripwire, strict current-object classification/reference census, listed path/class/size retain-all comparisons at one/four/eight folds, referenced-cut retry reuse, and deterministic high-entropy near-cap local/configured-RustFS cells. The near-cap cell proves that an admitted 8,192-row/near-32-MiB generation acknowledges without graph visibility, then folds and publishes exactly once after logical-slice charging plus dense per-scanner-batch take. The isolated fold RSS delta measured 284,934,144 bytes (about 272 MiB), below a 384-MiB remeasurement tripwire; that tripwire is not a runtime allocator limit. B2a adds 1/8/32/128 local and configured-RustFS retained-history sweeps whose terms remain separate: warm ack, cold reopen/replay, fold, visibility, table/graph-manifest/adapter work, advisory current-object bytes, and whole-process peak RSS. Older retained roots must receive zero reads, writes, or deletes. The only allowed delete shape is Lance's losing manifest-CAS `.binpb.tmp.` staging; canonical durable MemWAL delete requests remain zero. LIST totals, wall times, and RSS are advisory—not a quota, SLO, isolated WAL slope, or provider billing. A green test proves private closure/retention behavior; it does not activate a public API. | | `durable_head_lookup_cost.rs` | RFC-024 Gate A decision instrument, isolated from the production manifest schema/publisher. At fixed catalog width 10 it runs the full absent/reconciled/one-uncovered/eight-uncovered/reconciled-after-tail matrix over compacted and uncompacted histories, with cold-open and warm-repeat measurements on local FS and bucket-gated S3/RustFS. Default depths are 20/80; the ignored decision-scale cell runs 10/100/1,000. Correct exact heads, flat indexed `rows_scanned`/range work, an index-absent growing negative control, and observable bounded tails all pass; after the eight-fragment tail, `optimize_indices` returns coverage to zero uncovered and representative `rows_scanned`/range work from 27→10 / 17→10. The test deliberately pins the no-go: uncompacted RustFS cold object reads/bytes and compacted byte terms grow, while RC.1 also crosses a bounded one-operation boundary by 1,000 commits, so RFC-024 remains research-blocked. `rows_scanned` is an RC.1 debug proxy, not a universal decoded-row counter. Object-store wrapper bytes and Lance execution-summary bytes are separate fixture-owned metrics and are not additive | | `checkpoint_retention_cost.rs` | RFC-025 Gate 0 decision instrument, isolated from the production manifest schema. It models three live checkpoints at catalog width 10 and measures complete list, exact show, and cleanup-root authority reads across absent/reconciled/eight-uncovered index states, compacted/uncompacted layouts, and cold/warm access. It also owns the reference V1 name-normalization matrix. Default local depths 20/80 pass the checked-in **no-go-preservation** assertions; the RC.1 ignored 10/100/1,000 run shows reconciled uncompacted work and the bounded tail flat, but rejects the current format shape after compaction: list/cleanup scan bytes grow 17,012→38,000 cold and 12,336→15,064 warm; show grows 29,348→53,064 and 24,672→30,128; scan operations add one at 1,000. The S3/RustFS cell is bucket-gated and was not run for this decision. The result keeps RFC-025 research-blocked; current v8 adds no checkpoint state | | `warm_read_cost.rs` | Cost-budget tests for the warm read/control path (query-latency work), measured at the object-store boundary with Lance `IOTracker` (the LanceDB IO-counted pattern): a warm same-branch read does 0 manifest opens, 1 version probe, validates the schema once (Fix 1 / finding A / Fix 2 at commit-history depth); a cold other-branch resolution derives snapshot state and lineage from one coherent manifest open/scan; native branch create and create-from each use one post-gate open/scan, while delete uses one target capture plus one native-ref opener and only one row scan; stale same-branch reads perform exactly 2 probes and refresh manifest-only; recreated non-main branches with the same Lance version refresh by incarnation; recreated branch-owned table handles are distinguished by table e_tag or refresh-time cache clearing; recreated traversal topology is protected by per-edge-table e_tag in the graph-index cache key or refresh-time cache clearing; a warm *repeat* read does 0 table opens via the held-handle cache and a write re-opens only the changed table at its new version/e_tag (Fix 3/6A). Also the CSR topology-build cost guards: `fresh_branch_traversal_reuses_main_graph_index` (A1 — a lazy-fork branch reuses main's cached CSR index, 0 rebuilds via `graph_build_count`) and `single_edge_query_builds_only_referenced_edge` (A2 — a one-edge query builds only that edge via `graph_edges_built`); both force CSR via the scoped `with_traversal_mode` seam, so they need no `#[serial]`. See "Cost-budget tests" below. | @@ -332,7 +333,44 @@ cargo test -p omnigraph-engine --features failpoints --test memwal_stream_cost cargo test -p omnigraph-engine --features failpoints --test memwal_stream_cost widest_legal_generation_records_no_roll_estimates_and_peak_rss -- --exact --nocapture ``` -B1 does not add parser/server/CLI tests because it has no public surface. The +### RFC-026 Phase B2a coverage ownership (private gate implemented) + +B2a reuses the B1 owners instead of creating a second streaming stack. The +shared `tests/helpers/memwal.rs` classifier is the sole strict test authority +for canonical WAL, shard-manifest, generation, PK, Bloom, deletion, and user- +index paths plus decoded reference agreement. `memwal_stream.rs` injects real +provider failures at the Lance table-store boundary and proves complete and +partial orphan output remains retained, non-authoritative, and untouched below +its root through blocked admission, retry, and cold reopen. A parent +`list_with_delimiter` may reveal the orphan common prefix during shard +discovery; no production path may descend into, read, mutate, adopt, or delete +the subtree. The exact configured-RustFS provider cell is non-vacuous in CI. + +`memwal_stream_cost.rs` owns the separate retained-history instrument. The +small 1/8 cell runs on every relevant CI change; ignored local and configured- +RustFS cells sweep 1/8/32/128. It records warm acknowledgement, cold replay, +fold, visibility, table-store, graph-manifest-store, adapter, advisory object, +and whole-process RSS terms independently. It asserts zero IO against every +older retained generation root and zero canonical durable MemWAL deletes. Lance +may remove a losing shard-manifest-CAS `.binpb.tmp.` staging object; the +classifier accepts only that exact shape because it never became authority. +The observed LIST bytes, wall time, and RSS are diagnostics, not quotas, +latency SLOs, provider billing, or an isolated MemWAL history slope. + +Run the relevant local/CI B2a cells with: + +```bash +cargo test -p omnigraph-engine --features failpoints --test memwal_stream provider_ +cargo test -p omnigraph-engine --features failpoints --test memwal_stream_cost b2a_ +``` + +Run the decision-scale local sweep explicitly with: + +```bash +cargo test -p omnigraph-engine --features failpoints --test memwal_stream_cost b2a_retained_history_decision_scale_sweeps_to_128_generations -- --ignored --exact --nocapture +``` + +B1/B2a do not add parser/server/CLI tests because they have no public surface. The common product contract inventory specifies the missing contracts without activating them. Any public implementation must extend the existing compiler, server/OpenAPI, CLI parity, Cedar, shutdown, @@ -454,6 +492,7 @@ sidecar arm or graph movement. - Call sites are inserted at sensitive transaction boundaries (branch create, graph publish commit, the recovery sweep's classify→roll-forward-publish window, cluster apply's payload→state-write window, etc.). - **Serialize and rendezvous, never sleep.** The `fail` registry is process-global, so every failpoint test carries `#[serial]` (`serial_test`). For concurrent tests, use `helpers::failpoint::Rendezvous` (`tests/helpers/failpoint.rs`): `park_first(name)` parks the first thread to hit the point until `release()`, and `wait_until_reached().await` blocks on that condition (it doubles as a fired-assertion). Do not coordinate threads with fixed `sleep`s. - Activated tests: `crates/omnigraph/tests/failpoints.rs`, + `crates/omnigraph/tests/memwal_stream.rs`, `crates/omnigraph/tests/memwal_stream_cost.rs`, and `crates/omnigraph-cluster/tests/failpoints.rs` (integration binaries, never in-source — the fail registry is process-global). Run the main suites with @@ -471,6 +510,7 @@ CI runs these S3-backed **correctness** tests against a containerized RustFS ser - `cargo test -p omnigraph-cluster --test s3_cluster` (full control-plane lifecycle on the bucket) - `cargo test -p omnigraph-cli --test system_local local_cli_s3_end_to_end_init_load_read_flow` - `cargo test -p omnigraph-engine --features failpoints --test failpoints s3_` (recovery-sidecar lifecycle on a real bucket) +- `cargo test -p omnigraph-engine --features failpoints --test memwal_stream s3_provider_shard_manifest_failure_retains_unreferenced_generation -- --exact --nocapture` (RFC-026 B2a provider-boundary failure after complete randomized generation output; CI rejects an explicit skip and proves the root remains non-authoritative and untouched below its root through retry/reopen) - `cargo test -p omnigraph-engine --features failpoints --test memwal_stream_cost gate_r0_widest_generation_closes_and_records_retain_all_growth_on_configured_rustfs -- --exact --nocapture` (RFC-026's configured-object-store 8,192-row/near-32-MiB closure twin; the workflow requires its measured `rustfs` line and positive-test polarity, so an explicit skip is a failure) Locally, set `OMNIGRAPH_S3_TEST_BUCKET` (and the usual `AWS_*` vars including `AWS_ENDPOINT_URL_S3` for non-AWS) before running. Without those, S3 tests skip gracefully. @@ -483,6 +523,11 @@ RFC-025's S3 **cost** matrix is likewise on demand and was not run for the 2026-07-17 local no-go decision: `OMNIGRAPH_S3_TEST_BUCKET=… cargo test -p omnigraph-engine --test checkpoint_retention_cost s3_checkpoint_retention_matrix_is_exact_and_records_the_current_no_go -- --exact --nocapture`. +RFC-026 B2a's configured-RustFS 1/8/32/128 retained-history **cost** sweep is +also on demand; its LIST totals, wall time, and RSS are advisory and its terms +must remain separate: +`OMNIGRAPH_S3_TEST_BUCKET=… cargo test -p omnigraph-engine --features failpoints --test memwal_stream_cost b2a_retained_history_decision_scale_sweeps_to_128_generations_on_configured_rustfs -- --ignored --exact --nocapture`. + RFC-026 Gate E0's configured RustFS cell is both classifier and complete exact-probe evidence. It owns the positive lost-result/index/empty-shard/reopen sequence, listing-dependent foreign/malformed/loose/data/cursor/corrupt @@ -711,6 +756,7 @@ Correctness bugs fail loudly in tests; cost-scaling bugs pass every test and deg - **Apply the same rule to RFC-025.** `checkpoint_retention_cost.rs` keeps live checkpoint count and catalog width fixed while unrelated journal history grows, and counts complete list/show/cleanup-root authority reads. The uncompacted reconciled counters and bounded tail are flat; compacted scan bytes and the 1,000-commit operation boundary are not, so the assertions preserve a no-go. Run the default local matrix with `cargo test -p omnigraph-engine --test checkpoint_retention_cost local_checkpoint_retention_matrix_is_exact_and_records_the_current_no_go -- --exact --nocapture`; run the ignored decision scale with `cargo test -p omnigraph-engine --test checkpoint_retention_cost local_checkpoint_retention_matrix_at_one_thousand_commits -- --ignored --exact --nocapture`. A green test means the known result was reproduced, not that RFC-025 passed Gate 0. - **Keep RFC-026 Gate E0 reproducible.** The first `checkout_latest`/`IOTracker` instrument was false-green because local `read_dir` escaped tracking; it is not acceptance evidence. The green harness uses the public but guide-hidden `Dataset::has_successor_version` from freshly ABA-verified exact `N`, probes only `N + 1`, then uses exact `N + 1` to reject buried `N + 2`. `AttemptTracker` records before forwarding, including failed/`NotFound` HEADs, and versions 8/80 must retain the identical four-success-HEAD + one-NotFound-HEAD + one-success-GET shape with zero lists. The Unix execute-only `_versions` tripwire must keep exact probing green while latest enumeration fails, and an unreadable exact HEAD must error. Run the 14-substantive-cell local file with `cargo test -p omnigraph-engine --test memwal_enrollment_gate -- --nocapture`; its fifteenth bucket-gated cell logs an explicit skip when unconfigured. Run the exact configured RustFS command above for its positive plus listing-dependent negative matrix. Green E0 authorized only Phase A; Phase A has now activated v7 foundation state, but E0 never authorizes row admission, acknowledgement, or fold. - **Preserve RFC-026 Gate R0's closure and retain-all evidence.** Its current-object census is useful only inside its stated observation boundary; never label LIST totals as provider-retained or billed bytes, and never turn them into a quota claim. The source audit remains pinned to the exact Lance lockfile revision. The high-entropy cell must continue to admit a legal 8,192-row/near-32-MiB generation, keep acknowledgement graph-invisible, close through logical-slice charge plus dense per-scanner-batch take, publish exactly once, and preserve every previously listed immutable path/class/size. Use `cargo test -p omnigraph-engine --features failpoints --test memwal_stream_cost gate_r0_ -- --nocapture`. Also run `widest_legal_generation_records_no_roll_estimates_and_peak_rss` exactly: its observed fold delta was 284,934,144 bytes (about 272 MiB), and 384 MiB is a remeasurement tripwire for the single-exclusive-fold implementation shape, not a runtime hard allocator limit. A green run proves the private evidence only; public product contracts remain inactive. +- **Preserve RFC-026 B2a's no-delete and provider-failure evidence.** Reuse the shared strict MemWAL classifier; do not weaken it with a second path parser. Complete and partial unreferenced generation roots must stay non-authoritative and receive zero subtree reads, writes, deletes, or adoption through retry/reopen. Parent shard discovery may observe their prefix. Canonical durable MemWAL deletion remains zero; only an exact losing shard-manifest-CAS `.binpb.tmp.` staging path is allowed. Run `cargo test -p omnigraph-engine --features failpoints --test memwal_stream provider_` and `cargo test -p omnigraph-engine --features failpoints --test memwal_stream_cost b2a_`. Run the ignored 1/8/32/128 local/RustFS sweeps when the retained-history shape or Lance pin changes. Keep every term separate and describe LIST bytes, wall time, and RSS as advisory diagnostics, never as a quota, SLO, provider bill, or isolated WAL slope. - **Count on the handle that does the reads, not just the one a measured op opens.** Lance's IO-counted tests attach the `IOTracker` to the (warm, cached) dataset and read `incremental_stats()` per request — the tracker MUST be on the handle performing the reads, or warm-handle reads escape. A per-op tracker installed at measure time cannot see reads on a long-lived handle opened earlier (the warm coordinator's `__manifest` handle, reused across writes), so such reads were silently undercounted. Wrap a depth-swept body in `cost_harness` so the manifest tracker is installed before the graph opens and `manifest_reads` is **ground truth** (handle-age-irrelevant). The `version_probes` counter is the freshness-probe *call* count; ground truth additionally reveals that a write's probe does ~3 object-store RPCs (a read's probe is a 0-IO cache hit). `manifest_reads_capture_warm_probe` is the guard that this stays true. - This is the testing companion to invariant 15 in [docs/dev/invariants.md](invariants.md) (hot-path cost is bounded by work, not history). diff --git a/docs/dev/wal-thinking.md b/docs/dev/wal-thinking.md index 8fec7745..3293ac3c 100644 --- a/docs/dev/wal-thinking.md +++ b/docs/dev/wal-thinking.md @@ -7,8 +7,8 @@ contract difference between an interactive graph commit and durable stream admission, the expected performance shape and evidence still required, and the build inventory. -Current boundary: RFC-026 Phase A and the Phase-B1 private core are built, but -public streaming is not. Gate R0 found and the follow-up fixed the one known +Current boundary: RFC-026 Phase A, the Phase-B1 private core, and the private +B2a unbounded retain-all gate are built, but public streaming is not. Gate R0 found and the follow-up fixed the one known all-shape closure failure: a legal high-entropy near-cap generation is durably acknowledged, materialized, folded, and published without lowering the 8,192-row/32-MiB admission cap. The fold now charges logical Arrow slices and @@ -33,12 +33,21 @@ product surface. There is still no `@stream`, public enrollment or row admission, SDK/HTTP/CLI/OpenAPI route, operator drain/resume workflow, or fresh read. RFC-026 remains Draft. -RFC-026 now selects **unbounded retain-all on stock Lance** as the first storage -profile. OmniGraph deletes no `_mem_wal` object and advertises no file-count or -retained-byte limit. RC.1's missing durable materialization-attempt receipt and -complete physical-output envelope therefore remain useful facts, but no longer -block activation: the selected contract deliberately accepts monotonic storage -and loud provider-capacity exhaustion. Managed reclamation and the Lance patch +RFC-026 selects **unbounded retain-all on stock Lance** as the first storage +profile, and its private B2a gate is implemented. OmniGraph deletes no canonical +durable `_mem_wal` object and advertises no file-count or retained-byte limit. +Lance may remove only a losing manifest-CAS temporary staging object. Complete +and partial orphan generation output remains non-authoritative and untouched +below its root through retry/reopen, though parent discovery may observe the +prefix. Local/configured-RustFS provider failures pin the typed outcomes. The +1/8/32/128 retained-history instrument keeps warm acknowledgement, replay, +fold, visibility, table, graph-manifest, adapter, object, and RSS terms separate; +it shows growing combined retained-history work without creating a quota or +SLO. RC.1's missing durable materialization-attempt receipt and +complete physical-output envelope therefore remain useful facts, but did not +block the private B2a gate: the selected contract deliberately accepts +monotonic storage and loud provider-capacity exhaustion. Public activation +still waits on B2-common. Managed reclamation and the Lance patch are deferred optimizations. Common compare-and-chain token, trusted attribution, persistent lifecycle/correction, authorization, and product- parity contracts remain specified and inactive. @@ -211,13 +220,15 @@ publication and recovery chassis. It reuses that chassis, but it adds stream binding and generation authority, fold-time validation, merge progress, and reject/audit atomicity. -The selected retain-all profile never deletes folded MemWAL data. WAL entries, +The selected retain-all profile never deletes canonical durable MemWAL data. WAL entries, flushed generations, random failed-attempt subtrees, and fence sentinels remain in the root even after the exact fold is graph-visible and recovery is settled. That is deliberate: generic table-version cleanup leaves `_mem_wal` untouched, and deleting the successor's empty WAL fence sentinel can let a stale writer report a later WAL PUT as durable because RC.1 lacks a post-success epoch -check. OmniGraph therefore never deletes raw MemWAL paths. +check. OmniGraph therefore never deletes a canonical durable MemWAL object. +Lance's removal of a losing `.binpb.tmp.` manifest-CAS staging object is +allowed because it never became shard authority or retained history. A Lance-owned inspect/plan/execute protocol can add managed reclamation later. Its attempt receipts, inventory, fencing, and accounting are no longer the next @@ -245,8 +256,8 @@ activation slice because the first profile promises no retained-storage bound. block a strict fold or be atomically dead-lettered when that mode is configured. The acknowledgement cannot be withdrawn. 3. **More lifecycle machinery.** Enrollment, owner routing, backpressure, - drain/seal/resume, replay, fold health, and cleanup become operator-visible - responsibilities. + drain/seal/resume, replay, fold health, retained-capacity provisioning, and + retention diagnostics become operator-visible responsibilities. 4. **Write and monotonic storage amplification.** Data may exist in a WAL entry, a flushed Lance generation, the base dataset, and maintained indexes. Under retain-all, those MemWAL copies are not reclaimed in place. @@ -396,7 +407,7 @@ These are real substrate capabilities, not a turnkey graph-streaming product: | **`merged_generations`** | Merge progress updated atomically with a base-table merge. It supplies the marker needed for idempotent per-table folding; RFC-022 still owns graph publication and partial-effect recovery. | | **WAL replay** | Durable entries are replayed to reconstruct the MemTable under the next valid claimant. The guarantee depends on respecting fencing, lifecycle, compatibility, and safe GC. | | **LSM merging reads** | A scanner over the base table, selected flushed generations not yet safely replaceable by the base read plan, and optionally same-process active/frozen MemTables. OmniGraph must build and retain a coherent graph-level fresh-read cut. It does not query raw WAL files as a normal read source. | -| **Retention behavior** | Upstream describes when generations may be obsolete and warns that deleting WAL files can weaken fencing. The selected profile does not turn eligibility into deletion: generic cleanup ignores `_mem_wal`, OmniGraph never deletes raw paths, and retained storage grows monotonically. A later managed profile may consume a Lance-owned opaque reclamation primitive. | +| **Retention behavior** | Upstream describes when generations may be obsolete and warns that deleting WAL files can weaken fencing. Private B2a implements the selected profile without turning eligibility into deletion: generic cleanup ignores `_mem_wal`, OmniGraph never deletes a canonical durable object, and retained storage grows monotonically. Lance may remove only a losing manifest-CAS `.binpb.tmp.` staging object, which never became authority. A later managed profile may consume a Lance-owned opaque reclamation primitive. | | **Staged merge-insert** | `execute_uncommitted` plus atomic `merged_generations` fits the RFC-022 staged shape. There is no one-call MemWAL fold API. | | **Shared sessions/store parameters in RC.1** | Writer-created generations and base-backed scanner paths reuse the base dataset's access context; fresh-only construction still receives that context from its caller. OmniGraph owns long-lived writer/session lifecycle and scanner integration. | | **Key-based sharding** *(later)* | Horizontal scale when every occurrence of one key deterministically maps to one shard. | @@ -405,10 +416,12 @@ OmniGraph already has reusable chassis: the manifest publisher, generic recovery-sidecar framework, graph lineage, stable table identity, keyed write adapter, and validation components. “Reusable” does not mean “unchanged”: Phase A added stream lifecycle authority and enrollment recovery. Private B1 -now adds one nominally bounded admission worker, watcher success plus the same +adds one nominally bounded admission worker, watcher success plus the same writer's post-durability epoch check, replay and flushed-generation classification, and one recovery-owned fold. The all-shape closure repair is -implemented. Later phases still need the common token/attribution, persistent +implemented. Private B2a adds the no-delete structural guard, provider-failure/ +orphan matrix, and retained-history instrument over that core. The next phase +still needs the B2-common token/attribution, persistent operator lifecycle/correction, authorization, and product contracts; then the public caller, reject participants, and fresh cuts. Managed cleanup is optional future work rather than an activation prerequisite. @@ -450,13 +463,14 @@ workflow is implemented. | **Durability batching** | **Phase B1 implemented privately:** one admitted call is one non-empty, already-normalized physical `RecordBatch` and one Lance put. The worker owns the final check, invocation, watcher, and post-durability epoch check; only watcher success followed by same-writer `check_fenced()` success acknowledges. Anything ambiguous after invocation is typed `AckUnknown`; replay preserves possible residue but never resolves that attempt. Automatic rollover is disabled and the writer retires before a successor-generation put. There is no hidden group-commit policy until an instrument justifies one. | | **Pre-ack validation** | **Phase B1 implemented privately:** apply every rule that needs no graph/base-table read before WAL persistence. The private seam consumes physical vectors supplied in the normalized batch; it neither calls an external embedding provider nor invents unspecified fold-derived fields. | | **Fold adapter** | **Phase B1 implemented privately and closure-green:** capture one exact stream binding and post-drain shard snapshot, independently prove empty frozen refs plus the exact authoritative generation/cursor (RC.1's drain waiter alone is insufficient), run base-dependent validation, require the LWW output to fit 8,192 rows / 32 MiB, stage one merge-insert with `merged_generations`, and publish the exact table/lifecycle/lineage outcome through recovery schema v11 plus one `__manifest` CAS. Scanner slices are charged by logical size and rebuilt densely before retention. Already-normalized physical vectors pass through unchanged. Seal/drain/abort stay background-owned across caller deadlines; recognized unreferenced generation subtrees are retained forever under the selected profile. | +| **Retain-all storage gate** | **Private B2a implemented:** structural guards forbid a production reclamation/adoption side door; injected local/configured-RustFS provider failures keep complete/partial orphan output retained, non-authoritative, and untouched below its root through retry/reopen; and the 1/8/32/128 instrument keeps acknowledgement, replay, fold, visibility, table, graph-manifest, adapter, object, and RSS terms separate. Canonical durable delete requests and older-root IO remain zero. Lance losing manifest-CAS temp staging is allowed; LIST/timing/RSS results are advisory. | | **Strict and dead-letter disposition** | **B1/common B2:** strict only; a permanent deferred-validation failure leaves durable unmerged input and blocks progress loudly. B1 has no correction lane. The common B2 inventory specifies bounded `REPLACE`/`WITHDRAW` correction over the immutable blocked cut, without a second generation or silent drop. **Phase C:** dead letters only after restart-stable reject identity and retention are defined. | | **Policy, lineage, and audit** | B1 records only fixed mechanism lineage and has no public caller. The common B2 inventory stores trusted contributor/write metadata with the row, publishes current token state with the base version, and commits a fixed winner summary; it promises durable attribution for visible winners/current withdrawals, not unbounded audit retention for every superseded acknowledgement. Phase C consumes that evidence for rejects. | -| **Quiescence and rebind** | The common B2 inventory specifies durable `OPEN -> DRAINING -> SEALED -> OPEN`, strictly monotonic lifecycle revisions, bounded complete terminal management receipts, authoritative status, roll-forward-only resume/abort-drain recovery, rebuild preflight, and bounded correction. Retain-all is selected, but these lifecycle contracts still must be implemented and proved before public activation. Resume rechecks the bounded no-named-branch topology; an incompatible branch operation leaves the stream `SEALED`. `SEALED` permits export/rebuild, not in-place maintenance. Phase D integrates automatic operation drain and physical rebind. | +| **Quiescence and rebind** | The common B2 inventory specifies durable `OPEN -> DRAINING -> SEALED -> OPEN`, strictly monotonic lifecycle revisions, bounded complete terminal management receipts, authoritative status, roll-forward-only resume/abort-drain recovery, rebuild preflight, and bounded correction. Private B2a implements retain-all, but these lifecycle contracts still must be implemented and proved before public activation. Resume rechecks the narrow main-only/no-named-branch topology; an incompatible branch operation leaves the stream `SEALED`. `SEALED` permits export/rebuild, not in-place maintenance. Phase D integrates automatic operation drain and physical rebind. | | **Fresh reads** *(later)* | Explicit committed/fresh IR mode, exact base-plus-MemWAL cut, merged-generation exclusion, retention guards, and documented lack of cross-table atomicity. | -| **Retention profile** | **Unbounded retain-all is selected:** no OmniGraph MemWAL GC, no file-count or retained-byte admission limit, and monotonic storage for the root. Provider exhaustion is an accepted loud operational risk. RC.1's missing durable attempt receipt and complete output envelope remain documented but do not block a contract that promises neither limit. **Managed reclamation** retains the Lance-owned durable inspect/plan/execute design as an optional later optimization. OmniGraph never deletes raw `_mem_wal` paths. | +| **Retention profile** | **Private B2a implements unbounded retain-all:** no OmniGraph MemWAL GC, no file-count or retained-byte admission limit, and monotonic canonical durable storage for the root. Complete/partial orphan roots remain non-authoritative and untouched below their roots through retry/reopen; parent discovery may observe a prefix. Lance may remove only a losing manifest-CAS temporary staging object. Provider exhaustion is an accepted loud operational risk. RC.1's missing durable attempt receipt and complete output envelope remain documented but do not block a contract that promises neither limit. **Managed reclamation** retains the Lance-owned durable inspect/plan/execute design as an optional later optimization. | | **Graph-manifest lifetime** | Retain-all also places no finite-lifetime bound on base/token or shared `__manifest` history. `GraphHistoryBudget` is not part of the first profile. If a later product promises a hard whole-root bound, it must earn a separate RFC and physical-growth evidence across every manifest publisher. | -| **Evidence and operations** | **B1 closure green:** surface guards, genuine v7↔v8 refusal/rebuild, the graph-level behavior/crash/race cells, and qualified cost/PK-index/RSS evidence pass. The legal high-entropy near-cap cell now acknowledges, materializes, folds, publishes exactly once, and retains every listed path. The full-fold RSS delta measured 284,934,144 bytes (~272 MiB); the 384-MiB check is a CI remeasurement tripwire for one exclusive fold, not a runtime allocator limit. The revision-pinned source audit and current-object sweeps remain useful descriptions of monotonic storage, not a rejected-gate disposition. Two guards continue to prove why generic cleanup and raw fence deletion are unsafe. Public metrics/status, API tests, CLI parity, and operator evidence remain absent by design. | +| **Evidence and operations** | **B1 closure and private B2a gate green:** surface guards, genuine v7↔v8 refusal/rebuild, graph-level behavior/crash/race cells, and qualified cost/PK-index/RSS evidence pass. The legal high-entropy near-cap cell acknowledges, materializes, folds, publishes exactly once, and retains every listed path. The full-fold RSS delta measured 284,934,144 bytes (~272 MiB); the 384-MiB check is a CI remeasurement tripwire for one exclusive fold, not a runtime allocator limit. B2a adds structural no-delete checks, typed local/configured-RustFS provider failures, inert complete/partial orphan evidence, and a 1/8/32/128 retained-history instrument. Older roots receive zero IO and canonical durable deletes remain zero; warm-ack operations stay flat while authority/history bytes and combined fold work grow. LIST totals, timing, and RSS are advisory. Two guards continue to prove why generic cleanup and raw fence deletion are unsafe. Public metrics/status, API tests, CLI parity, and operator evidence remain absent by design. | The table intentionally omits “small/medium” estimates. Atomic rejection, quiescence, fresh-read retention, and any future GC are correctness protocols; @@ -489,9 +503,10 @@ covering both: An upstream exact receipt and public admission lifecycle would simplify this substantially. We may propose that change, but its review and release timing is not a prerequisite for retain-all. Gate E0 passed; the -upstream shape still gates overlapping-process topology. Bounded Phase A and -the private B1 core are implemented, and the Gate R0 closure gap is repaired. -The common product and operational gates remain before public row activation. +upstream shape still gates overlapping-process topology. Narrow Phase A and +the private B1 core are implemented, the Gate R0 closure gap is repaired, and +the private B2a retain-all gate is implemented. The common product and +operational gates remain before public row activation. ### The RC.1 reclamation gap, stated precisely (B2b) @@ -619,7 +634,8 @@ remain green. Gate R0's legal high-entropy near-cap failure is now repaired and the closure cell publishes successfully. The missing stock-RC.1 attempt receipt and complete-output envelope remain factual limits on any future bounded- storage claim, but do not block the selected unbounded retain-all profile. The -common token, attribution, lifecycle/correction, authorization, and product +private B2a structural/provider/history gate is implemented; the B2-common +token, attribution, lifecycle/correction, authorization, and product contracts must still close before any public row acknowledgement. ### Bottom line @@ -641,10 +657,11 @@ The plan is: 3. retain the repaired logical-slice/dense fold and its local/configured-RustFS near-cap closure plus full-fold RSS remeasurement instruments as regression gates; do not silently shrink the 8,192-row/32-MiB admission cap; -4. implement the selected unbounded retain-all profile: delete no MemWAL path, - add no file/byte admission limit, expose observed growth only as advisory, - and surface provider-capacity failure loudly through recovery; -5. retain the common B2 compare-and-chain token, trusted attribution, +4. preserve the implemented private B2a unbounded retain-all gate: delete no + canonical durable MemWAL object, add no file/byte admission limit, keep + complete/partial orphan roots inert, expose observed growth only as + advisory, and surface provider-capacity failure loudly through recovery; +5. implement the B2-common compare-and-chain token, trusted attribution, manifest-selected token participant, protocol-v2 lifecycle, and bounded `REPLACE`/`WITHDRAW` correction as the implementation contract; 6. keep B2b Lance-owned reclamation and a whole-root finite-lifetime budget as @@ -666,8 +683,8 @@ calendar. Phase A's v7 foundation and private B1's current v8/config-v2/v11 worker/fold core are implemented, and the widest admitted high-entropy shape now closes. RFC-026 remains Draft because the public token, attribution, lifecycle/correction, authorization, and product surfaces are not implemented. -The selected first storage posture is unbounded retain-all: no OmniGraph -MemWAL GC and no file/byte limit. Two negative RC.1 reclamation guards remain +The selected first storage posture is the implemented private B2a unbounded +retain-all gate: no OmniGraph MemWAL GC and no file/byte limit. Two negative RC.1 reclamation guards remain checked in as rationale for that no-deletion rule. No schema-v9 contract, production enrollment, acknowledgement, fold, operator, or public stream path exists. diff --git a/docs/dev/writes.md b/docs/dev/writes.md index 6a626f8b..f81a75b4 100644 --- a/docs/dev/writes.md +++ b/docs/dev/writes.md @@ -543,7 +543,7 @@ Empty-table overwrite is represented as a valid zero-fragment Lance `Overwrite` transaction, not as truncate-then-append. -### RFC-026 private stream foundation and B1 core +### RFC-026 private stream foundation, B1 core, and B2a retain-all gate Internal schema v8 activates the deliberately bounded, data-bearing MemWAL format. Phase A's enrollment foundation and Phase B1's one-generation row/fold @@ -652,11 +652,19 @@ subprocess measurement recorded a 284,934,144-byte isolated fold RSS delta the measured implementation shape; changing the admission or fold strategy requires remeasurement. -The selected next profile is **unbounded retain-all**. It accepts that +The private B2a gate now implements **unbounded retain-all**. It accepts that stock-RC.1 materialization uses randomized generation roots and that a crash or cold replay can leave more physical objects. OmniGraph adds no retained-byte, object-count, file-count, or history quota, no quota-enforcement attempt ledger, -and no raw `_mem_wal` deletion. Current-object LIST evidence remains a useful +and never deletes a canonical durable `_mem_wal` object. Lance may remove only +its losing shard-manifest-CAS `.binpb.tmp.` staging object; that object +never became shard authority or retained history. Complete and partial +unreferenced generation output remains non-authoritative. Parent shard discovery +may list its common prefix, but admission, retry, and cold reopen never descend +into, read, mutate, adopt, or delete the subtree. A shared strict classifier and +local/configured-RustFS provider-failure cells pin those rules and the typed +effect-free `Lance`, post-invocation `AckUnknown`, and post-cut +`RecoveryRequired` outcomes. Current-object LIST evidence remains a useful retention/path-shape check but does not claim coverage of incomplete multipart uploads, provider versions/delete markers, local staging, or billed bytes. Provider exhaustion is surfaced as a storage failure and can halt admission, @@ -666,8 +674,13 @@ Arrow admission envelope, deadlines, retry counts, and ambiguity outcomes remain bounded. RC.1's missing cross-open materialization receipt and complete physical-output envelope are therefore not blockers for this profile. They remain relevant to -the future B2b managed-reclamation design. This decision activates no format or -product state. +the future B2b managed-reclamation design. The checked-in 1/8/32/128 local and +configured-RustFS instrument separates warm acknowledgement, cold replay, fold, +visibility, table, graph-manifest, adapter, advisory object, and whole-process +RSS terms. Warm acknowledgement retains a flat operation shape while serialized +authority and combined retained-history work grow; the observations are not a +quota, latency SLO, provider bill, or isolated WAL slope. B2a activates no new +format or product state. RFC-026's common B2 inventory specifies—but does not implement—the next product contract. @@ -840,8 +853,8 @@ identity, incarnation, path, and Lance version. > reader of `recovery.rs`, `failpoints.rs`, or this document only > encounters phase letters in the per-writer context. -RFC-026's named “Phase A foundation” and “Phase B1 private core” are roadmap -slices, not steps in that four-phase convention. Recovery-v10 +RFC-026's named “Phase A foundation”, “Phase B1 private core”, and “B2a +retain-all gate” are RFC slice names, not steps in that four-phase convention. Recovery-v10 `StreamEnrollment` uses a dedicated exact initializer classifier: no effect retires, index-only provisions the fixed empty shard, and index-plus-empty-shard publishes pointer + lifecycle. Recovery-v11 `StreamFold` diff --git a/docs/dev/writing-path-state-of-affairs.md b/docs/dev/writing-path-state-of-affairs.md index 03cd1fec..adfdd518 100644 --- a/docs/dev/writing-path-state-of-affairs.md +++ b/docs/dev/writing-path-state-of-affairs.md @@ -10,8 +10,8 @@ decision points **Change-set boundary:** this page describes current main through RFC-026's Phase A foundation, private Phase B1 core, the Gate R0 findings, the subsequent -all-shape closure repair, and the decision to ship an unbounded retain-all -profile first. +all-shape closure repair, and the implemented private B2a unbounded retain-all +gate. RFC-024, RFC-025, and RFC-027 remain research-blocked at their recorded evidence gates. Gate E0 first proved the bounded public Lance state classifier; Phase A activated internal schema v7 with recoverable empty enrollment, durable @@ -27,18 +27,24 @@ charges logical slices and rebuilds dense arrays with `take`; that exact shape acknowledges, materializes, folds, and publishes. The measured RSS delta for one exclusive full fold is 284,934,144 bytes (about 272 MiB). CI's 384-MiB threshold is a remeasurement tripwire, not a runtime allocator or hard memory -limit. The 2026-07-20 post-containment -local warm-ack probe remains history-flat at 9 table reads / 219 bytes; -configured RustFS retains only its 2026-07-19 pre-containment baseline and must -be rerun before a current object-store ack-cost claim. +limit. Gate R0 historically rejected a *bounded, finite-lifetime* retain-all claim on stock RC.1 because materialization has no durable attempt receipt or complete physical-output envelope. We have deliberately dropped that claim. The first -profile is unbounded retain-all: OmniGraph never deletes MemWAL objects, sets -no retained-byte or file-count limit, and treats provider exhaustion as a loud -operational failure. The missing receipt/envelope therefore does not block -activation. Managed reclamation remains optional Lance-owned work for a later +profile is unbounded retain-all. Its private B2a gate is implemented: OmniGraph +never deletes a canonical durable MemWAL object, sets no retained-byte or file- +count limit, and treats provider exhaustion as a loud operational failure. +Lance may remove only a losing shard-manifest-CAS temporary staging object. +Complete and partial orphan output stays non-authoritative and is not descended +into, read, mutated, adopted, or deleted through retry/reopen; parent discovery +may still observe its prefix. Local and configured-RustFS provider-failure +tests pin those rules. The missing receipt/envelope therefore does not block +this profile. The 1/8/32/128 retained-history instrument keeps acknowledgement, +replay, fold, visibility, table, graph-manifest, adapter, object, and RSS terms +separate. It shows flat warm-ack operation shape alongside growing serialized +authority/combined history work; its timings, RSS, and LIST totals are advisory, +not quotas or SLOs. Managed reclamation remains optional Lance-owned work for a later profile. The common explicit enrollment, compare-and-chain token, trusted attribution, and lifecycle-receipt/correction contracts remain specified and inactive; a `GraphHistoryBudget` is not part of the immediate plan. @@ -101,8 +107,10 @@ facts. First, the original accounting could reject one legal admitted near-cap shape after durable acknowledgement; logical-slice accounting plus dense rebuilding now closes that shape. Second, stock RC.1 cannot prove a lifetime bound for materialization attempts or their complete physical growth. The -selected first profile makes no such claim: it retains every MemWAL object -without an OmniGraph file/byte limit and accepts loud provider exhaustion. +selected first profile makes no such claim. Its private B2a gate retains every +canonical durable MemWAL object without an OmniGraph file/byte limit, accepts +loud provider exhaustion, keeps complete/partial orphan roots inert, and +measures retained-history terms locally and on configured RustFS. Neither a test-only attempt ledger nor managed reclamation is on the immediate activation path. The next work is the common private token, attribution, lifecycle, and correction machinery, followed by product contracts only after @@ -228,7 +236,7 @@ retry rules, recovery classification, and the full owned limits. | [023 — Key-conflict fencing](../rfcs/0023-key-conflict-fencing.md) | **Implemented** | Internal schema v6 introduced exact-`id` PK metadata, closed keyed routing, typed conflicts, bounded replay, rebuild/refusal, and accepted performance evidence; v8 preserves that contract. | | [024 — Durable table heads](../rfcs/0024-durable-table-heads.md) | **Research-blocked** | The first in-manifest BTREE candidate has a specified logical contract and flat indexed row/range work, but fails the complete physical-I/O gate. No head rows or heads format are active. | | [025 — Checkpoint retention](../rfcs/0025-checkpoint-retention.md) | **Research-blocked** | Lance tag/pin semantics pass, but the proposed in-manifest registry access shape is not history-flat after compaction. No checkpoint rows, `ogcp_` production tags, API, or cleanup integration are active. | -| [026 — MemWAL streaming ingest](../rfcs/0026-memwal-streaming-ingest.md) | **Draft; Phase A/B1 private core implemented; widest-shape closure green; unbounded retain-all selected; public inactive** | Schema v8 preserves v7/recovery-v10 enrollment and adds config-v2 one-generation admission plus recovery-v11 strict fold. A clean acknowledgement still requires watcher success and a same-writer post-durability epoch check; logical-slice charging plus dense copies close the legal near-cap fold shape. Gate R0's historical no-go applies to a finite storage/lifetime promise: stock RC.1 cannot prove an attempt cap or complete physical-growth envelope. The selected first profile makes no such promise—it never deletes MemWAL objects and sets no retained-byte, object, file, or history quota. Common enrollment/token/attribution/lifecycle/correction, authorization, and product-parity contracts remain specified and inactive. `GraphHistoryBudget` and B2b managed reclamation belong to an optional future bounded profile. No schema v9 or product surface is active. | +| [026 — MemWAL streaming ingest](../rfcs/0026-memwal-streaming-ingest.md) | **Draft; Phase A/B1 private core and B2a retain-all gate implemented; public inactive** | Schema v8 preserves v7/recovery-v10 enrollment and adds config-v2 one-generation admission plus recovery-v11 strict fold. A clean acknowledgement requires watcher success and a same-writer post-durability epoch check; logical-slice charging plus dense copies close the legal near-cap fold shape. Gate R0's historical no-go applies to a finite storage/lifetime promise: stock RC.1 cannot prove an attempt cap or complete physical-growth envelope. Private B2a makes no such promise: it never deletes a canonical durable MemWAL object, keeps complete/partial orphan roots non-authoritative and untouched below their roots through retry/reopen, sets no retained-byte/object/file/history quota, pins provider failures locally and on configured RustFS, and records advisory 1/8/32/128 retained-history evidence. Lance's losing manifest-CAS temp staging is not retained authority. Common enrollment/token/attribution/lifecycle/correction, authorization, and product-parity contracts remain specified and inactive. `GraphHistoryBudget` and B2b managed reclamation belong to an optional future bounded profile. No schema v9 or product surface is active. | | [027 — Lineage merge deltas](../rfcs/0027-lineage-merge-deltas.md) | **Research-blocked** | The desired O(delta) classifier and fallback contract are specified. Selective live-row and deletion-delta discovery are not yet bounded, so `OrderedTableCursor` remains the correctness path. | | [028 — Stable schema identity](../rfcs/0028-stable-schema-identity.md) | **Implemented** | Rename-stable IDs, table incarnation, identity-derived paths, schema/recovery integration, and strict rebuild activation were introduced in v5 and remain active in v8. | @@ -240,7 +248,7 @@ retry rules, recovery classification, and the full owned limits. | History-flat authority | RFC-024/025 show flat BTREE rows/ranges/pages can coexist with history-growing manifest discovery or compacted bytes. No heads/checkpoint format is active; mutable tip caches and a second authority remain rejected. | A new Lance-native access shape—or revised measured operational contract—passes the original cold/warm, compacted/uncompacted, local/object-store gate. | | Internal history GC | Safe live-writer cleanup needs a durable resurrection/retention boundary; otherwise a stalled writer can recreate a collected version. | An evidence-backed cleanup watermark/fence before automated `__manifest` version GC. | | MemWAL delivery and closure | RC.1 initializes the system index and claims shards as separate effects without a caller-minted combined receipt or cross-process seal. Phase A recovers that gap exactly for main-only, one-shard, one-live-writer-process empty enrollment. At the row boundary, the durability watermark is writer-wide while batch positions reset after MemTable rollover, `put_no_wait` may mutate before returning `Err`, replay leaves its fresh BatchStore WAL watermark unset, and `wait_for_flush_drain` can lose a completed failure before a late waiter snapshots it. Neither batch positions nor WAL statistics are durable receipts. Gate R0 also found a private closure bug caused by sparse scanner arrays retaining oversized backing buffers; logical-slice charging and dense copies now close the exact legal near-cap shape. | Implement and prove the common compare-and-chain token, trusted attribution, revisioned lifecycle receipts, bounded correction, authorization, and product-parity contracts privately before adding a public caller. A public receipt/seal or accepted distributed fence remains the exit for overlapping processes and failover, not for the current single-live-writer-process profile. | -| MemWAL retained growth | A clean retained generation has measurable current objects, and the Gate R0 sweep proves that referenced currently listed immutable paths retain their class and size at one/four/eight folds. RC.1 still provides no durable cross-open attempt cap, complete physical-output receipt, or provider-billed-byte inventory. That prevents OmniGraph from promising a finite retained-storage bound. | The selected first profile is deliberately unbounded: never delete raw `_mem_wal` objects, impose no retained-byte/object/file/history quota, and fail loudly if the provider refuses further writes. No attempt ledger or physical-growth reservation is required for a contract that promises no storage bound. Add those only if a later profile claims bounded retention. | +| MemWAL retained growth | A clean retained generation has measurable current objects, and the Gate R0 sweep proves that referenced currently listed immutable paths retain their class and size at one/four/eight folds. RC.1 still provides no durable cross-open attempt cap, complete physical-output receipt, or provider-billed-byte inventory. That prevents OmniGraph from promising a finite retained-storage bound. | Private B2a implements the deliberately unbounded profile: never delete a canonical durable `_mem_wal` object, keep complete/partial orphan subtrees non-authoritative and untouched below their roots through retry/reopen, impose no retained-byte/object/file/history quota, and fail loudly if the provider refuses further writes. Lance may remove only its losing manifest-CAS temp staging. The 1/8/32/128 local/RustFS instrument records separate advisory terms and shows combined retained-history work grows; it does not create a bound. Add attempt ledgers or physical-growth reservation only if a later profile claims bounded retention. | | MemWAL reclamation (optional B2b) | Stock RC.1 exposes evidence-level raw listing and manifest reads, not a complete classified inventory or safe MemWAL delete/GC primitive. Generic `cleanup_old_versions` leaves `_mem_wal` unchanged. Worse, deleting the successor's empty WAL fence sentinel can let a stale writer complete a WAL PUT and report watcher success because RC.1 has no post-success epoch check. Private B1 contains that result for its own acknowledgement but cannot retract durable bytes or protect raw Lance callers. Raw path deletion in OmniGraph remains forbidden. | Keep B2b as optional Lance-owned durable inspect/plan/execute, attempt/receipt recovery, post-success fencing, bounded history checkpoint, strong inventory/accounting, and enforced-watermark work. It is not on the immediate retain-all activation path. | | O(delta) merge | A version-column predicate is still O(rows) without a selective source, and deleted rows have no live version columns. Full ID differencing remains correct. | Bounded live-row and deletion/change discovery, exact shadow agreement, and a table-size-flat one-row-delete gate. | | Optimize provenance | Compaction/reindex has no stable caller-minted transaction covering the complete effect. Optimize therefore uses bounded, not exact, provenance. | Both an upstream maintenance transaction API and distributed recovery fencing. | @@ -297,11 +305,14 @@ or slightly lower incremental RSS. That is not a roadmap signal. ### Now -1. **Keep the repaired B1 closure cell as a permanent regression gate.** The - exact 8,192-row high-entropy shape must continue to acknowledge, materialize, - fold, and publish locally and on configured RustFS. Keep 384 MiB as a CI - remeasurement tripwire for the isolated fold RSS delta, not as a runtime - allocation promise or a retained-storage limit. +1. **Keep the B1 closure and B2a retain-all evidence as permanent regression + gates.** The exact 8,192-row high-entropy shape must continue to acknowledge, + materialize, fold, and publish locally and on configured RustFS. Provider + failures must remain typed; complete/partial orphan roots must remain inert; + canonical durable MemWAL deletion must remain zero; and older retained roots + must receive zero IO. Keep 384 MiB as a CI remeasurement tripwire for the + isolated fold RSS delta, not as a runtime allocation promise or a retained- + storage limit. Treat 1/8/32/128 LIST, timing, and RSS results as advisory. 2. **Implement the common B2 contracts privately for the selected retain-all profile.** Add explicit first-use enrollment, compare-and-chain token authority, trusted hidden contributor attribution, persistent revisioned @@ -309,33 +320,27 @@ or slightly lower incremental RSS. That is not a roadmap signal. correction. Keep every step under recovery and the manifest visibility CAS. Do not add a storage quota, materialization-attempt ledger, `GraphHistoryBudget`, or raw `_mem_wal` deletion. -3. **Prove retain-all failure behavior.** Add local and configured-RustFS crash - coverage showing that acknowledged generations remain recoverable, unknown - objects are never adopted or deleted, provider write failures are loud, and - unresolved recovery blocks new progress. Preserve the existing row, - logical-Arrow-memory, deadline, retry, and ambiguity bounds even though - retained storage is unbounded. -4. **Add product surfaces last.** Only after the private common machinery and +3. **Add product surfaces last.** Only after the private common machinery and evidence are green should schema intent, SDK, HTTP, CLI, Cedar, OpenAPI, shutdown ownership, and authoritative status converge on the same core. -5. **Keep B2b managed reclamation independent and optional.** If a bounded +4. **Keep B2b managed reclamation independent and optional.** If a bounded profile is scheduled later, author Lance-owned durable inspect/plan/execute, attempt/receipt recovery, post-success epoch fencing, strong inventory or durable accounting, and an enforced retained-storage watermark. A graph-global `GraphHistoryBudget` would require its own RFC and every-writer evidence. Never delete `_mem_wal` paths from OmniGraph. -6. **Keep RFC-024/025/027 stopped at their research no-gos.** Their blockers are +5. **Keep RFC-024/025/027 stopped at their research no-gos.** Their blockers are independent of the v8 stream core; do not add RFC-024 heads, RFC-025 graph checkpoint rows/format, or RFC-027 lineage-delta state as incidental B2 work. -7. **Give a distributed recovery fence its own design and evidence gate.** +6. **Give a distributed recovery fence its own design and evidence gate.** Define authority, expiry/renewal, fencing tokens, and crash semantics before implementation; require adversarial multi-process tests on local and object storage. -8. **Continue low-risk v8 hardening.** Add missing resource/time budgets, +7. **Continue low-risk v8 hardening.** Add missing resource/time budgets, preserve cost-at-history-depth gates, and reduce constant factors only where the existing authority model remains intact. -9. **Coordinate upstream without making it the calendar.** The additional +8. **Coordinate upstream without making it the calendar.** The additional Lance asks are replay initializing the per-MemTable WAL watermark, drain completion that cannot lose a finished error, recoverable MemWAL enrollment/admission, conditional native ref operations, exact maintenance @@ -344,8 +349,8 @@ or slightly lower incremental RSS. That is not a roadmap signal. ### Only when an evidence trigger fires - **Public MemWAL row activation:** Phase A passed its bounded gate; private B1 - now closes the exact legal near-cap shape, and unbounded retain-all is the - selected storage posture. The RFC remains Draft and public activation remains + now closes the exact legal near-cap shape, and private B2a implements the + selected unbounded retain-all storage posture. The RFC remains Draft and public activation remains off until explicit enrollment, durable contributor attribution, compare-and-chain sequencing, strict correction/disposition, persistent revisioned lifecycle with bounded management receipts, authorization, @@ -414,6 +419,7 @@ Do not: | MemWAL Phase A lifecycle/exclusion/recovery | [`failpoints.rs`](../../crates/omnigraph/tests/failpoints.rs), [`forbidden_apis.rs`](../../crates/omnigraph/tests/forbidden_apis.rs), manifest/write-queue unit tests, [RFC-026 §12.2](../rfcs/0026-memwal-streaming-ingest.md) | | MemWAL private Phase B1 admission/fold/crash and cost evidence | [`memwal_stream.rs`](../../crates/omnigraph/tests/memwal_stream.rs), [`memwal_stream_cost.rs`](../../crates/omnigraph/tests/memwal_stream_cost.rs), worker/recovery unit tests, [RFC-026 §12.3](../rfcs/0026-memwal-streaming-ingest.md) | | MemWAL Gate R0 retention decision, current-object census, attempt reuse, repaired near-cap closure, and fold RSS tripwire | [`memwal_stream_cost.rs`](../../crates/omnigraph/tests/memwal_stream_cost.rs), [RFC-026 §0.2 and §12.4](../rfcs/0026-memwal-streaming-ingest.md) | +| MemWAL private B2a no-delete structure, provider-failure/orphan behavior, and 1/8/32/128 retained-history instrument | [`forbidden_apis.rs`](../../crates/omnigraph/tests/forbidden_apis.rs), [`memwal_stream.rs`](../../crates/omnigraph/tests/memwal_stream.rs), [`memwal_stream_cost.rs`](../../crates/omnigraph/tests/memwal_stream_cost.rs), shared [`helpers/memwal.rs`](../../crates/omnigraph/tests/helpers/memwal.rs), [RFC-026 §12.5](../rfcs/0026-memwal-streaming-ingest.md) | | MemWAL B2b reclamation ownership/no-go guards | [`lance_surface_guards.rs`](../../crates/omnigraph/tests/lance_surface_guards.rs), [RFC-026 §4.5.2](../rfcs/0026-memwal-streaming-ingest.md) | | Cross-version refusal/rebuild, including v6↔v7 and v7↔v8 | [`crossversion_upgrade.rs`](../../crates/omnigraph-cli/tests/crossversion_upgrade.rs) | @@ -443,12 +449,18 @@ materialization are followed by one successful fold and manifest publication. The one/four/eight-fold census and source audit still prove that current LIST cannot establish a lifetime/provider-billed bound and that RC.1 has no durable attempt cap or reserve-first complete-output envelope. Those are accepted facts -for unbounded retain-all, not blockers: the selected profile promises no -retained-storage ceiling and activates no collector. The RFC remains Draft and -every public surface remains inactive pending the common correctness and product -contracts. B2b's two additional guards prove why stock RC.1 generic cleanup and -raw successor-fence-sentinel deletion cannot implement a later managed profile; -they are a no-go boundary for reclamation, not for retain-all. +for unbounded retain-all, not blockers. Private B2a now adds the structural no- +delete guard, shared strict classifier, local/configured-RustFS complete-orphan +provider matrix, local partial-orphan matrix, and 1/8/32/128 cost sweep. Older +roots receive zero IO, canonical durable MemWAL delete requests remain zero, +and only Lance's exact losing manifest-CAS temp staging may be removed. Warm ack +operation shape stays flat while serialized authority and combined retained- +history work grow; those measurements are advisory and create no ceiling. The +RFC remains Draft and every public surface remains inactive pending the common +correctness and product contracts. B2b's two additional guards prove why stock +RC.1 generic cleanup and raw successor-fence-sentinel deletion cannot implement +a later managed profile; they are a no-go boundary for reclamation, not for +retain-all. ## Updating this page diff --git a/docs/rfcs/0026-memwal-streaming-ingest.md b/docs/rfcs/0026-memwal-streaming-ingest.md index b9637ead..7e27af54 100644 --- a/docs/rfcs/0026-memwal-streaming-ingest.md +++ b/docs/rfcs/0026-memwal-streaming-ingest.md @@ -10,9 +10,8 @@ owner: OmniGraph maintainers # RFC-026 — MemWAL streaming ingest -**Status:** Draft / Phase A foundation and the private Phase B1 row/fold core -implemented; the first production profile is unbounded retain-all with no -MemWAL GC; public streaming inactive +**Status:** Draft / Phase A, private Phase B1, and the private B2a unbounded +retain-all evidence gate implemented; common B2/public streaming inactive **Date:** 2026-07-10 **Gate E0 evaluated:** 2026-07-18 **Phase A foundation completed:** 2026-07-18 @@ -37,6 +36,10 @@ retain-all profile on stock RC.1; its storage findings are accepted limitations of the selected unbounded profile and its fold blocker is now closed (§0.2) **Retention decision:** 2026-07-21 — select unbounded retain-all/no-GC for the first profile; managed reclamation remains deferred (§4.5) +**Private Phase B2a gate completed:** 2026-07-21 — structural no-reclamation +guard, complete/partial provider-residue recovery matrix, and local/configured- +RustFS retained-history evidence passed (§12.5); no schema or product surface +was activated **Author track:** Maintainer design series **Depends on:** [RFC-022](0022-unified-write-path.md)'s unified write and generic recovery-sidecar protocol, plus @@ -128,11 +131,14 @@ privately** for one admission-bounded, no-roll generation, from admission through crash replay and one strict RFC-022 fold. The 2026-07-21 dense-scan repair closes the legal near-cap row shape that Gate R0 exposed. The core is reachable only through one feature-gated, doc-hidden engine test seam and is -not a product surface. The first production profile is now **B2a unbounded -retain-all**: stock Lance owns the WAL, OmniGraph never deletes or reclaims a -MemWAL object, and physical storage is allowed to grow monotonically without a -file, object, or byte quota. **B2b** remains the deferred managed-reclamation -profile using a Lance-owned primitive. The token, trusted-attribution, +not a product surface. The private **B2a unbounded retain-all** evidence gate +is also implemented: stock Lance owns the WAL, OmniGraph never deletes or +reclaims a canonical durable MemWAL object, and physical storage is allowed to +grow monotonically without a file, object, or byte quota. Lance may delete its +own losing `.binpb.tmp.` shard-manifest CAS staging files before they +become canonical objects; this is atomic-write cleanup, not MemWAL GC. **B2b** +remains the deferred managed-reclamation profile using a Lance-owned primitive. +The token, trusted-attribution, recovery, lifecycle, correction, and product-parity contracts in §4.1–§4.4 and §4.6 still apply. `GraphHistoryBudget`, physical-storage admission, and aggregate receipt-capacity reservations do not. No schema intent, production @@ -204,9 +210,11 @@ file/object/byte ceiling, no retained-storage admission watermark, no source-derived physical-output envelope, and no requirement that stock Lance return a materialization-attempt receipt. Referenced generations, WAL, fence sentinels, and unreferenced partial or complete materialization subtrees are -retained indefinitely. They are never adopted from path shape or listing, and -ordinary `cleanup` never treats them as reclaimable. Operators must provision -the backing store accordingly. Provider exhaustion remains an explicit +retained indefinitely. Parent `_mem_wal` listing may observe an orphan prefix +while discovering shard authority, but production never descends into its +subtree, reads it as a generation, mutates it, deletes it, or adopts it from +path shape. Ordinary `cleanup` never treats it as reclaimable. Operators must +provision the backing store accordingly. Provider exhaustion remains an explicit storage failure handled by the existing pre-/post-invocation and recovery rules; it is never permission to acknowledge, discard, or fabricate success. @@ -243,7 +251,7 @@ default snapshot isolation. Stream-mode deletes remain out of the first delivery and require the Lance tombstone surface plus a separate acceptance pass. -The bounded initial profile additionally excludes non-main streams, +The narrow initial main-only topology additionally excludes non-main streams, multi-shard ownership, overlapping writer processes, raw Lance writers, cross-process `Fresh`, and concurrent interactive/maintenance HEAD movement on an `OPEN` enrolled table. These are support-boundary refusals, not implicit @@ -1258,11 +1266,14 @@ discard, base-row delete, and direct `DRAINING -> OPEN` CAS are forbidden. #### 4.5.1 B2a unbounded retain-all/no-GC profile (selected) -B2a deletes no MemWAL object after fold, optimize, cleanup, restart, -correction, quiesce, or `SEALED`. Referenced generations, WAL files, historical -fence sentinels, and unreferenced partial or complete randomized -materialization subtrees all remain in the source root indefinitely. The -profile intentionally has no MemWAL file/object/byte limit, retained-storage +B2a deletes no canonical durable MemWAL object after fold, optimize, cleanup, +restart, correction, quiesce, or `SEALED`. Referenced generations, WAL files, +historical fence sentinels, and unreferenced partial or complete randomized +materialization subtrees all remain in the source root indefinitely. Lance's +local atomic-put implementation may delete only its own losing +`manifest/.binpb.tmp.` CAS staging files; they never become +shard authority and are not retained-history objects. The profile intentionally +has no MemWAL file/object/byte limit, retained-storage ledger, quota, admission watermark, physical-output estimator, storage reservation, or materialization-attempt receipt requirement. It does not promise bounded disk/object-store usage or indefinite service under finite @@ -1276,8 +1287,10 @@ acknowledgement still requires watcher durability plus the same-writer fence check. Every authority-changing effect still uses the existing manifest and recovery protocols, and unresolved authority ambiguity still blocks progress. No path, listing result, timestamp, or compatible-looking object may establish -ownership. Unreferenced physical residue is simply inert: it is retained, -never scanned as a generation, never adopted, and never deleted. +ownership. Unreferenced physical residue is simply inert: shard discovery may +observe its parent-level prefix, but production never descends into or reads +the subtree as a generation, never adopts it, never mutates it, and never +deletes it. Gate R0's first two findings in §0.2 therefore become accepted operational limitations rather than activation blockers. Its third finding is closed by @@ -1327,8 +1340,9 @@ seal/failover, or authorize any `_mem_wal` deletion or public B2b surface. OmniGraph therefore does not raw-delete `_mem_wal` objects, route them through `delete_unverified`, or compact shard-manifest history. B2b requires the narrow Lance-owned primitive below; an upstream proposal remains useful, but its -calendar does not block the separate OmniGraph-only successor evidence gate for -B2a. No temporary B2a work may impersonate reclamation or delete a path. +calendar did not block the completed private OmniGraph-only B2a evidence gate. +Only B2b reclamation remains deferred on that primitive. No B2a work may +impersonate reclamation or delete a canonical durable path. Stock RC.1 exposes only evidence-level raw object listing plus manifest reads, not the owned classified inventory below. The required public Lance shape is @@ -2289,7 +2303,8 @@ before it CASes the shard manifest. A crash in that interval can therefore leave a generation directory that no manifest references. The active-stream inventory classifies an unreferenced recognized `{hash}_gen_N/` subtree under the bound shard—complete or partial—as derived orphan output: it is never -adopted as a generation, never scanned or folded, and never deleted by B1. It +adopted as a generation, never descended into/read or folded, and never deleted +by B1. Parent-level shard discovery may observe only its prefix. It does not make the durable WAL residue unrecoverable. Any loose object outside that recognized generation-output subtree shape still fails closed. B2a keeps these orphans inertly and unmetered for the root's lifetime. Only a future B2b @@ -3098,9 +3113,10 @@ RFC remains draft. retract a WAL effect, make raw sentinel deletion safe, fence raw Lance callers, or provide a reclamation receipt, cross-process seal, or failover protocol. The Lance-owned post-success fence and retention patch therefore - remain B2b managed-reclamation and broader-topology gates. They do not block - public activation of the no-delete, single-live-writer-process B2a profile, - which retains the wrapper's post-watcher fence check. + remain B2b managed-reclamation and broader-topology gates. They did not block + the completed private no-delete, single-live-writer-process B2a gate, which + retains the wrapper's post-watcher fence check. Public activation still waits + on B2-common. **Phase B1 disposition — accepted 2026-07-19, closure gap found 2026-07-20 and repaired 2026-07-21** @@ -3122,7 +3138,9 @@ forged pre-claim/pre-put stream-sidecar cell. Gate R0 correctly found that the old scanner retained sparse backing buffers. The dense copy releases those buffers and restores the intended logical 32-MiB closure check; the near-cap cell now closes. This amendment changes no -product surface. The future contracts in §4.1–§4.6 remain unimplemented. +product surface. The B2-common contracts in §4.1–§4.4 and §4.6 remain +unimplemented; §4.5.1's private B2a gate is implemented, while §4.5.2's B2b +managed-reclamation profile remains optional and inactive. ### 12.4 Gate R0 — historical bounded-retention no-go; closure repaired @@ -3157,13 +3175,14 @@ for one exclusive fold. That tripwire is not a runtime hard allocator limit. Configured RustFS remains a post-merge/tag regression signal and is not used to invent a storage bound. -### 12.5 Phase B2a — unbounded retain-all/no-GC implementation gate +### 12.5 Phase B2a — unbounded retain-all/no-GC gate implemented -B2a is the selected next profile. Its implementation gate is deliberately -narrow: +B2a is the selected first profile. Its private implementation gate was +completed on 2026-07-21 and is deliberately narrow: -- retain every `_mem_wal` object and make every OmniGraph cleanup/repair path - structurally incapable of deleting or reclassifying it; +- retain every canonical durable `_mem_wal` object and make every OmniGraph + cleanup/repair path structurally incapable of deleting or reclassifying it; + Lance-owned losing atomic-CAS temp-file cleanup remains permitted; - preserve the existing 8,192-row/32-MiB logical generation limit, one resident writer, one exclusive fold, bounded queues/deadlines/retries, and dense near- cap closure evidence; @@ -3172,20 +3191,54 @@ narrow: listings or path shape; - prove cold replay, fold-only retry, strict-block retention, and every existing acknowledgement/fold crash boundary while unreferenced materialization - residue remains inert and untouched; and + residue remains retained and no production path descends into, reads, + mutates, deletes, or adopts its subtree; and - expose retained-object measurements only as advisory diagnostics, with no file/object/byte limit, quota, reservation, aggregate receipt cap, `GraphHistoryBudget`, or materialization-attempt receipt requirement. -This slice adds no schema v9, production caller, SDK/HTTP/CLI/OpenAPI/Cedar +The checked-in result is: + +- one AST/source guard inventories the only production `_mem_wal` literal + owners, keeps the raw classifier module-private, forbids reclamation/adoption + symbols and destructive primitives in the adapter, and keeps generic + maintenance unaware of the namespace. It is defense in depth beside Rust + visibility, exact cleanup call-site guards, and runtime evidence—not a claim + of whole-program alias/dataflow analysis; +- real object-store write refusal covers effect-free cold claim, + post-invocation WAL `AckUnknown`, and post-cut `RecoveryRequired`. Complete + and partial randomized generation output remains unreferenced and retained; + blocked admission, authoritative retry, and cold reopen perform no object + access at or below either orphan root. Retry publishes one fresh root, never + the orphan, and configured RustFS repeats the complete-output cell; +- the shared fail-closed inventory is the single test authority for canonical + WAL, shard-manifest, and generation paths. Unknown/malformed paths or broken + manifest authority fail rather than becoming residue evidence; and +- the B2a cost instrument separates warm acknowledgement, cold reopen/replay, + fold, and visibility probe at 1/8 generations in CI and 1/8/32/128 locally + and on configured RustFS on demand. Older roots receive zero reads, writes, + or deletes and canonical MemWAL delete requests remain zero. Local + shard-manifest CAS may delete only validated `.binpb.tmp.` staging. + +The 1→128 reference sweep keeps warm-ack operation counts flat (local: 9 table +reads, 2 writes, 21 adapter operations; RustFS: 12, 1, and 21), while bytes read +from growing shard authority increase from 351 to 16,521. Advisory currently +listed immutable bytes grow from about 37 KiB to 5.3 MiB. At depth 128 the +configured-RustFS fold observes 198 table reads and 2,097 graph-manifest-store +reads; this is the real combined accumulated base-table, graph-manifest, and +retained-WAL shape, not an isolated WAL slope. Wall times and whole-process RSS +are printed as diagnostics only and enforce no product threshold. + +This result adds no schema v9, production caller, SDK/HTTP/CLI/OpenAPI/Cedar surface, or lifecycle/correction implementation. Those remain §12.6 work. ### 12.6 Common public and B2b managed-reclamation activation gates -The common public contracts in §4.1–§4.4 and §4.6 proceed only after the private -§12.5 B2a gate passes. This section owns those common gates and B2b's optional -managed-reclamation gates. B2a does not need any B2b-only bullet. Public -activation waits for every common implementation/evidence item to be green. +The private §12.5 B2a gate has passed. The common public contracts in +§4.1–§4.4 and §4.6 are therefore the next implementation work. This section +owns those common gates and B2b's optional managed-reclamation gates. B2a does +not need any B2b-only bullet. Public activation still waits for every common +implementation/evidence item to be green. The design does not waive the persistent escape requirement: a user must never be left with a table that ordinary writers refuse but cannot be corrected, quiesced, or rebuilt. @@ -3404,7 +3457,7 @@ ordinary writers refuse but cannot be corrected, quiesced, or rebuilt. | A | bounded main/unsharded/single-live-writer enrollment adapter, all-lifecycle effect exclusion with only the `SEALED` native-branch exception, lifecycle/admission lease, then graph-format capability/refusal and strict rebuild | **Implemented 2026-07-18 (§12.2):** internal schema v7, recovery-v10 enrollment, durable lifecycle CAS, process-local exclusion, crash/partial-format refusal, and genuine v6↔v7 strand evidence; no public enrollment or row path | | B1 | **Implemented privately 2026-07-19; acknowledgement containment added 2026-07-20; widest-shape closure repaired 2026-07-21:** internal schema v8/config-v2, root-scoped one-generation admission worker, durability-watcher success followed by a same-writer post-durability epoch check, conservative active-state reopen/replay, the pinned RC.1 replay-watermark bridge, and one explicit strict RFC-022 fold; no production caller | The graph-level behavior/crash/race suite and genuine v7↔v8 refusal/rebuild remain green. Fold now charges logical slices and copies each scanner emission into dense owned arrays. The legal 8,192-row high-entropy near-cap generation folds and publishes exactly once without changing the 32-MiB admission cap; its isolated fold RSS delta is guarded by the 384-MiB remeasurement tripwire (§12.4) | | R0 | production-neutral retained-growth/source audit; current-object census; referenced-cut retry; legal high-entropy near-cap materialize/fold cell; no schema, public caller, or deletion | **Historical bounded-retention no-go 2026-07-20; disposition amended 2026-07-21 (§0.2/§12.4):** RC.1 still exposes neither a complete reserve-first physical envelope/receipt nor a durable cross-open randomized-attempt cap. Those facts prohibit a finite storage promise but do not block selected unbounded retain-all. The formerly red widest cell is now green locally and on the configured-RustFS CI path; current-object observations remain advisory retention evidence, not provider billing/accounting | -| B2a | selected unbounded retain-all/no-GC profile on stock Lance | **Selected 2026-07-21; implementation gate remains (§12.5):** no OmniGraph byte/object/file/history quota, no raw `_mem_wal` deletion, and loud provider exhaustion. Keep row/memory/deadline/retry/ambiguity bounds and prove no-delete, recovery, correction, lifecycle, token, attribution, authorization, and product parity before activation. No schema v9 or product surface is active yet | +| B2a | selected unbounded retain-all/no-GC profile on stock Lance | **Private gate implemented 2026-07-21 (§12.5):** no OmniGraph byte/object/file/history quota; zero canonical `_mem_wal` deletion; complete/partial provider residue remains retained, unreferenced, and untouched below its root through retry/reopen; provider failures are loud; local/configured-RustFS history sweeps are advisory. No schema v9 or product surface is active | | B2b | candidate managed-reclamation retention profile | Inactive. Requires the Lance-owned durable inspect/plan/execute + receipt, post-success fencing, bounded checkpoint/inventory/accounting, local/RustFS enforced-bound validation, and the profile-specific crash matrix (§4.5.2/§12.6). Passing it alone activates no product surface | | B2-common | explicit enrollment, compare-and-chain token/attribution, revision-fenced lifecycle/correction, schema v9/config-v3/recovery-v12, SDK, HTTP, CLI, Cedar, and OpenAPI | Inactive. Public activation requires the common crash/no-delete/provider-failure/cross-version, authorization, cancellation/shutdown, API-compatibility, and product-parity gates in §12.6. `GraphHistoryBudget` belongs only to a future bounded/managed profile, not selected retain-all | | C | restart-stable reject-row identity, atomic dead letter, richer status, and evidence-backed configurable bounds | reject crash matrix; reject-retention proof; backpressure and RSS/latency evidence |