test: close RFC-026 B2a retain-all gate

.github/workflows/ci.yml

@@ -88,8 +88,9 @@ jobs:
               .github/workflows/ci.yml|Cargo.toml|Cargo.lock|crates/*/Cargo.toml) run_rustfs_ci=true ;;
               crates/omnigraph/src/storage.rs) run_rustfs_ci=true ;;
               crates/omnigraph/src/db/manifest.rs|crates/omnigraph/src/db/manifest/*) run_rustfs_ci=true ;;
-              crates/omnigraph/tests/s3_storage.rs|crates/omnigraph/tests/write_cost_s3.rs|crates/omnigraph/tests/lance_surface_guards.rs|crates/omnigraph/tests/memwal_enrollment_gate.rs|crates/omnigraph/tests/helpers/*) run_rustfs_ci=true ;;
-              crates/omnigraph/src/table_store.rs|crates/omnigraph/src/instrumentation.rs) run_rustfs_ci=true ;;
+              crates/omnigraph/src/db/omnigraph.rs|crates/omnigraph/src/db/omnigraph/stream_enrollment.rs|crates/omnigraph/src/db/omnigraph/stream_ingest.rs) run_rustfs_ci=true ;;
+              crates/omnigraph/tests/s3_storage.rs|crates/omnigraph/tests/write_cost_s3.rs|crates/omnigraph/tests/lance_surface_guards.rs|crates/omnigraph/tests/memwal_enrollment_gate.rs|crates/omnigraph/tests/memwal_stream.rs|crates/omnigraph/tests/memwal_stream_cost.rs|crates/omnigraph/tests/failpoints.rs|crates/omnigraph/tests/helpers/*) run_rustfs_ci=true ;;
+              crates/omnigraph/src/table_store.rs|crates/omnigraph/src/table_store/mem_wal.rs|crates/omnigraph/src/table_store/mem_wal/*|crates/omnigraph/src/instrumentation.rs) run_rustfs_ci=true ;;
               crates/omnigraph/src/runtime_cache.rs|crates/omnigraph/src/graph_index/*) run_rustfs_ci=true ;;
               crates/omnigraph-cluster/src/store.rs|crates/omnigraph-cluster/src/serve.rs) run_rustfs_ci=true ;;
               crates/omnigraph-cluster/tests/s3_cluster.rs) run_rustfs_ci=true ;;
@@ -211,6 +212,7 @@ jobs:
         # omnigraph/failpoints, so each line rebuilds only its crate.
         run: |
           cargo test --locked -p omnigraph-engine --features failpoints --test failpoints
+          cargo test --locked -p omnigraph-engine --features failpoints --test memwal_stream
           set +e
           output=$(cargo test --locked -p omnigraph-engine --features failpoints --test memwal_stream_cost gate_r0_ -- --nocapture 2>&1); status=$?
           set -e
@@ -243,6 +245,18 @@ jobs:
               || { echo "::error::required negative Gate R0 cell '${test_name}' did not execute as should-panic"; exit 1; }
           done
 
+          set +e
+          output=$(cargo test --locked -p omnigraph-engine --features failpoints --test memwal_stream_cost b2a_ -- --nocapture 2>&1); status=$?
+          set -e
+          echo "$output"
+          [ "$status" -eq 0 ] || exit "$status"
+          echo "$output" | grep -Eq "test result: ok\. 1 passed; 0 failed; 2 ignored" \
+            || { echo "::error::B2a retained-history filter matched the wrong test set"; exit 1; }
+          echo "$output" | grep -Eq "test b2a_retained_history_small_depth_keeps_terms_and_old_roots_separate \.\.\. ok" \
+            || { echo "::error::required B2a retained-history regression cell did not execute normally"; exit 1; }
+          echo "$output" | grep -Fq "B2a retained-history ci depth=" \
+            || { echo "::error::B2a retained-history regression cell produced no evidence"; exit 1; }
+
           # This is a remeasurement tripwire for the current single-exclusive-
           # fold shape, not a runtime allocator limit. Keep it outside the
           # gate_r0_ filter because the subprocess owner has a stable older name.
@@ -512,13 +526,29 @@ jobs:
         # grep guards against the filter going vacuous (cargo passes
         # with 0 tests matched) if those tests are ever renamed.
         run: |
+          set +e
           output=$(cargo test --locked -p omnigraph-engine --features failpoints --test failpoints s3_ -- --nocapture 2>&1); status=$?
+          set -e
           echo "$output"
           [ "$status" -eq 0 ] || exit "$status"
           echo "$output" | grep -Eq "test result: ok\. [1-9][0-9]* passed" \
             || { echo "::error::filter 's3_' matched no tests — vacuous pass"; exit 1; }
 
+          set +e
+          output=$(cargo test --locked -p omnigraph-engine --features failpoints --test memwal_stream s3_provider_shard_manifest_failure_retains_unreferenced_generation -- --exact --nocapture 2>&1); status=$?
+          set -e
+          echo "$output"
+          [ "$status" -eq 0 ] || exit "$status"
+          echo "$output" | grep -Fq "SKIP s3_provider_shard_manifest_failure_retains_unreferenced_generation" \
+            && { echo "::error::RustFS B2a provider-failure cell skipped — vacuous pass"; exit 1; }
+          echo "$output" | grep -Eq "test result: ok\. 1 passed" \
+            || { echo "::error::RustFS B2a provider-failure filter matched no test — vacuous pass"; exit 1; }
+          echo "$output" | grep -Eq "test s3_provider_shard_manifest_failure_retains_unreferenced_generation \.\.\. ok" \
+            || { echo "::error::RustFS B2a provider-failure cell did not execute normally"; exit 1; }
+
+          set +e
           output=$(cargo test --locked -p omnigraph-engine --features failpoints --test memwal_stream_cost gate_r0_widest_generation_closes_and_records_retain_all_growth_on_configured_rustfs -- --exact --nocapture 2>&1); status=$?
+          set -e
           echo "$output"
           [ "$status" -eq 0 ] || exit "$status"
           echo "$output" | grep -Eq "test result: ok\. 1 passed" \

AGENTS.md

@@ -33,7 +33,7 @@ OmniGraph is a typed property-graph engine built as a coordination layer over ma
 - **Multi-modal querying**: vector ANN (`nearest`), full-text (`search`/`fuzzy`/`match_text`/`bm25`), Reciprocal Rank Fusion (`rrf`), and graph traversal (`Expand`, anti-join `not { … }`) in one runtime.
 - **Branches and commits across the whole graph**: Git-style — every successful publish appends to a commit DAG; merges are three-way at the row level.
 - **Atomic per-query writes**: `mutate_as` and `load` accumulate insert/update batches into an in-memory `MutationStaging.pending` per touched table. Strict insert and upsert both route through the sealed exact-`id`, filter-bearing adapter; bare Lance Append is test-only. Their RFC-022 adapter resolves or rejects relevant recovery intents before base capture, captures `(native branch id, exact graph_head, schema identity)`, then rechecks recovery and authority under schema → branch → table gates. Mutation/Load keeps one keyed transaction per table and rejects more than 8,192 rows or 32 MiB before recovery arm; mutation update scans stream into the remaining table budget after pending-key shadowing, with blob sizes checked before payload reads. It then arms an identity-bearing recovery-v9 sidecar containing exact Lance transaction identities + pre-minted lineage, commits each table with zero transparent conflict retries, confirms the achieved effects, and publishes under the same token. Unrelated retryable pre-effect authority movement may fully reprepare without changing logical mode. A proven effect-free strict conflict returns `KeyConflict` only after a fresh exact-ID probe; no exact match triggers bounded full strict-mode reprepare, while an effect-free upsert conflict also fully reprepares. Any earlier effect or ambiguity returns `RecoveryRequired` with the sidecar retained. Strict read-set conflicts return `ReadSetChanged`. Deletes stage through the same path. D₂ at parse time remains the constructive (insert/update) XOR destructive (delete) boundary.
-- **RFC-026 private streaming core**: Phase A's main-only adapter enrolls one exact-`id` table into one empty unsharded Lance MemWAL under recovery-v10, then publishes its physical binding and `OPEN` lifecycle. Phase B1 adds a root-singleflight worker for one no-roll generation (8,192 rows / 32 MiB), watcher success plus the same writer's post-durability `check_fenced()` before clean acknowledgement, conservative replay/fold-only recovery, and one recovery-v11 strict exact-`id` fold whose sole `__manifest` CAS advances the table pointer, lifecycle witness/epoch floor, and graph lineage together. Cheap raw bounds and exact post-tombstone validation run before recovery; put ownership follows charge → shared admission → same-key queue → worker mode; cold replay installs exact fold-only accounting; exclusive admission spans seal, drain proof, table effect, and publication. Fold now charges the scanner's logical slices against the same 32-MiB generation cap and copies each scanner emission into dense owned arrays before retaining it, so the 8,192-row high-entropy near-cap closure cell folds and publishes successfully; its isolated fold RSS delta was 284,934,144 bytes (about 272 MiB), below the 384-MiB remeasurement tripwire. Post-invocation ambiguity is `AckUnknown`; unresolved recovery blocks progress. The topology remains main-only, unsharded, one resident writer and one live writer process, with no fresh reads or generation GC. The selected next profile is **unbounded retain-all**: OmniGraph imposes no retained-byte, object-count, file-count, or history quota and never deletes raw `_mem_wal` paths. Provider exhaustion is a loud storage failure and may halt admission, fold, or recovery progress; it never permits silent loss or publication of partial state. Row count, Arrow memory, deadlines, retries, and ambiguous outcomes remain bounded. RC.1's missing cross-open materialization-attempt receipt and complete physical-output envelope are therefore not blockers for this profile; they remain relevant to the future B2b managed-reclamation design. Compare-and-chain tokens, trusted attribution, lifecycle receipts, correction, authorization, and product parity remain required and inactive. The core remains crate-private behind one feature-gated, doc-hidden test seam; no schema-v9, SDK, CLI, HTTP, Cedar, or OpenAPI streaming contract ships from this result.
+- **RFC-026 private streaming core**: Phase A's main-only adapter enrolls one exact-`id` table into one empty unsharded Lance MemWAL under recovery-v10, then publishes its physical binding and `OPEN` lifecycle. Phase B1 adds a root-singleflight worker for one no-roll generation (8,192 rows / 32 MiB), watcher success plus the same writer's post-durability `check_fenced()` before clean acknowledgement, conservative replay/fold-only recovery, and one recovery-v11 strict exact-`id` fold whose sole `__manifest` CAS advances the table pointer, lifecycle witness/epoch floor, and graph lineage together. Cheap raw bounds and exact post-tombstone validation run before recovery; put ownership follows charge → shared admission → same-key queue → worker mode; cold replay installs exact fold-only accounting; exclusive admission spans seal, drain proof, table effect, and publication. Fold now charges the scanner's logical slices against the same 32-MiB generation cap and copies each scanner emission into dense owned arrays before retaining it, so the 8,192-row high-entropy near-cap closure cell folds and publishes successfully; its isolated fold RSS delta was 284,934,144 bytes (about 272 MiB), below the 384-MiB remeasurement tripwire. Post-invocation ambiguity is `AckUnknown`; unresolved recovery blocks progress. The topology remains main-only, unsharded, one resident writer and one live writer process, with no fresh reads or generation GC. The private **B2a unbounded retain-all gate is implemented**: OmniGraph imposes no retained-byte, object-count, file-count, or history quota and never deletes a canonical durable `_mem_wal` object. Lance may clean only its losing `.binpb.tmp.<uuid>` atomic-CAS staging. Complete and partial unreferenced generation residue stays non-authoritative and is never descended into, read, mutated, adopted, or deleted through recovery/reopen; local and configured-RustFS provider-failure cells prove loud typed outcomes. The 1/8/32/128 history instrument keeps warm-ack operation count flat while exposing growing serialized authority and combined fold/history cost; its LIST totals, wall time, and RSS are advisory, not quotas or SLOs. Provider exhaustion may halt admission, fold, or recovery progress; it never permits silent loss or publication of partial state. Row count, Arrow memory, deadlines, retries, and ambiguous outcomes remain bounded. RC.1's missing cross-open materialization-attempt receipt and complete physical-output envelope are therefore not blockers for this profile; they remain relevant to the future B2b managed-reclamation design. Compare-and-chain tokens, trusted attribution, lifecycle receipts, correction, authorization, and product parity remain required and inactive. The core remains crate-private behind one feature-gated, doc-hidden test seam; no schema-v9, SDK, CLI, HTTP, Cedar, or OpenAPI streaming contract ships from this result.
 - **HTTP server**: Axum + utoipa OpenAPI, bearer auth (SHA-256 hashed, optional AWS Secrets Manager). Cedar policy enforcement is engine-wide — every `_as` writer calls `Omnigraph::enforce(action, scope, actor)`, so HTTP, CLI, and embedded SDK consumers all hit the same gate. **Cluster-only boot** (RFC-011): the server always boots from a cluster directory (`--cluster <dir | s3://…>`, RFC-005) and serves N graphs (N ≥ 1) under multi-graph routes (`/graphs/{graph_id}/...` + read-only `GET /graphs` enumeration); there are no single-graph flat routes and no positional-URI boot. Per-graph + server-level Cedar policies. Runtime add/remove (`POST /graphs`, `DELETE /graphs/{id}`) is not exposed — operators run `cluster apply` and restart.
 - **CLI** with two-surface config (RFC-007/008): the team-owned cluster directory (`cluster.yaml`) plus the per-operator `~/.omnigraph/config.yaml` (servers, clusters, credentials, actor, profiles, aliases, defaults). Graphs are addressed via `--store`/`--server`/`--cluster`/`--profile`/operator defaults (RFC-011). Multi-format output (json/jsonl/csv/kv/table).
 
@@ -280,7 +280,7 @@ omnigraph policy explain --cluster ./company-brain --graph knowledge --actor act
 | Cleanup (`cleanup_old_versions`) | ✅ | `omnigraph cleanup` derives requested `--keep` / `--older-than` cutoffs from each table's available versions; Lance refs plus OmniGraph's live-lazy-branch and recovery floors may retain additional versions. It fails closed on unopenable pins, recovery intent, or uncovered main-table HEAD drift |
 | BTREE / inverted (FTS) / vector indexes | ✅ | `@index`/`@key` declares intent; the physical index is derived state that never fails a logical op. Built per column through one chokepoint (`build_indices_on_dataset_for_catalog`, type-dispatched by `node_prop_index_kind`: enum + orderable scalar → BTREE, free-text String → FTS, Vector → vector); idempotent; lazy across branches. **Schema apply and mutation/load build no indexes inline**: they publish only logical data/schema effects, leaving physical intent pending. `ensure_indices` first runs the roll-forward-only recovery barrier before base capture or planning, then materializes every declared-but-missing artifact for one table through one staged mixed CreateIndex transaction under its identity-bearing recovery-v9 authority/lineage/delta envelope; it continues to report untrainable Vector columns as pending. `Armed` is rollback-only, `EffectsConfirmed` rolls forward only while captured authority holds, and first-touch refs carry exact identity. `optimize` separately restores fragment coverage through its bounded identity-bearing v9 maintenance envelope. |
 | Strict insert / upsert ingestion | ✅ transaction conflict filters + uncommitted fragment staging | Internal schema v6 introduced the explicit logical mode and v8 preserves it. General strict insert and upsert use the sealed exact-`id`, forced-v2 MergeInsert adapter; strict insert exact-probes its pinned parent before minting `omnigraph.insert_absence=v1`, while an all-new upsert may mint the same optional certificate only from its completed effect statistics. Mutation/Load remains one transaction per table, capped before arm at 8,192 rows / 32 MiB. BranchMerge's proven all-new route accepts only a complete certificate chain plus final source/target native-incarnation checks; it stages bounded fragments with `InsertBuilder`, commits them as exact-`id` filtered `Update` transactions, and performs zero target preflights, target merge joins, or committed Appends. Missing or malformed proof falls back to the general ordered diff. Raw Lance writers are outside the supported graph-writer topology, and the certificate is an internal, non-cryptographic capability. The final five-pair production gate passed at 10K (3.875× median; 24,297,472-byte max paired RSS overhead) and 100K (~3.886×; 32,604,160 bytes). |
-| MemWAL streaming foundation | ✅ MemWAL system index, durable WAL generations, epoch-fenced shard writer | Internal schema v8 carries exact identity-keyed physical binding, current-HEAD witness, `OPEN`/`DRAINING`/`SEALED` state, per-shard epoch floor, and stream-config v2. Phase A's recovery-v10 adapter enrolls one empty unsharded shard. Private B1 adds one root-singleflight, no-roll generation capped at 8,192 rows / 32 MiB; watcher success plus same-writer post-durability epoch check; detached ownership; fold-only replay; seal/drain proof; and recovery-v11 exact-transaction fold followed by one graph-visibility CAS. The fold charges logical slices and densifies each scanner emission, so the deterministic 8,192-row/near-32-MiB high-entropy generation now closes; the measured isolated fold RSS delta is 284,934,144 bytes (about 272 MiB), below the 384-MiB remeasurement tripwire. It performs no generation GC or fresh reads. The selected unbounded retain-all profile has no OmniGraph byte/object/file/history quota and no raw `_mem_wal` deletion; provider exhaustion fails loudly and may stop progress. Row/logical-Arrow admission, deadline, retry, and ambiguity bounds remain. B2b retains the future Lance-owned reclamation design. Common schema-v9/config-v3/state-v2/recovery-v12 enrollment/token/attribution/lifecycle/correction contracts remain inactive. There is no public stream schema, enrollment, put/fold/status API, Cedar action, CLI, HTTP route, or OpenAPI contract. |
+| MemWAL streaming foundation | ✅ MemWAL system index, durable WAL generations, epoch-fenced shard writer | Internal schema v8 carries exact identity-keyed physical binding, current-HEAD witness, `OPEN`/`DRAINING`/`SEALED` state, per-shard epoch floor, and stream-config v2. Phase A's recovery-v10 adapter enrolls one empty unsharded shard. Private B1 adds one root-singleflight, no-roll generation capped at 8,192 rows / 32 MiB; watcher success plus same-writer post-durability epoch check; detached ownership; fold-only replay; seal/drain proof; and recovery-v11 exact-transaction fold followed by one graph-visibility CAS. The fold charges logical slices and densifies each scanner emission, so the deterministic 8,192-row/near-32-MiB high-entropy generation now closes; the measured isolated fold RSS delta is 284,934,144 bytes (about 272 MiB), below the 384-MiB remeasurement tripwire. It performs no generation GC or fresh reads. The private B2a gate implements unbounded retain-all: no OmniGraph byte/object/file/history quota; no canonical durable `_mem_wal` deletion; typed provider failure; and complete/partial orphan residue retained, non-authoritative, and untouched below its root through retry/reopen. Lance-owned losing manifest temp-file cleanup is allowed. The 1/8/32/128 local/RustFS instrument is advisory and exposes growing authority/history cost without creating a threshold. Row/logical-Arrow admission, deadline, retry, and ambiguity bounds remain. B2b retains the future Lance-owned reclamation design. Common schema-v9/config-v3/state-v2/recovery-v12 enrollment/token/attribution/lifecycle/correction contracts remain inactive. There is no public stream schema, enrollment, put/fold/status API, Cedar action, CLI, HTTP route, or OpenAPI contract. |
 | Vector search | ✅ | `nearest()` query op; embedding pipeline (Gemini / OpenAI clients); `@embed` in schema |
 | Full-text search | ✅ | `search/fuzzy/match_text/bm25` query ops |
 | Hybrid ranking | — | `rrf(...)` Reciprocal Rank Fusion in one runtime |