feat(netflow-db): add coordinated active-source subsets

[flamboh]

Note

🤖 Fable 5 on behalf of Oliver

Problem

Building multiple user-traffic subsets required a separate full traversal and decode for each output, duplicating nfcapd discovery, the daily activity scan, and capture decoding even when the datasets shared the same physical members and time window.

Solution

• pipeline --dataset is repeatable for compatible daily-active subset products. Coordination validates compatibility once (shared root, source layout, timezone, configuration, MAAD, nfdump identity) and returns a witness the executor requires, so unvalidated pipelines cannot run.
• One registry snapshot, one capture-tree discovery, one daily-activity scan for all prefixes, and one decode per capture, fanned into per-product aggregation. Coordinated mode drives the same per-day publication loop as a single-output run (single output is the N=1 case) — there is exactly one day-loop implementation.
• Daily-active selection contract: IPv4 source in the configured prefix, TCP or UDP, source port above 1023, then at least 3 flows, 20 packets, and 2000 bytes across each unique physical member union.
• Each database is bound to its selection, source membership, result configuration, and pinned nfdump identity. Each day publishes in a single immediate transaction per output; resume uses per-day completion markers keyed by the product fingerprint.
• Output, capture-root, and member paths are canonicalized and rejected on overlap (including SQLite sidecar and lock paths) before any output mutation.
• Registry entries use strict deserialization: unknown fields are load errors instead of silently degrading the selection.

How to review

Required setup: a native nfcapd tree with complete five-minute captures for at least one local day; two registry entries sharing root and physical member layout with distinct db_path values and daily_active_sources selections using different ip_prefix values; the repository-pinned nfdump binary.

netflow-db pipeline \
  --datasets datasets.json \
  --dataset campus-a \
  --dataset campus-b \
  --nfdump target/nfdump/libexec/nfdump \
  --start-date 2025-06-01 \
  --end-date 2025-06-01 \
  --require-complete

Expected results:

• Both outputs contain complete five-minute coverage and MAAD rows; discovery, eligibility, and decoding run once for the pair.
• Repeating the command publishes zero new five-minute buckets.
• Each output passes verify --require-data --require-maad-data --require-processed --require-rollup-parity --require-no-raw-ip.

Edge cases and decisions

• An incomplete physical day is not published to either coordinated output; --require-complete reports the missing captures.
• Duplicate dataset IDs, incompatible roots/layouts/timezones, and output path collisions fail before any database is created.
• Outputs commit sequentially; a crash between commits can leave siblings divergent by at most the in-flight day, and the next run self-heals via completion markers. This is documented in docs/code/pipeline-contract.md.
• A day with a missing or mismatched completion marker is rebuilt whole; there is no partial-repair path.
• DST-day completeness handling remains separate work in Mark DST fall-back days incomplete when repeated-hour capture evidence is absent #104.

Verification

• bun run format, bun run lint, bun run typecheck, bun run test:db (131 unit tests plus CLI, comparison, MAAD conformance, and singularity integration tests).
• The two tests that previously failed in CI now pass with nfdump removed from PATH; they use self-contained executable fixtures.
• Manual one-day coordinated cold run, no-op rerun, and full verifier pass on Barbera predate the trim; the restructure preserves the same observable contract and is covered by the tests above.

---

Initial implementation by GPT-5.6 Sol (Codex/T3). Restructured by GPT-5.6 Sol under direction of Fable 5 in Claude Code; reviewed with gpt-5.6-luna explorers.