diff --git a/.github/workflows/publish-weights-cvc.yml b/.github/workflows/publish-weights-cvc.yml index e7f9d96..723c4c0 100644 --- a/.github/workflows/publish-weights-cvc.yml +++ b/.github/workflows/publish-weights-cvc.yml @@ -1,15 +1,15 @@ name: publish-weights-cvc # Publish the grl-snam-weights data package (base GRL-SNAM nav coefficients — the -# research tier, NO RF-comm forces) to the PUBLIC `cvc` org on cvcpkg.org, using +# geometry-only research tier) to the PUBLIC `cvc` org on cvcpkg.org, using # this repo's CVCPKG_TOKEN (the cvc-org publisher) — the same token that ships the # grl-snam-cpXXX columns. Kept separate from publish-cvcpkg.yml (the interpreter # matrix) because the weights are a single noarch data package with no py column. # # Org packages belong in their own repo's recipes, never in libcvc-deps: this -# recipe lives here, in cvcpkg/recipes/grl-snam-weights. The DBG comm-aware weights -# are the SEPARATE private cvc-dbg-weights on the utdbg org — the research/ -# development seam is the org boundary; this workflow only ever targets --org cvc. +# recipe lives here, in cvcpkg/recipes/grl-snam-weights. Weights that a downstream +# application trains with its own extra force terms are published by that project +# to its own org; this workflow only ever targets --org cvc. # # workflow_dispatch: dry_run=true (default) packs only; dry_run=false publishes. on: diff --git a/README.md b/README.md index 534b6e7..006f57d 100644 --- a/README.md +++ b/README.md @@ -146,7 +146,7 @@ GRL-SNAM (not the other way round) and keep their dataset-specific code and mode of this general library. Such an extension reuses exactly the pieces shown above — the differentiable surrogate, the coefficient network, and the `pycvc`/`pycvc_gl` scene bindings — and versions its own additions in its own repository. This library stays -general; the extensions stay separate and private. +general; the extensions stay separate. ## Repository Structure diff --git a/cvcpkg/recipes/grl-snam-weights/payload/PROVENANCE.md b/cvcpkg/recipes/grl-snam-weights/payload/PROVENANCE.md index de6dd9b..1323b68 100644 --- a/cvcpkg/recipes/grl-snam-weights/payload/PROVENANCE.md +++ b/cvcpkg/recipes/grl-snam-weights/payload/PROVENANCE.md @@ -1,6 +1,6 @@ # grl-snam-weights — base GRL-SNAM nav coefficient provenance -Base GRL-SNAM navigation coefficients (no RF-comm forces) for the differentiable +Base GRL-SNAM navigation coefficients (geometry only) for the differentiable SDF navigator's `CoefMLP` — predicts `(alpha, beta, gamma)`. - `coef_sdf.cvcnav` — native `CVNV` blob for the C++ `cvc::nav` forward. @@ -9,7 +9,7 @@ SDF navigator's `CoefMLP` — predicts `(alpha, beta, gamma)`. ## How it was trained (v1.0.0) - **Pipeline:** `grl-snam train ` (self-supervised SDF-coefficient - training — geometry only, NO comm-force term) → `grl_snam.tools.coef_export` + training — geometry only, NO external-force term) → `grl_snam.tools.coef_export` (checkpoint → `CVNV`). - **Data:** the `austin_south` navigation SDF (`nav_sdf.npz`, 1024×1024 φ field over the public Austin geometry). @@ -18,5 +18,8 @@ SDF navigator's `CoefMLP` — predicts `(alpha, beta, gamma)`. - **Character:** an initial, deliberately short proof-of-pipeline run — enough that the coefficients have moved and the weights are usable, NOT a full training campaign. Regenerate from a longer run for production accuracy. +- **Metadata:** the checkpoint's `meta` and the `.cvcnav` provenance trailer + name the training SDF as `austin_south/nav_sdf.npz` (scene-relative). Since + revision 2; the weights are the same as in revision 1. Bump `cvc_revision` in `recipe.yaml` whenever these bytes change. diff --git a/cvcpkg/recipes/grl-snam-weights/payload/coef_sdf.cvcnav b/cvcpkg/recipes/grl-snam-weights/payload/coef_sdf.cvcnav index 0db7586..04ed363 100644 Binary files a/cvcpkg/recipes/grl-snam-weights/payload/coef_sdf.cvcnav and b/cvcpkg/recipes/grl-snam-weights/payload/coef_sdf.cvcnav differ diff --git a/cvcpkg/recipes/grl-snam-weights/payload/coef_sdf.pt b/cvcpkg/recipes/grl-snam-weights/payload/coef_sdf.pt index e53e1cb..4b22264 100644 Binary files a/cvcpkg/recipes/grl-snam-weights/payload/coef_sdf.pt and b/cvcpkg/recipes/grl-snam-weights/payload/coef_sdf.pt differ diff --git a/cvcpkg/recipes/grl-snam-weights/recipe.yaml b/cvcpkg/recipes/grl-snam-weights/recipe.yaml index f4d26b5..a590518 100644 --- a/cvcpkg/recipes/grl-snam-weights/recipe.yaml +++ b/cvcpkg/recipes/grl-snam-weights/recipe.yaml @@ -6,7 +6,7 @@ recipe: # BUMP cvc_revision on ANY change to the staged bytes (a new checkpoint/blob, or # an edited PROVENANCE.md that ships under share/) — a same-revision re-push # silently no-ops against the already-published artifact. - cvc_revision: 1 + cvc_revision: 2 kind: data maintainer: "cvcpkg group" maintainer_email: "info@cvcpkg.org" @@ -14,7 +14,7 @@ recipe: license: MIT tags: [data, grl-snam, weights, nav] description: >- - Pretrained base GRL-SNAM navigation coefficients (no RF-comm forces): the + Pretrained base GRL-SNAM navigation coefficients (geometry only): the self-supervised SDF CoefMLP that predicts (alpha, beta, gamma) for the differentiable SDF navigator. Ships BOTH the native CVNV blob (coef_sdf.cvcnav, for the C++ cvc::nav forward) and the torch checkpoint diff --git a/docs/CVCNAV_CPP_PORT_ROADMAP.md b/docs/CVCNAV_CPP_PORT_ROADMAP.md index 3d9dfd6..f0c0fc9 100644 --- a/docs/CVCNAV_CPP_PORT_ROADMAP.md +++ b/docs/CVCNAV_CPP_PORT_ROADMAP.md @@ -62,7 +62,7 @@ Boundary is drawn **exactly at the bilinear sample**. Everything upstream stays Namespace `cvc::nav`, raw-pointer SoA cores, `int num_threads` on every batch, all new TUs compiled under the existing discipline (**no `-ffast-math`, no `-ffp-contract=fast`**, float32 interior to track torch f32). **Name collision to avoid:** `cvc::nav::sdf_field` already exists in `grid_nav.h` (the *built* field). The sampler stack is named **`field_stack`**. -Files (all under `/home/joe/src/cvc/wt-libcvc-nav/`): +Files (all under the libcvc repository root): ``` inc/cvc/nav/detail/parallel.h NEW (extract parallel_for from grid_nav.cpp) @@ -270,14 +270,14 @@ arch descriptor (hashed): pack("bytes` walks the **live** `nn.Module` (activations discovered from `model.net`, `bias` buffer present), plus `serialize_checkpoint(pt_path)` that rebuilds a `CoefMLP` from a saved state_dict. `write_coef_mlp(model, path)`; CLI `python -m grl_snam.coef_export coef_sdf.pt coef_mlp.cvcnav`. Wire `write_coef_mlp` into `grl_snam/tools/train.py` and `grl_snam/tools/pipeline.py` so every checkpoint ships its deployable twin. +- **Exporter** `grl_snam/coef_export.py` (this repo): `serialize_coef_mlp(model)->bytes` walks the **live** `nn.Module` (activations discovered from `model.net`, `bias` buffer present), plus `serialize_checkpoint(pt_path)` that rebuilds a `CoefMLP` from a saved state_dict. `write_coef_mlp(model, path)`; CLI `python -m grl_snam.coef_export coef_sdf.pt coef_mlp.cvcnav`. Wire `write_coef_mlp` into `grl_snam/tools/train.py` and `grl_snam/tools/pipeline.py` so every checkpoint ships its deployable twin. - **In-memory path is the parity substrate:** the same `bytes` feed torch (its source) and `pycvc.NavCoefMLP(blob)` — no file round-trip in the tolerance test. --- ## 5. Python-green seam + migration order -**pycvc surface** (append to `/home/joe/src/cvc/wt-libcvc-nav/bindings/pycvc/pycvc_nav.i`, following the existing writable-borrow discipline — validate exact dtype/shape/contig, never coerce-copy an in-place buffer, `Py_BEGIN_ALLOW_THREADS` across compute): +**pycvc surface** (append to libcvc's `bindings/pycvc/pycvc_nav.i`, following the existing writable-borrow discipline — validate exact dtype/shape/contig, never coerce-copy an in-place buffer, `Py_BEGIN_ALLOW_THREADS` across compute): - Fine-grained (parity + optional acceleration): `nav_sdf_sample(field[M,3,H,W]f32, o[N,2]f32, plane[N]i32|None, mnx..S) -> (phi[N], nrm[N,2])`; `NavCoefMLP(bytes)` / `nav_coef_mlp_load(path)` with `.forward(feat[N,5])->(N,3)`; `nav_drive_step(field, plane|None, , mlp, veh scalars, integ, reach_tol, mnx..S, num_threads) -> None`. - Object surface (pure-C++ twin + full delegation): opaque `nav_sim_world_create(...)`, `_step`, `_snapshot(->dict of fresh numpy)`, `_retarget`, `_add_obstacle`, `_free`. @@ -355,15 +355,15 @@ def drive_enabled(): - **FSM source of truth:** port `swarm.py._plan_carrot` (ring-buffer form), **not** `nav.py._plan_carrot` (list-pop form). They compute `moved` against different history references; `swarm.py` is the vectorized reference the tests pin, and its integer state is exact. - **`allow_reverse`:** `bicycle_rollout`'s own default is `False`, but the deployment path (`SdfNavigator.VEHICLE_DEFAULTS`, inherited by `Swarm.veh`) is `True`. C++ `veh_params` default must be `true` (§2.3), and the golden must exercise the reverse branch. - **Weight bias:** keep it **raw** in the file (D2), not pre-folded (D5) — `log(expm1)` is a load-time constant folded once in f32, and a raw mirror of the state_dict is auditable. D1's fixed-4742-float blob is rejected in favor of D2's generic+arch_hash layout. -- **CMake gtest wiring:** each new `*_test` needs **all five** `nav_test` entries in `/home/joe/src/cvc/wt-libcvc-nav/src/cvc/tests/CMakeLists.txt` — `add_executable` (L100), the `TEST_TARGETS`/list membership (L168), `target_link_libraries` (L320), `target_compile_features(... cxx_std_17)` (L850), `gtest_discover_tests` (L1014) — or the target is silently omitted (the "four-entry" trap in MEMORY). +- **CMake gtest wiring:** each new `*_test` needs **all five** `nav_test` entries in libcvc's `src/cvc/tests/CMakeLists.txt` — `add_executable` (L100), the `TEST_TARGETS`/list membership (L168), `target_link_libraries` (L320), `target_compile_features(... cxx_std_17)` (L850), `gtest_discover_tests` (L1014) — or the target is silently omitted (the "four-entry" trap in MEMORY). **Open questions needing a decision (do not block P0–P5):** - **Deployment belief mode:** does a C++ renderer/game-engine host need clustered/private belief, or is **shared (M=1)** the whole deployment path? Shared is the thousands-of-agents target; if only shared, `map_id` is all-zeros and the M>1 COW/rebuild cost is untested-in-anger. Confirm before P6 sizing. - **`nsub` at deploy** (`meta.get("nsub",1)`): if always 1, the golden and gtests can pin `nsub=1`; if >1, the multi-substep transcendental accumulation must be in the golden. - **Whole-drive tolerance/horizon sign-off:** the proposed short-horizon `1e-3` normalized, `<0.5%` flip budget are from the existing `Squad(batched_drive)` 5e-3 tier and need empirical confirmation across all six stories and multiple seeds in P3/P6 — this is the single number that gates P6, and it is the top fidelity risk. -- **Canonical `.cvcnav` home + provenance policy:** where the blessed weights live (libcvc test-data vs pycvc-published vs per-deployment bundle) (kept out of public repos where required); and whether the provenance trailer is required for an audit trail. +- **Canonical `.cvcnav` home + provenance policy:** where the blessed weights live (libcvc test-data vs pycvc-published vs per-deployment bundle); and whether the provenance trailer is required for an audit trail. -Files to create are listed in §2; the two files to edit are `/home/joe/src/cvc/wt-libcvc-nav/src/cvc/CMakeLists.txt` (add headers ~L88, sources ~L171 next to `nav/grid_nav.cpp`) and `/home/joe/src/cvc/wt-libcvc-nav/bindings/pycvc/pycvc_nav.i` (append the marshalling), plus the new grl-snam `coef_export.py`, `nav_native.py` additions, and the parity tests under `/home/joe/src/cvc/wt-grl-snam-nav/tests/`. +Files to create are listed in §2; the two files to edit are libcvc's `src/cvc/CMakeLists.txt` (add headers ~L88, sources ~L171 next to `nav/grid_nav.cpp`) and `bindings/pycvc/pycvc_nav.i` (append the marshalling), plus the new grl-snam `coef_export.py`, `nav_native.py` additions, and the parity tests under this repo's `tests/`. --- ## 10. Decisions on the §9 open questions diff --git a/docs/CVCNAV_CUDA_ASSESSMENT.md b/docs/CVCNAV_CUDA_ASSESSMENT.md index 8278227..bfae43e 100644 --- a/docs/CVCNAV_CUDA_ASSESSMENT.md +++ b/docs/CVCNAV_CUDA_ASSESSMENT.md @@ -12,12 +12,12 @@ I have verified every load-bearing claim against the actual code. Synthesis foll ## Verified ground truth (I read the code, not just the analyses) -- **CMake CUDA wiring** (`wt-libcvc-nav/CMakeLists.txt`): `CVC_ENABLE_CUDA` default ON, auto-disables without nvcc (L5, L25, L80-81); arch list `50 60 70 75 80` set **before** `enable_language(CUDA)` (L45, L75, L78 — the documented sm_52 fix); `+90` when nvcc≥11.8, `+100 120` when ≥12.8 (L63-67) — **local nvcc 12.0 compiles 50/60/70/75/80/90**; `CUDA_RESOLVE_DEVICE_SYMBOLS ON` so device-link emits SASS and **PTX is stripped — the arch list is the entire compat surface** (L54-57, SetupCUDA.cmake L29). `CVC_USING_CUDA` is a PUBLIC define (`src/cvc/CMakeLists.txt` L815); `voxels_kernels.cu` is appended to `SOURCE_FILES` only under `if(CVC_ENABLE_CUDA)` (L611-612) — the exact slot a nav `.cu` drops into. +- **CMake CUDA wiring** (libcvc `CMakeLists.txt`): `CVC_ENABLE_CUDA` default ON, auto-disables without nvcc (L5, L25, L80-81); arch list `50 60 70 75 80` set **before** `enable_language(CUDA)` (L45, L75, L78 — the documented sm_52 fix); `+90` when nvcc≥11.8, `+100 120` when ≥12.8 (L63-67) — **local nvcc 12.0 compiles 50/60/70/75/80/90**; `CUDA_RESOLVE_DEVICE_SYMBOLS ON` so device-link emits SASS and **PTX is stripped — the arch list is the entire compat surface** (L54-57, SetupCUDA.cmake L29). `CVC_USING_CUDA` is a PUBLIC define (`src/cvc/CMakeLists.txt` L815); `voxels_kernels.cu` is appended to `SOURCE_FILES` only under `if(CVC_ENABLE_CUDA)` (L611-612) — the exact slot a nav `.cu` drops into. - **The `--use_fast_math` bug is REAL** (`CMake/SetupCUDA.cmake` L88-96): it is attached **target-wide** to `cvc` for `$` × `$`, not per-source. Any future nav parity `.cu` compiled into `cvc` in Release inherits `ftz=1`, `prec-div=0`, `prec-sqrt=0`, and fast-intrinsic transcendentals — **silently breaking the float-equivalence contract**. This is a hard prerequisite to fix, confirmed by inspection (A3/A4 flagged it; it checks out). - **`sense_batch` structure** (`src/cvc/nav/grid_nav.cpp`): `parallel_for(planes.K, …)` (L593) — threads **across planes**, so shared M=1 ⇒ K=1 ⇒ one worker. The per-cell fold is accumulate-then-clamp reading the clamped prior value (L561-569, "Model C: running delta is f32"), i.e. **order-dependent**; the ray-march reads only `truth` (L497-551), i.e. **order-free**. The split the analyses propose is valid against the actual code. - **PERFORMANCE.md**: batched EDT 0.440 ms/agent vs 9.285 ms single-core = GPU ≈ 21 CPU cores (not a win vs 32); A* GPU 18.4 ms vs CPU 2.84 ms = **6.5× worse**; D2H 0.301 + compute 0.440 = 0.741 ms **loses to 16-core CPU 0.580 ms** ("a GPU EDT with a host consumer never wins at any agent count"); the "~60 agents" crossover is EDT-only/device-resident/private-ish, **not** the shared deployment crossover. - **Recipe** (`cvcpkg/recipes/libcvc-cuda/recipe.yaml`): `provides:[libcvc]`, `requires_capabilities:[cuda]`, `CVC_ENABLE_CUDA=ON`, linux+windows only (no macOS — no CUDA on Apple), self-hosted CUDA runner, `cvc_revision: 2`. Peers exist: `cvcgl-cuda`, `pycvc-cp311-cuda`, `pycvc-gl-cp311-cuda`. **No new recipe needed.** -- **Roadmap fidelity boundary** (`wt-grl-snam-nav/docs/CVCNAV_CPP_PORT_ROADMAP.md`): drawn **exactly at the bilinear sample** — belief→to_occupancy→EDT→field is **BIT**; sample→MLP→rollout is **FLOAT-EQUIV**. Rule 1: only BIT tiers may be a transparent Python default; torch stays the reference/golden forever. §10.1 ships all three belief modes (shared M=1 / clustered / private M=N); §10.2 deploy `nsub=1`, goldens pinned nsub=1. Flags: `GRL_SNAM_NAV_DRIVE` (drive, default `torch`) separate from `GRL_SNAM_NAV_BACKEND` (bit-kernels, default `native`). +- **Roadmap fidelity boundary** (`docs/CVCNAV_CPP_PORT_ROADMAP.md`): drawn **exactly at the bilinear sample** — belief→to_occupancy→EDT→field is **BIT**; sample→MLP→rollout is **FLOAT-EQUIV**. Rule 1: only BIT tiers may be a transparent Python default; torch stays the reference/golden forever. §10.1 ships all three belief modes (shared M=1 / clustered / private M=N); §10.2 deploy `nsub=1`, goldens pinned nsub=1. Flags: `GRL_SNAM_NAV_DRIVE` (drive, default `torch`) separate from `GRL_SNAM_NAV_BACKEND` (bit-kernels, default `native`). ## (1) Per-component GPU-fit table @@ -81,4 +81,4 @@ Speedups are device-resident on sm_80-class; "vs current" = vs today's single-th **The single decision variable:** after Step 1, does any *named* deployment's real N-at-frame-rate on *its own* hardware exceed the agent-parallel CPU ceiling? The GPU crushes today's *broken* single-threaded sense but only ties-to-modestly-beats a *fixed* CPU sense at N~1k; it pulls decisively ahead only as N outgrows core-count saturation. Measure that number before writing a single `.cu`. -Relevant files: `/home/joe/src/cvc/wt-libcvc-nav/CMake/SetupCUDA.cmake` (L88-96, the fast-math fix), `/home/joe/src/cvc/wt-libcvc-nav/CMakeLists.txt` (L42-78, arch list), `/home/joe/src/cvc/wt-libcvc-nav/src/cvc/CMakeLists.txt` (L611-612, L815, wiring slot), `/home/joe/src/cvc/wt-libcvc-nav/src/cvc/nav/grid_nav.cpp` (L497-601, sense split target), `/home/joe/src/cvc/wt-libcvc-nav/cvcpkg/recipes/libcvc-cuda/recipe.yaml`, `/home/joe/src/cvc/wt-grl-snam-nav/docs/CVCNAV_CPP_PORT_ROADMAP.md` (§1 fidelity boundary, §10 belief modes/nsub), `/home/joe/src/cvc/wt-grl-snam-nav/docs/PERFORMANCE.md` (L59-79, CUDA verdict). \ No newline at end of file +Relevant files: libcvc `CMake/SetupCUDA.cmake` (L88-96, the fast-math fix), `CMakeLists.txt` (L42-78, arch list), `src/cvc/CMakeLists.txt` (L611-612, L815, wiring slot), `src/cvc/nav/grid_nav.cpp` (L497-601, sense split target), `cvcpkg/recipes/libcvc-cuda/recipe.yaml`; this repo's `docs/CVCNAV_CPP_PORT_ROADMAP.md` (§1 fidelity boundary, §10 belief modes/nsub), `docs/PERFORMANCE.md` (L59-79, CUDA verdict). \ No newline at end of file diff --git a/docs/CVCNAV_MATERIAL_PORT_ROADMAP.md b/docs/CVCNAV_MATERIAL_PORT_ROADMAP.md index 410ad8a..07ec4bc 100644 --- a/docs/CVCNAV_MATERIAL_PORT_ROADMAP.md +++ b/docs/CVCNAV_MATERIAL_PORT_ROADMAP.md @@ -19,8 +19,8 @@ and hyperparameter. None of them are in this repo. They resolve as `$GRL_SNAM_MATERIAL_FORK/full_code/train_material.py` — a checkout of `github.com/SetasAditya/material-aware-grl-snam`, which `tests/test_material_fork_xcheck.py:22-24` skips against when the env var is -unset. A clone exists at `/home/joe/src/cvc/material-aware-grl-snam`. Set -`GRL_SNAM_MATERIAL_FORK` to it before touching any parity claim; the bit-identity +unset. Clone it and set +`GRL_SNAM_MATERIAL_FORK` to the clone before touching any parity claim; the bit-identity cross-check is otherwise silently green. **Companion C++-side doc:** libcvc `docs/NAV_MATERIAL.md` describes the same diff --git a/docs/NAV_STATS.md b/docs/NAV_STATS.md index b10d199..d7b1a15 100644 --- a/docs/NAV_STATS.md +++ b/docs/NAV_STATS.md @@ -1,9 +1,10 @@ # Navigation statistics & the base-policy scorecard -grl-snam is the **base (RF-free)** owner of the two-layer nav-stats design: per-vehicle + per-episode -navigation telemetry and a corpus scorecard that ranks base-policy checkpoints. It is field-mirrored -by the C++ `cvc::nav::nav_stats` (libcvc `docs/NAV_STATS.md`) — the same schema and the same -hand-computed numbers — and the RF/comms layer (grl_snam_dbg) extends it, joined by vehicle index. +grl-snam is the **base (domain-neutral)** owner of the two-layer nav-stats design: per-vehicle + +per-episode navigation telemetry and a corpus scorecard that ranks base-policy checkpoints. It is +field-mirrored by the C++ `cvc::nav::nav_stats` (libcvc `docs/NAV_STATS.md`) — the same schema and +the same hand-computed numbers. The second layer is an extension seam: a downstream layer that needs +domain-specific metrics keeps them in its own record, joined to this one by vehicle index. ## The schema (`grl_snam.metrics`) @@ -36,14 +37,15 @@ All three collect the **same** base row, so a checkpoint scores identically whic `EpisodeStats.from_nav_stats(vehicles, straights_m=, arrival_times_s=, veh_contacts=, min_sep_m=)` reduces a list of per-vehicle `NavStats` (plus the fields `NavStats` doesn't carry) into one episode record; `aggregate_nav(episodes, checkpoint)` reduces a corpus into a single `NavScorecard` — the -RF-free fitness row (arrival, economy, safety, material). Mirrors C++ `cvc::nav::nav_scorecard`. +domain-neutral fitness row (arrival, economy, safety, material). Mirrors C++ +`cvc::nav::nav_scorecard`. ## Material & terrain-risk buckets The scorecard's material dimension is **discrete**, mirroring C++ `veh_nav_stats`: each vehicle's `NavStats` carries `time_over_material_s[13]` and `dist_over_material_m[13]`, indexed by the shared 13-class palette (`grl_snam.material_palette`, positional with `cvc::nav::kNumMaterials` and the -`cvc::dbg` `MATERIAL_TABLE`). `aggregate_nav` reduces the time buckets into +`cvc::nav` `material_id` enum). `aggregate_nav` reduces the time buckets into `NavScorecard.material_time_share[13]`, and `terrain_risk_share(...)` sums the traversable terrain-risk classes (foliage/soil/water/rock) into the single **"time in terrain-risk areas"** number the eval CLI prints as `risk-time%`. @@ -122,7 +124,7 @@ the floor keeps coverage everywhere. Magnitude scales with `--curriculum-eps`/bi ## Parity `tests/test_scorecard.py` pins the reducer to the SAME hand-computed corpus as libcvc's -`nav_stats_test.cpp` and cvcdbg's `tests/nav_stats_test.cpp` (the shared-schema contract); +`nav_stats_test.cpp` (the shared-schema contract); `tests/test_swarm.py` checks the `Swarm` collector against N serial `SdfNavigator`s to float32; `tests/test_material_buckets.py` hand-computes the material buckets (the time/dist-per-id contract mirrored from `nav_stats_test.cpp`), the id-raster sampler, and the byte-identical no-material default; diff --git a/docs/VEHICLE_REFINEMENTS.md b/docs/VEHICLE_REFINEMENTS.md index a8fc700..9c879bf 100644 --- a/docs/VEHICLE_REFINEMENTS.md +++ b/docs/VEHICLE_REFINEMENTS.md @@ -433,7 +433,7 @@ binding. That is deliberate: a green tick from an absent feature reads as coverage this repo does not have. Before pushing, run the linters CI runs — they are not installable here -(`black`/`ruff` are absent from `dbg-deps` and from the cvcpkg catalog for this +(`black`/`ruff` are absent from the local deps prefix and from the cvcpkg catalog for this platform tuple), so use a throwaway venv: ```bash diff --git a/docs/cvc-nav-and-grl-snam-guide.md b/docs/cvc-nav-and-grl-snam-guide.md index 100fa6c..731e3aa 100644 --- a/docs/cvc-nav-and-grl-snam-guide.md +++ b/docs/cvc-nav-and-grl-snam-guide.md @@ -736,9 +736,8 @@ cvcpkg install grl-snam-weights --prefix ./nav-env > on the free-space rollout surrogate currently *regresses* versus the hand-tuned > `(1,3,4)` basin. Use this package to exercise the pipeline and as a starting > checkpoint; keep the seed unless a longer run beats it on your own reach eval. -> The **RF-comm-aware** DBG policy is a *separate, private* package -> (`cvc-dbg-weights` on the `utdbg` org) — the research/development seam is kept at -> the org boundary, never mixed into this public package. +> A policy that a downstream application trains with its own extra force terms is +> published by that project, never mixed into this package. --- diff --git a/docs/cvc-nav-and-grl-snam-guide.pdf b/docs/cvc-nav-and-grl-snam-guide.pdf index bd55eab..9da24ea 100644 Binary files a/docs/cvc-nav-and-grl-snam-guide.pdf and b/docs/cvc-nav-and-grl-snam-guide.pdf differ diff --git a/docs/cvc-nav-and-grl-snam-quickstart.pdf b/docs/cvc-nav-and-grl-snam-quickstart.pdf index 6fb4fc9..40c5be1 100644 Binary files a/docs/cvc-nav-and-grl-snam-quickstart.pdf and b/docs/cvc-nav-and-grl-snam-quickstart.pdf differ diff --git a/docs/pypi-publishing-roadmap.md b/docs/pypi-publishing-roadmap.md index ccb73ed..5ff486b 100644 --- a/docs/pypi-publishing-roadmap.md +++ b/docs/pypi-publishing-roadmap.md @@ -4,9 +4,8 @@ current Python packaging standards, so it installs with a plain `pip install grl-snam`. -> **Scope.** This covers the public `grl-snam` package only. Any sensitive -> downstream extensions are **not** publicly distributed; they are out of scope -> here and must never be published. Once `grl-snam` is on PyPI, downstreams can +> **Scope.** This covers the public `grl-snam` package only; downstream +> extensions are out of scope here. Once `grl-snam` is on PyPI, downstreams can > depend on it by version instead of a GitHub URL. --- diff --git a/grl_snam/__init__.py b/grl_snam/__init__.py index 5cbbc90..93851eb 100644 --- a/grl_snam/__init__.py +++ b/grl_snam/__init__.py @@ -4,8 +4,8 @@ (`train_coef_energy.py`, `surrogate_robust.py`, `eval_coef_energy.py`, `src/utils/`, `scripts/`, `experiments/`). The flat layout is preserved for backward compatibility with the original research code; this package -exposes a stable, importable API for downstream consumers (e.g. a -private downstream extension). +exposes a stable, importable API for downstream consumers (e.g. an +application-specific extension layer). Typical use:: diff --git a/grl_snam/material_palette.py b/grl_snam/material_palette.py index 628f3f4..80aea9c 100644 --- a/grl_snam/material_palette.py +++ b/grl_snam/material_palette.py @@ -3,20 +3,21 @@ This is the SINGLE source of the palette shared across the stack: it mirrors ``cvc::nav::kNumMaterials`` (libcvc ``inc/cvc/nav/nav_stats.h``) and is positional -with the ``cvc::dbg`` ``MATERIAL_TABLE`` (cvcdbg ``src/cvc/dbg/channel.cpp``), so a -material id means the same class in the C++ collector, the DBG RF layer, and here. -Keep the ORDER byte-for-byte with that table — the ids are the wire format. +with the ``cvc::nav`` ``material_id`` enum (libcvc ``inc/cvc/nav/material_raster.h``), +so a material id means the same class in the C++ collector, in any downstream table +keyed by these ids, and here. Keep the ORDER byte-for-byte with that enum — the ids +are the wire format. The discrete buckets (``time_over_material_s`` / ``dist_over_material_m``) are the parity field with the C++ scorecard. "Time in terrain-risk areas" is DERIVED from them — :func:`terrain_risk_share` sums the traversable outdoor classes that carry mobility risk — rather than stored as a separate schema field, so the scorecard -stays field-for-field with ``cvc::dbg::nav_scorecard``. +stays field-for-field with ``cvc::nav::nav_scorecard``. """ from __future__ import annotations -# Positional — index == material id. MUST match cvc::dbg MATERIAL_TABLE order. +# Positional — index == material id. MUST match the cvc::nav material_id enum order. MATERIALS = ( "reinforced_concrete", # 0 "brick", # 1 diff --git a/grl_snam/nav.py b/grl_snam/nav.py index 18ceb67..07990b1 100644 --- a/grl_snam/nav.py +++ b/grl_snam/nav.py @@ -92,7 +92,7 @@ def __init__( # ``o[B,2] -> [B,2]`` returning an extra acceleration (rollout/normalized # frame) summed alongside F_bar/F_goal each substep. Unlike carrot_bias_fn # (which nudges the pure-pursuit carrot), this enters the force law itself - # — the seam the DBG comm force uses as a genuine force term. ``None`` + # — the seam an application-specific force uses as a genuine force term. ``None`` # (default) is additively inert and bit-for-bit unchanged. self.ext_force_fn = None # Optional material-aware runtime (grl_snam.material.MaterialRuntime). diff --git a/grl_snam/planner.py b/grl_snam/planner.py index 940dd45..aead1a9 100644 --- a/grl_snam/planner.py +++ b/grl_snam/planner.py @@ -97,7 +97,7 @@ def astar(occ: np.ndarray, start, goal, cost: np.ndarray | None = None): parameter existed — the golden traces stay valid. The surcharge is domain-agnostic on purpose: it is "this cell is expensive", - not "this cell is dangerous/jammed/steep". Whoever builds the raster owns + not "this cell is dangerous/slow/steep". Whoever builds the raster owns the meaning. Keep values modest — a surcharge much larger than the map's diameter makes the heuristic wildly inadmissible and A* degenerates toward Dijkstra, exploring the whole grid for a route it was always going to take. diff --git a/grl_snam/scenario.py b/grl_snam/scenario.py index a87e8b8..c3cda3d 100644 --- a/grl_snam/scenario.py +++ b/grl_snam/scenario.py @@ -444,8 +444,8 @@ def _route_cost(self) -> np.ndarray | None: (the documented downstream seam — never clobbered), composed additively with the material cost raster when a MaterialGrid is attached. - route_cost_fn is the seam a downstream package (e.g. an RF-aware - planner) plugs a cost surface into; base GRL-SNAM never interprets it. + route_cost_fn is the seam a downstream package (e.g. an application- + specific planner) plugs a cost surface into; base GRL-SNAM never interprets it. """ cost = self.route_cost_fn() if self.route_cost_fn is not None else None if self._material is not None: diff --git a/grl_snam/scorecard.py b/grl_snam/scorecard.py index aad3f5e..9cc687d 100644 --- a/grl_snam/scorecard.py +++ b/grl_snam/scorecard.py @@ -1,10 +1,10 @@ """Base navigation SCORECARD — aggregate a corpus of episodes into one fitness row. This is the Python twin of the C++ ``cvc::nav::nav_scorecard`` (transfix/libcvc -``inc/cvc/nav/nav_stats.h``): the RF-free nav-fitness row grl-snam uses to rank its -OWN base-policy checkpoints over a scene corpus. The DBG campaign layers an -``rf_scorecard`` on top; the base half here is shared, so a grl-snam training run and -a cvcdbg ``dbg_arrival_check3 --episodes`` run report identical base numbers (the +``inc/cvc/nav/nav_stats.h``): the domain-neutral nav-fitness row grl-snam uses to rank +its OWN base-policy checkpoints over a scene corpus. A downstream extension can layer its +own domain-specific scorecard on top; the base half here is shared, so a grl-snam training +run and a C++ ``cvc::nav`` collector run report identical base numbers (the aggregation logic + JSON keys are field-for-field the same, and ``tests/test_scorecard`` pins them to the same hand-computed values the C++ ``nav_stats_test`` uses). @@ -20,7 +20,7 @@ import math from dataclasses import dataclass, field -from .material_palette import NUM_MATERIALS # 13 == cvc::dbg kNumMaterials (channel.cpp table) +from .material_palette import NUM_MATERIALS # 13 == cvc::nav::kNumMaterials @dataclass @@ -220,15 +220,15 @@ def to_json(self) -> str: @classmethod def from_dict(cls, d: dict) -> NavScorecard: """Parse a scorecard dict back into a ``NavScorecard`` — the inverse of :meth:`to_dict`, and - the reader for the C++ ``cvc::nav::scorecard_json`` output (the keys are field-for-field the - same, so a ``dbg_arrival_check3 --episodes`` / ``scorecard_json`` row loads directly). This is + the reader for the C++ ``cvc::nav::nav_scorecard::to_json()`` output (the keys are + field-for-field the same, so a recorded C++ scorecard row loads directly). This is the plumbing a selection/loss stage reads a recorded corpus fitness from; nothing consumes it yet (Track 4 Phase 1+ wires it into training). Tolerant of a partial producer: any missing key keeps the dataclass default, so an older or - newer writer (or one that omits zero fields) still loads. An ``rf`` sub-object, if present - (the DBG ``scorecard_json`` embeds one), is ignored here — the base reader is RF-free; the DBG - side reads the RF scorecard separately. + newer writer (or one that omits zero fields) still loads. An extension sub-object, if + present (a downstream writer may embed its own record), is ignored here — the base reader + is domain-neutral; the extension reads its own sub-record separately. """ cov = d.get("mean_coverage") or {} return cls( @@ -274,7 +274,7 @@ def from_dict(cls, d: dict) -> NavScorecard: @classmethod def from_json(cls, s: str) -> NavScorecard: - """Parse a scorecard JSON string (``to_json`` / C++ ``scorecard_json``) into a + """Parse a scorecard JSON string (``to_json`` / C++ ``nav_scorecard::to_json()``) into a ``NavScorecard``. See :meth:`from_dict`.""" return cls.from_dict(json.loads(s)) diff --git a/grl_snam/sdf_nav.py b/grl_snam/sdf_nav.py index cce0d4a..280e2c7 100644 --- a/grl_snam/sdf_nav.py +++ b/grl_snam/sdf_nav.py @@ -424,7 +424,7 @@ def sdf_rollout( returning an extra force (accel units) at the current positions each substep, summed into the acceleration alongside ``F_bar``/``F_goal``/``F_mat`` — the physics-agnostic Python twin of ``cvc::nav``'s ``ext_force`` port. It carries - NO RF/comms vocabulary; a private consumer (DBG's comm force) supplies it. + NO domain vocabulary; the caller (e.g. an application-specific force) supplies it. ``None`` (the default) is additively inert and bit-for-bit unchanged.""" hdt = dt / nsub minclr = torch.full((o.shape[0],), 9.9, device=o.device) @@ -440,7 +440,7 @@ def sdf_rollout( a = F_bar + F_goal + F_mat - ga.unsqueeze(-1) * v else: a = F_bar + F_goal - ga.unsqueeze(-1) * v - if ext_force_fn is not None: # generic external force (e.g. DBG comm force) + if ext_force_fn is not None: # generic external force (caller-supplied) a = a + ext_force_fn(o) v = v + hdt * a sp = v.norm(dim=-1, keepdim=True) @@ -647,7 +647,7 @@ def bicycle_rollout( a_max_e, a_lat_e = a_max * _mu, a_lat_max * _mu F_goal = -be.unsqueeze(-1) * (o - goal) - # Generic external force (e.g. DBG comm force). Evaluated once and, like + # Generic external force (caller-supplied). Evaluated once and, like # F_mat, joins BOTH couplings — the longitudinal projection AND the # steering bias below. None (default) is additively inert. F_ext = ext_force_fn(o) if ext_force_fn is not None else None diff --git a/grl_snam/selection.py b/grl_snam/selection.py index c217ed5..0ddd1bd 100644 --- a/grl_snam/selection.py +++ b/grl_snam/selection.py @@ -4,7 +4,7 @@ This is deliberately separate from :mod:`grl_snam.scorecard` (which stays a pure schema + reducer): selection reads finished ``NavScorecard`` rows and never serializes back, so it can never perturb the C++<->Python parity surface (the composite is a free function, NOT a scorecard field). It also stays -RF-free — the DBG campaign composes an RF fitness on top of ``composite_fitness`` in ``grl_snam_dbg``; +domain-neutral — a downstream extension can compose its own fitness on top of ``composite_fitness``; nothing here touches the loss or the rollout (that is Phase 2). Typical use — rank recorded corpus rows (the C++/native collector fills the formation/coverage/grip @@ -109,9 +109,10 @@ def select_best( def _main(argv: list[str] | None = None) -> int: """`python -m grl_snam.selection card1.json card2.json [--weights '{"w_form_arrival":1.0}']` — - load recorded scorecard rows (cvc::nav ``scorecard_json`` / ``NavScorecard.to_json``) and print - them best-first by composite fitness. This is the offline SELECTION entry point; the recorded - (``from_json``) row is where the formation/coverage/grip fields are actually non-zero.""" + load recorded scorecard rows (C++ ``cvc::nav::nav_scorecard::to_json()`` / + ``NavScorecard.to_json``) and print them best-first by composite fitness. This is the offline + SELECTION entry point; the recorded (``from_json``) row is where the formation/coverage/grip + fields are actually non-zero.""" import argparse ap = argparse.ArgumentParser( diff --git a/grl_snam/selftest.py b/grl_snam/selftest.py index fcae4e2..b886c6a 100644 --- a/grl_snam/selftest.py +++ b/grl_snam/selftest.py @@ -13,8 +13,8 @@ 4. SHAPES: (o, v, min_clear) come back as (B,2), (B,2), (B,). This is the correctness counterpart to the visual grl_snam_lab demo. It needs -torch (a declared GRL-SNAM dependency); the applied full-gym end-to-end demo lives -in a separate, private downstream project. +torch (a declared GRL-SNAM dependency); applied end-to-end demos live in +downstream projects. Run: ``grl-snam-selftest`` (installed console script) or ``python -m grl_snam.selftest``. """ diff --git a/grl_snam/tools/austin.py b/grl_snam/tools/austin.py index 5313c9c..89e8b76 100644 --- a/grl_snam/tools/austin.py +++ b/grl_snam/tools/austin.py @@ -15,7 +15,7 @@ The bundle path is always supplied at runtime and never defaulted: the DATA is OpenStreetMap (ODbL) and SRTM (US public domain), so renders are publishable, -but the bundle itself is a local artifact of a private project and its path +but the bundle itself is a local artifact and its path does not belong in this repo. """ diff --git a/grl_snam/tools/material_raster.py b/grl_snam/tools/material_raster.py index 5947607..497a15e 100644 --- a/grl_snam/tools/material_raster.py +++ b/grl_snam/tools/material_raster.py @@ -15,9 +15,9 @@ learned model — reproducible and reviewable; tune :func:`classify` + the palette's ``GRIP_MU`` / ``TERRAIN_RISK`` for other biomes. -NOTE on the palette: it is an RF/building-material table (``reinforced_concrete`` etc. carry -penetration-loss dB) plus a few natural-terrain classes, with NO asphalt/road class. ``reinforced_ -concrete`` means building walls, not pavement; buildings are obstacles the convoy never drives +NOTE on the palette: it is a building-material table (``reinforced_concrete``, ``brick``, +``glass`` etc.) plus a few natural-terrain classes, with NO asphalt/road class. ``reinforced_ +concrete`` means building walls, not pavement; buildings are obstacles a vehicle never drives (tagged from the scene's building metadata), so this GROUND classifier only ever emits the drivable classes ``open_air`` / ``foliage`` / ``soil`` / ``water`` / ``rock``. """ @@ -103,7 +103,10 @@ def to_material_json(material_id: np.ndarray, bounds) -> dict: ids = sorted(int(v) for v in np.unique(material_id)) return { "schema": "cvc-scene-material/1", - "provenance": "scene land cover (grl-snam material-raster: masks if present, else satellite); ids = cvc::dbg MATERIAL_TABLE", + # This provenance value matches the C++ cvc::nav::material_raster::to_json byte for byte + # (libcvc src/cvc/nav/material_raster.cpp). The documents as a whole are not + # byte-identical: the two writers format floats (bounds, mu/risk) differently. + "provenance": "scene land cover (cvc::nav material_raster: masks if present, else satellite); ids = the 13-class material palette", "rows": rows, "cols": cols, "bounds": _bounds_dict(bounds), diff --git a/grl_snam/tools/scorecard_eval.py b/grl_snam/tools/scorecard_eval.py index cdc9325..744e2c7 100644 --- a/grl_snam/tools/scorecard_eval.py +++ b/grl_snam/tools/scorecard_eval.py @@ -1,13 +1,13 @@ -"""Base-policy scorecard eval — rank a CoefMLP checkpoint by its RF-free navigation +"""Base-policy scorecard eval — rank a CoefMLP checkpoint by its domain-neutral navigation fitness over a scene corpus. Runs the vectorized :class:`~grl_snam.swarm.Swarm` with its opt-in base nav_stats collector across a corpus of city scenes, reduces each episode to an :class:`~grl_snam.scorecard.EpisodeStats`, and aggregates them into one :class:`~grl_snam.scorecard.NavScorecard` — the single fitness row grl-snam uses to -rank its own base-policy checkpoints (arrival, economy, safety, material), RF-free. -This is the base half of the two-layer nav-stats design's training bridge (the DBG -campaign layers an ``rf_scorecard`` on top); the scorecard field-mirrors the C++ +rank its own base-policy checkpoints (arrival, economy, safety, material). +This is the base half of the two-layer nav-stats design's training bridge (a downstream +extension can layer its own scorecard on top); the scorecard field-mirrors the C++ ``cvc::nav::nav_scorecard`` and the native ``sim_world`` collector, so a checkpoint scores identically whichever path evaluates it. diff --git a/tests/test_ext_force.py b/tests/test_ext_force.py index 9a5fcf7..3f68a48 100644 --- a/tests/test_ext_force.py +++ b/tests/test_ext_force.py @@ -1,8 +1,9 @@ """ext_force_fn parity — the generic external-force channel in sdf_rollout and bicycle_rollout (the Python twin of cvc::nav's ext_force port). A ``None`` hook (the default) and a zero force are byte-identical to the plain rollout; a live -force bends the trace. This is what carries the DBG comm force once it moves off -the carrot bias: F_comm/F_jam sum with F_bar/F_goal in the integrator. +force bends the trace. This is what carries an application-specific force as a +genuine force term rather than a carrot bias: F_ext sums with F_bar/F_goal in the +integrator. """ from __future__ import annotations diff --git a/tests/test_nav_ext_force.py b/tests/test_nav_ext_force.py index 45b6ceb..79e8773 100644 --- a/tests/test_nav_ext_force.py +++ b/tests/test_nav_ext_force.py @@ -3,7 +3,7 @@ #66 added the generic ext_force_fn hook to sdf_rollout / bicycle_rollout; this pins that SdfNavigator threads it through step() correctly: a None or zero-force hook leaves every trace bit-for-bit unchanged (the byte-identical-off contract -the DBG comm-force cutover relies on), and a non-zero hook actually bends the +an external-force caller relies on), and a non-zero hook actually bends the path. """ diff --git a/tests/test_nav_native_drive.py b/tests/test_nav_native_drive.py index 85673a2..31b6eea 100644 --- a/tests/test_nav_native_drive.py +++ b/tests/test_nav_native_drive.py @@ -1,6 +1,6 @@ """SdfNavigator native drive dispatch (``GRL_SNAM_NAV_DRIVE=native``). -The per-agent navigator the DBG convoy drives (Squad -> Scenario -> +The per-agent navigator a Squad convoy drives (Squad -> Scenario -> ``SdfNavigator``) can optionally run the torch-free C++ fused drive (``nav_native.drive_step``) instead of the torch coef-net + bicycle rollout, keeping the carrot FSM in Python. This is the SdfNavigator assembly of the diff --git a/tests/test_scorecard.py b/tests/test_scorecard.py index d8c13eb..14ee963 100644 --- a/tests/test_scorecard.py +++ b/tests/test_scorecard.py @@ -76,7 +76,7 @@ def test_scorecard_json_keys(): "material_time_share", ): assert k in d - assert "rf" not in d # base-only shape (grl-snam-non-DBG) + assert "ext" not in d # base-only shape (no extension sub-object) def test_navstats_accumulates_turn_and_fuel(): @@ -248,13 +248,14 @@ def test_scorecard_json_has_track1_keys(): } -# --- Track 4 Phase 0: the scorecard reader (inverse of to_dict/to_json; loads C++ scorecard_json) --- +# --- Track 4 Phase 0: the scorecard reader (inverse of to_dict/to_json; loads C++ nav_scorecard) --- def test_scorecard_reader_round_trips(): # A fully-populated scorecard (all Track-1 field families non-default) survives to_dict->from_dict # and to_json->from_json unchanged. Since to_dict's keys are field-for-field the C++ - # cvc::nav::scorecard_json keys, this is also the guarantee that a recorded C++ scorecard row loads. + # cvc::nav::nav_scorecard::to_json() keys, this is also the guarantee that a recorded C++ + # scorecard row loads. e0 = EpisodeStats( success=True, makespan_s=20, @@ -292,11 +293,11 @@ def test_scorecard_reader_round_trips(): assert NavScorecard.from_json(sc.to_json()).to_dict() == d # JSON round-trip too -def test_scorecard_reader_tolerates_partial_and_ignores_rf(): - # A partial producer (missing keys) loads with dataclass defaults; an embedded "rf" sub-object - # (the DBG scorecard_json shape) is ignored by the base reader. +def test_scorecard_reader_tolerates_partial_and_ignores_extension(): + # A partial producer (missing keys) loads with dataclass defaults; an embedded extension + # sub-object (a downstream writer's own record) is ignored by the base reader. sc = NavScorecard.from_dict( - {"checkpoint": "x", "success_rate": 0.5, "rf": {"outage_rate": 0.9}} + {"checkpoint": "x", "success_rate": 0.5, "ext": {"extra_metric": 0.9}} ) assert sc.checkpoint == "x" assert math.isclose(sc.success_rate, 0.5) @@ -305,4 +306,4 @@ def test_scorecard_reader_tolerates_partial_and_ignores_rf(): assert math.isclose(sc.mean_coverage.phantom_frac, 0.0) # missing nested -> default # missing list field -> the length-NUM_MATERIALS zero default, NOT [] (so share[m] never IndexErrors) assert sc.material_time_share == [0.0] * NUM_MATERIALS - assert not hasattr(sc, "rf") # rf sub-object dropped, not smuggled onto the base row + assert not hasattr(sc, "ext") # extension sub-object dropped, not smuggled onto the base row