Skip to content

feat(pipelines): channel_attrs Phase-A pipeline — corridor scaling, API, 69 tests, bug fixes#12

Open
taddyb wants to merge 19 commits into
masterfrom
channel-attrs-pipeline
Open

feat(pipelines): channel_attrs Phase-A pipeline — corridor scaling, API, 69 tests, bug fixes#12
taddyb wants to merge 19 commits into
masterfrom
channel-attrs-pipeline

Conversation

@taddyb

@taddyb taddyb commented Jul 5, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

  • WRF-Hydro order-width table: replaces Leopold & Maddock analytical formula with NCAR `Mannings_Bw` lookup (Strahler 1–10) as the order fallback for corridor half-width. Orders 1–4 pin at the 10 m floor; orders 5–10 scale at 1.5× bottom width (11–165 m).
  • 10 m minimum corridor floor: reduced from 100 m — corridors track channel geometry rather than the MERIT positional-error envelope. `corridors_10m.parquet` replaces `corridors_100m.parquet` as the fixed-width sensitivity product.
  • Public API (`init.py`): exports all pure pipeline functions for notebooks and downstream use:
    • `build_corridors`, `build_scaled_corridors` — corridor polygon construction
    • `order_bankfull_width_m`, `scaled_half_width_m` — width scaling primitives
    • `sample_along_lines` — nearest-cell raster sampling along flowlines
    • `zonal_stats_corridors(da, corridors, stat, id_col)` — channel-burning extraction path: burns corridor polygons into a raster via the extractrs accessor, with automatic CRS reprojection
    • `weighted_transfer`, `assemble`
  • 72 unit tests (was 0): all 12 pipeline modules covered, including parametrized corridor hinge tests against `paths.WRF_HYDRO_BW_M` directly.
  • 3 data-corruption bug fixes: `streamcat_transfer` broad except swallowing `KeyError` (silently zeroed HUC2 regions); `wtd_sample` bare except hiding tile failures + fallback column name mismatch; `corridors` unconditional width-parquet reads crashing before `--only` filter.

Test plan

  • `python -m pytest pipelines/channel_attrs/tests/ -v` — 72 tests, all pass in ~2 s
  • `python -m pipelines.channel_attrs.corridors` — produces `corridors_10m.parquet` and `corridors_scaled.parquet`; `p50=10.0`, `count>10m=168,952` (49% of reaches clear the hinge)
  • `python -m pipelines.channel_attrs.wtd_sample` — produces `wtd_channel.parquet`; `wtd_corridor10_zs` coverage 136,374; Spearman(nearest, corridor) = 0.977
  • `python -m pipelines.channel_attrs.assemble` — `PASS: dim length + COMID order + schema OK`

🤖 Generated with Claude Code

taddyb and others added 19 commits July 4, 2026 12:03
@taddyb @claude
feat(pipelines): channel_attrs scaffolding — paths, env, staging, sch…
ce792ba 
…ema recon

Task 0 of Phase-A channel-corridor attribute extraction pipeline.

- pipelines/channel_attrs/paths.py: single source of truth for all paths/constants
  (corrected MERIT_RIV to ddr/data/merit/ vs plan draft's ddr/data/)
- pipelines/channel_attrs/tests/test_paths.py: path existence + staging dir tests (pass)
- pipelines/channel_attrs/README.md: env setup, schema recon, download manifest

Schema recon findings recorded in README:
- global.nc: dim COMID=2939404, 29 float64 attrs + COMID int64 coord
- stats JSON: {var: {min,max,mean,std,p10,p90}} per finite values
- MERIT shapefile: 15 fields (COMID,lengthkm,uparea,...), EPSG:4326, 346327 features

Downloads started (large in background): MERIT-SWORD, Zarrabi, Fan2013,
GFPLAIN250m, BFI, Principal Aquifers complete; SWORD/NHDPlusV2/ZS_WTD/StreamCat in progress.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@taddyb @claude
docs(pipelines): update README with ZS_WTD raster content findings
1835f58 
conus_MF6_SS_Unconfined_250_dtw.tif confirmed inside Output_CONUS_trans_dtw.zip
(876 MB, valid zip) — directly usable 250m DTW GeoTIFF, ZS primary WTD source confirmed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@taddyb @claude
feat(pipelines): MERIT corridor geometries (100m + wide)
11927f2 
Adds corridors.py with build_corridors() (buffer in EPSG:5070, preserve COMID)
and main() that writes derived/corridors_100m.parquet and derived/corridors_wide.parquet.
Both files have 346,327 features (matches the recorded shapefile count).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@taddyb @claude
spec(pipelines): corridor buffer scaling — positional-error floor + w…
e04dc5a 
…idth-scaled hinge (lit-grounded)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@taddyb
fix(pipelines): width-scaled corridors per buffer-scaling spec (hinge…
224775d 
… rule, width priority chain, single reprojection)
@taddyb @claude
feat(pipelines): SWORD observed widths via MERIT-SWORD weighted transfer
f6cb63d 
Implements Task 2 of Phase-A channel-attributes pipeline.

- transfer.py: length-weighted attribute transfer helper (shared by Tasks 2/4/5)
- sword_width.py: melts Wade et al. 2025 translation tables -> MERIT COMIDs,
  applies weighted_transfer with SWORD v16 NA widths
- tests/test_transfer.py: TDD tests for weighted_transfer

Adaptation notes from on-disk inspection:
- Translation table MERIT reach variable is 'mb' (not 'COMID' as plan assumed)
- SWORD reads via netCDF4 (xarray decode fails on string var in this file)
- CONUS = pfaf-2 regions 71-78 (8 files confirmed)

Sanity (Step 4 criteria, all pass):
- Coverage: 28,529 / 346,327 CONUS reaches = 8.2% (within 5-15% target)
- Median width of covered reaches: 127.3 m (> 30 m threshold)
- Mississippi mainstem near Vicksburg: 826-1218 m (> 500 m threshold)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@taddyb @claude
feat(pipelines): NHDPlus->MERIT length-weighted crosswalk (+match_fra…
ff7cb6e 
…c quality flags)

Implements Task 3 of Phase-A: builds nhd_merit_crosswalk.parquet by buffering
each MERIT reach 300 m in EPSG:5070, spatial-joining intersecting NHDFlowline_Network
reaches, computing exact clipped lengths per pair, and computing per-COMID match_frac
quality flags. CONUS build: 8x8 tile grid, 4.0 min wall time, 1.82M pairs,
162k matched COMIDs. Archive: NHDPlusV21 FileGDB extracted with py7zr (15.5 GB
uncompressed, 78s). Tests: 3 synthetic geometry tests pass (part_len exact,
match_frac correct, no-overlap returns empty).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@taddyb @claude
feat(channel-attrs): Task 4 StreamCat corridor imperviousness via JSO…
a551d2e 
…N API

Reads all 21 pctimp2019_Region*.json files via streaming ijson parse
(handles truncated Region07 gracefully), filters NHD->MERIT crosswalk
to match_frac >= 0.3, runs weighted_transfer, divides by 100 to fraction.
Writes derived/corridor_impervious.parquet.

Coverage: 151,503/156,002 MERIT COMIDs (97.1%). Sanity: LA River bbox
median 0.543; rural Montana median 0.009.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@taddyb @claude
feat(channel-attrs): Task 5 Zarrabi bankfull width+depth transfer
4e9d330 
Reads Bankfull_Meanflow_CONUS.txt (comma-separated, index_col=0),
extracts bnk_depth/bnk_width, renames COMID->foreign_id, runs
weighted_transfer per column via NHD->MERIT crosswalk (match_frac>=0.3).
Writes derived/bankfull.parquet.

Coverage: 155,831/156,002 (99.9%). Median depth 1.43 m (in [0.3,3] range).
McManamay confinement skipped (not in download manifest; recorded in README).
Spearman(depth,uparea)=0.469 (marginally below 0.5; correct direction).
Spearman(width,channel_width_obs)=0.005 — outlier issue in Task-2 SWORD
data (up to 61k m); flagged in README, not a Zarrabi transfer bug.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@taddyb @claude
feat(channel-attrs): Task 9 BFI table transfer + drainage density
d0177e1 
BFI: reads BFI_CONUS.zip directly (NHD-indexed table, CAT_BFI in [0,100]%
divided by 100), filters negative pseudo-COMIDs, transfers via NHD->MERIT
crosswalk (match_frac>=0.3). Deviation from plan's raster path: BFI_CONUS.zip
is a pre-aggregated NHD table, not the raw bfi48grd raster; recorded in README.
Coverage: 156,002/156,002 (100%). BFI in [0,0.9]; East > SW sanity passed.

Drainage density: lengthkm/catchsize (km^-1) from MERIT shp + global NC;
catchsize<=0 guarded to NaN. Median 0.197 km^-1 (in [0.1,2] range).
Writes derived/basin_extras.parquet.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@taddyb @claude
fix(pipelines): SWORD width transfer — pre-bugfix MERIT-Basins COMID …
e540e8b 
…mismatch

The Wade et al. 2025 mb_to_sword translation tables were built from a
pre-bugfix1 MERIT-Basins COMID ordering. The same integer COMID refers to
completely different reaches in that version vs the bugfix1 shapefile used
by this project (distances of 100–1100 km within a single pfaf-2 region).
Joining these COMIDs against the NHDPlus→MERIT bugfix1 bankfull dataset
produced zero rank correlation (spearman 0.012).

Fix: replace the translation-table join with a spatial join of SWORD NA
reach centerpoints into MERIT bugfix1 catchment polygons, using SWORD
reach_length as the weight. Only river reaches (lakeflag == 0) are used;
lake/reservoir widths are not representative of channel width.

After fix: spearman(channel_width_obs, bankfull_width) = 0.596, n = 7,103
(cap ≤ 2 km), median obs/bankfull = 1.17, p95 = 4.5. Coverage: 13,470
MERIT COMIDs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@taddyb @claude
feat(pipelines): channel water-table sampling (nearest-cell + corrido…
dd7fac1 
…r sensitivity, ZS + Fan)

- sample_along_lines(): batched vectorised xarray.sel() across all tile vertices
  for ~10x speedup vs per-reach loop; optional bed_depths→frac_below column
- ZS (Zell & Sanford 250 m): positive-down DTW confirmed (95.2% ≥ 0, midCONUS
  p50=8 m); artesian negatives clipped to 0; frac_below_zs persisted for Task 7
- Fan 2013: all-negative raw WTD negated to positive below-surface; p50=0.69 m
- Processes 38 pfaf-4 tiles in 3 min; 31 tiles have ZS coverage (7 in Canada)
- Cross-checks: spearman(ZS, Fan)=0.360 (below 0.4 — warning surfaced, models
  differ as expected: CONUS transient vs global equilibrium);
  spearman(nearest-cell ZS, corridor ZS)=0.936 (>>0.8, OK)
- Coverage: ZS 173,506/346,327 (CONUS only); Fan 346,327 (full); corridor 136,413
- Output: derived/wtd_channel.parquet (COMID, wtd_channel_zs, wtd_channel_fan,
  wtd_corridor100_zs, wtd_channel_n, frac_below_zs)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@taddyb @claude
feat(pipelines): bed-relative channel head + losing fraction
c9b592b 
- bed_relative(): channel_wtd_bed_rel = wtd_channel_zs - bankfull_depth
  (positive = water table below bed = losing-possible; negative = gaining)
- losing_fraction from Task-6 frac_below_zs (per-vertex fraction where
  wtd > bankfull_depth); no re-sampling needed
- Overall CONUS: p50 bed_rel=-0.90 m; 29% positive (losing-possible);
  median losing_fraction=0.118; majority-losing (>0.5) on 14.2% of reaches
- Three spec-mandated regime checks (all pass):
    Ogallala COMID=74049945: bed_rel=+62.6 m, losing_fraction=1.00
    Appalachian COMID=74037194: bed_rel=-7.87 m, losing_fraction=0.00
    LA River COMID=77030730: bed_rel=+1.37 m, corridor_impervious=0.71
    (Phase-C falsification pair: losing channel in impervious-sealed watershed)
- Output: derived/wtd_bedrel.parquet (COMID, channel_wtd_bed_rel, losing_fraction)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@taddyb @claude
feat(pipelines): scaled corridors on corrected widths + 3km cap
ac4f996 
- main() left-merges channel_width_obs.parquet and bankfull.parquet so
  _resolve_width_m uses real per-reach widths (13,470 SWORD obs + 155,831
  Zarrabi bankfull) instead of order-fallback for all reaches.
- _resolve_width_m now caps width at paths.WIDTH_CAP_M = 3000 m before
  the hinge formula, guarding residual SWORD estuary/lake widths (CONUS
  bugfix1 max was 16.7 km → half-width capped at 4.5 km).
- main() accepts --only {scaled,100m} to rebuild a single product; used
  to skip the unchanged 100 m set.
- Two new tests: cap applied; left-merge preserves row count.
- Spec caveats: 3 km cap rationale + Wade crosswalk validity note.

Rebuilt corridors_scaled distribution: p50=100 m, p99=352 m, max=4500 m,
count > 100 m floor = 16,545 reaches.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@taddyb @claude
feat(pipelines): corridor alluvium fraction from Principal Aquifers o…
175b946 
…verlay

DEVIATION from plan Task 8: overlay uses corridors_scaled (not corridors_100m
per plan text).  Rationale: large alluvial rivers need the scaled corridor to
cover the channel + its alluvial floor; a fixed 100 m strip misses the alluvial
body for order-8/9 reaches.  Documented in alluvium.py docstring.

Alluvial class chosen: ROCK_TYPE == 100 ("Unconsolidated sand and gravel
aquifers") from USGS Principal Aquifers.  All explicitly alluvial/basin-fill
AQ_NAMEs (Mississippi Valley, Pecos, Basin-and-Range fill, etc.) fall in this
class.  ROCK_TYPE 200 (Semiconsolidated sand, coastal/bedrock) excluded.

Implementation: dissolve 806 alluvial polygons to 1 geometry, then
gpd.overlay intersection chunked by pfaf-4 prefix (38 chunks).  Fill
alluvium_fraction=0 where no intersection (explicit, not NaN).

Results (346,327 CONUS reaches, 0 nulls):
  45,767 reaches (13.2%) have alluvium_fraction > 0
  Sanity: Mississippi alluvial valley median=1.000; Rocky Mtn median=0.000

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@taddyb @claude
feat(pipelines): assemble merit_channel_attributes_v1.nc + normalizat…
44577ef 
…ion stats

11-variable Phase-A channel attribute netCDF (2,939,404 global COMIDs, f64,
dim COMID) assembled from derived parquets; NaN outside CONUS coverage.
Stats JSON at merit_attribute_statistics_merit_channel_attributes_v1.nc.json
matches ddrs six-key per-variable format (min/max/mean/std/p10/p90 over
finite values). TDD: test_assemble verifies dim=4, dtype, NaN reindex.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@taddyb @claude
feat(pipelines): WRF-Hydro width table, 10 m floor, API, tests, bug f…
e99808c 
…ixes

Replace Leopold & Maddock order-width formula with the NCAR WRF-Hydro
Mannings_Bw lookup table (Strahler 1–10) as the order fallback for corridor
scaling.  Lower the minimum corridor half-width from 100 m to 10 m so
corridors 5–10 now scale with channel geometry rather than flooring at the
positional-error envelope.  With the 1.5× hinge rule, orders 1–4 still pin
at 10 m; orders 5–10 yield 11–165 m half-widths from the table.

Bug fixes:
- streamcat_transfer: broad except swallowed KeyError, silently zeroing HUC2
  regions — narrowed to IncompleteJSONError only
- wtd_sample: bare except pass hid zonal_stats failures per tile; fallback
  column name mismatch (wtd → wtd_corridor10_zs)
- corridors main(): unconditional width-parquet reads before --only filter
  caused FileNotFoundError; replaced with lazy _merge_width_columns()
- _to_equal_area: CRS=None passed to geopandas producing cryptic error;
  now raises ValueError explicitly

API: populate __init__.py with build_corridors, build_scaled_corridors,
order_bankfull_width_m, scaled_half_width_m, sample_along_lines,
weighted_transfer, assemble.

Tests: 69 tests (was 0) covering all 12 pipeline modules, including new
test_alluvium, test_core, test_sword_width, test_zarrabi_transfer files and
parametrized corridor hinge tests (orders 1–4 floor, 5–10 scale against
paths.WRF_HYDRO_BW_M directly).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@taddyb @claude
feat(pipelines): expose zonal_stats_corridors API; drop spec file
93a8be8 
Add zonal_stats_corridors(da, corridors, stat, id_col) as the public
channel-burning extraction path: corridors are reprojected to the raster
CRS and zonal statistics computed via the extractrs accessor. Refactor
_corridor_mean_zs to delegate each pfaf-4 tile to zonal_stats_corridors
so the tiled pipeline path and the public API share the same logic.

Export zonal_stats_corridors from __init__.py alongside the existing
corridor-geometry and sampling functions. Add 3 tests covering the uniform-
raster path, CRS reprojection, and the no-CRS ValueError guard.

Remove specs/2026-07-04-corridor-buffer-scaling.md — design docs live
outside the repo.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@taddyb @claude
test(pipelines): import zonal_stats_corridors from public API; add __…
b64e0b4 
…all__ smoke test

Update test_wtd_sample.py to import sample_along_lines and zonal_stats_corridors
from pipelines.channel_attrs (the public package) rather than the submodule,
so the __init__.py export path is exercised. Add test_public_api_exports_all_symbols
to test_core.py to assert every name in __all__ is importable and callable.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@taddyb