diff --git a/.github/workflows/main.yaml b/.github/workflows/main.yaml index f1245ed6..26e89e62 100644 --- a/.github/workflows/main.yaml +++ b/.github/workflows/main.yaml @@ -192,7 +192,8 @@ jobs: - name: Tests with the codecs extra run: | python -m pip install -e ".[dev,codecs]" - pytest --no-cov tests/python/test_io_geotiff.py tests/python/test_geotiff_fixtures.py + pytest --no-cov tests/python/test_io_geotiff.py tests/python/test_geotiff_fixtures.py \ + tests/python/test_io_read_meta.py tests/python/test_cli_mesh_mosaic.py # Increment 13, ruling 10: read the .vtk output back with the readers # ParaView uses. The suite skips when vtk is absent, so it is run only diff --git a/CLAUDE.md b/CLAUDE.md index a819734e..2cdca9bd 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -14,9 +14,11 @@ This project is governed by specialized sub-agents. Always defer tasks to the co ## 2. Core Constraints & Technical Mandates * **Strict Size Limit:** Under **700 lines of production code per pull request**, - where a line counts unless it is a comment, a docstring, or the body of a raw - literal; tests excluded. The exclusions exist so the ceiling does not penalise - the comment density this project asks for. Lines count as written: packing + where a line counts unless it is blank, a comment, a docstring, or the body of + a raw literal; tests excluded. The exclusions exist so the ceiling does not + penalise the comment density this project asks for, and blank lines add no + reading (Ola, 2026-09-28; before that, 20b counted blank lines). Lines count + as written: packing code by hand under `# fmt: skip` / `# fmt: off` is allowed, provided the packed lines stay readable and the review says why each new region is packed (Ola, 2026-09-27, on `tools/bench.py`). This is the only statement of diff --git a/ROADMAP.md b/ROADMAP.md index 66abd450..09dc9062 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -42,8 +42,9 @@ increment that most needs a picture to check against | 20 | Quality start: before DEM refinement, Steiner points at the DEM node nearest each bad triangle's circumcentre until the start mesh has a 25° minimum angle (`--start-min-angle`, 0 is off); geometry only, input segments not split, serial and deterministic. Removes increment 16's boundary fans | landed with 20b as interim (C1-C3 open, to 20c; C4 -> 20b) | `docs/increments/20-start-quality.md` | | 20b | Minimum insertion distance from constraints: when refinement's worst node lies within ε = clamp(tol / slope, cell/100, cell/2) of a constraint segment, insert the foot on the segment (off-node, bilinear z) instead; a footed node still above tolerance is inserted after all, so the tolerance guarantee is unchanged. Removes the 0.0117° needle at 1 m (increment 20's C4) | shipped with 20 (#97) | `docs/increments/20b-min-insertion-distance.md` | | — | `tools/bench.py`: the 1 m benchmark and the thread-scaling sweep from one checked-in command, with power state, quality and commit recorded per run; the one-off scripts in `docs/benchmarks/2026-09-26/` are its specification. Rule 2's acceptance run needs it | shipped with branch `tools-bench`'s PR | `docs/benchmarks/bench-py.md` | -| — | The serial phase: profile refine's serial insert-and-flip phase, then parallelise what the profile blames. Scaling tops out at about 2.0-2.2×. Profiled 2026-09-27: serial part about a third of 1-thread refine, mostly Lawson legalisation; the scan stops near 5× from load imbalance | profiled; designed as increment 21 (Ola's rulings 2026-09-27: L1 determinism, one path, at most 2 % more triangles). **21a shipped with branch `increment21a-quick-wins`'s PR**: dynamic scan blocks, active merge, reused flip stack; mesh bit-identical; on AC refine -10 to -15 % at 8 threads, ceiling 2.1x -> 2.3x (`docs/benchmarks/2026-09-27/21a-acceptance.md`). **21b shipped with branch `increment21b-lattice-incircle`'s PR**: an int64 lattice incircle answers 99.97-100 % of refine's incircle tests; mesh bit-identical; on battery refine -15 % at 8 threads, ceiling 2.3x -> 2.5x (`docs/benchmarks/2026-09-27/21b-acceptance.md`). **21c measured** (`docs/benchmarks/2026-09-27/21c/README.md`): option C costs 6.5-10 % more triangles; A1 about 1 %; A0 is bit-identical (on these two inputs, not proven), and with evaluate-once is modelled at only 7-11 % faster refine at 8 threads and slower at 4. **21d is deferred** (Ola, 2026-09-27: "Review and push 21c, then basin-work"): the multi-tile DEM and the computation CRS come first, then domain decomposition, which parallelises scan and split together | `docs/increments/21-parallel-refine.md`, `docs/benchmarks/2026-09-27/serial-profile/README.md` | +| — | The serial phase: profile refine's serial insert-and-flip phase, then parallelise what the profile blames. Scaling tops out at about 2.0-2.2×. Profiled 2026-09-27: serial part about a third of 1-thread refine, mostly Lawson legalisation; the scan stops near 5× from load imbalance | profiled; designed as increment 21 (Ola's rulings 2026-09-27: L1 determinism, one path, at most 2 % more triangles). **21a shipped with branch `increment21a-quick-wins`'s PR**: dynamic scan blocks, active merge, reused flip stack; mesh bit-identical; on AC refine -10 to -15 % at 8 threads, ceiling 2.1x -> 2.3x (`docs/benchmarks/2026-09-27/21a-acceptance.md`). **21b shipped with branch `increment21b-lattice-incircle`'s PR**: an int64 lattice incircle answers 99.97-100 % of refine's incircle tests; mesh bit-identical; on battery refine -15 % at 8 threads, ceiling 2.3x -> 2.5x (`docs/benchmarks/2026-09-27/21b-acceptance.md`). **21c measured** (`docs/benchmarks/2026-09-27/21c/README.md`): option C costs 6.5-10 % more triangles; A1 about 1 %; A0 is bit-identical (on these two inputs, not proven), and with evaluate-once is modelled at only 7-11 % faster refine at 8 threads and slower at 4. **21d is deferred** (Ola, 2026-09-27: "Review and push 21c, then basin-work"), behind the work in "Order of work" below; domain decomposition parallelises scan and split together | `docs/increments/21-parallel-refine.md`, `docs/benchmarks/2026-09-27/serial-profile/README.md` | | — | Release hardening: measure libc++'s `_LIBCPP_HARDENING_MODE_FAST` (and libstdc++'s assertions on the GCC leg), which bounds-check `std::vector`, `span` and the like in Release, on the 1 m benchmark; switch it on if the cost is small. Today an out-of-range index in shipped code the tests miss is undefined behaviour and crashes Python (Ola, 2026-09-27: "I'm surprised we don't have proper memory control"; CI's ASan/UBSan/TSan cover what the tests reach) | to measure (Ola, 2026-09-27); 21d, which it was placed after, is deferred behind the basin work | none yet | +| 15 | A DEM in several tiles, and the domain in its own CRS: 15a Norway, many tiles in one projected CRS (Ola's 254 DTM10 UTM33 tiles; only the selected tiles must share a lattice, since 8 of them sit half a cell off); 15b the domain polygon reprojected into the DEM's CRS; 15c and 15d the São Francisco basin (geographic DEMs meshed in the DEM's own lattice frame, not resampled; window decoding and memory) | **15a shipped with branch `increment15-dem-mosaic`'s PR**: `--dem DIR` or several files, `--bbox`, tiles selected and stitched on one lattice, Q2-Q5 refusals; overlaps that disagree (real DTM10 tiles exported on different dates, up to 52 m) are split down the middle and reported per seam (Ola's Q1 revised, 2026-09-28). The design's acceptance box, 9 tiles and 10,051² nodes, meshes: 11.05 M triangles in 31 s on battery. 15b next (Ola, 2026-09-27: Norway first) | `docs/increments/15-dem-mosaic.md` | | 16b | Interior polygons and polylines as constraints ("terrain polygons": lakes, land cover, roads, rivers): `--features PATH`, a GeoJSON `FeatureCollection`, each feature naming a vocabulary property; closed or open `Breakline`s, crossings noded, off-node vertices with bilinear z. **Its working example is real data** (Ola, 2026-09-27): CORINE Land Cover 2018 over the benchmark tile `7908_3_10m_z33.tif`. The source is Ola's local copy, `rasputin_data/corine_sql/.../U2018_CLC2018_V2020_20u1.gpkg` (8.2 GB, EPSG:3035, a sibling of this repository, which also holds the 254-tile DTM10 archive for gap 6). It reads without GDAL: sqlite3 over its R-tree, the GeoPackage blob header stripped, `shapely.wkb`, then `pyproj` to 25833, so CRS stops in Python as before. Probed 2026-09-27 in 0.2 s: 60 polygons in 8 classes (heath, bare rock, sparse vegetation, bogs, intertidal flats, water, sea, urban), 11 068 vertices clipped to the tile, median segment 54 m against 10 m cells. The EEA's public ArcGIS service (`image.discomap.eea.europa.eu`, `Corine/CLC2018_WM`) returns the same 11 068 clipped vertices and is the route for anyone without the file. What it forces on 16b's design: neighbouring polygons share their boundaries, so each shared edge arrives twice; the polygons run past the domain and must be clipped; the extract is committed as a fixture with the Copernicus attribution. Placed before 20c because 20c may split constraint segments and should be designed and measured on inputs that have interior ones | designed in 16's R6, no increment file yet; after the serial phase, before 20c | `docs/increments/16-domain-polygon.md` (R6) | | 20c | Soft quality criterion: a penalty that each Steiner node or constraint split must pay for in angle gained, instead of 20's hard 25°; applied at the start and during DEM refinement; may split constraint segments when that improves the mesh. Ola's rulings on 20's C1-C3 | to design after 16b (`@architect` measures cost against 20 first) | `docs/increments/20-start-quality.md` (Ola's rulings) | | — | Auto-catchment: the watershed upstream of a coordinate, computed from the DEM and handed to `--domain`, so a catchment no longer has to be supplied as a file (Ola, 2026-09-27: "not far into the future"). The textbook route is depression handling (Priority-Flood, Barnes, Lehman and Mulla 2014), D8 flow directions (O'Callaghan and Mark 1984) and accumulation, the pour point snapped to the strongest flow nearby, the upstream cells traced and their outline turned into a polygon; the literature check is `@architect`'s. Open for its design: whether it runs in the C++ core (a 10 m tile is 25 M cells); how a stair-stepped cell outline becomes a domain polygon, which meets input coarsening; and that a real catchment crosses tile edges, so it needs gap 6 (a DEM in several tiles) first. Legacy has nothing on it (`grep -rliE "watershed|flow.?acc|flow.?dir|pour.?point|catchment" legacy` returns no files) | to design; after gap 6, which it needs; placed after 20c, can move ahead of it on Ola's word | none yet | @@ -87,7 +88,14 @@ as #93. extent from its header, selecting the tiles that cover the area, refusing mixed CRS, cell size or grid alignment, and stitching the selection into one grid. Aligned tiles are one bigger grid, so everything downstream is - unchanged. Planned after increment 14; no record yet. + unchanged. **Designed as increment 15** (`docs/increments/15-dem-mosaic.md`): + 15a and 15b are Norway (many tiles in one projected CRS; the domain in its + own CRS), 15c and 15d the São Francisco basin (geographic DEMs; memory). + +**Order of work, ruled by Ola on 2026-09-27** ("My priorities are to get the +Norwegian cases sorted first"): 15a and 15b (Norwegian multi-tile DEM), then +16b (CORINE terrain polygons over Norway), then auto-catchment. The basin's +15c and 15d follow; 21d and domain decomposition come after. Open and not MVP-blocking: **inputs in their own CRS**. The user (2026-09-26): "I don't think the domain CRS should have to match the DEM CRS in the future. diff --git a/docs/increments/12-dem-to-mesh.md b/docs/increments/12-dem-to-mesh.md index 2d221c70..af0ea269 100644 --- a/docs/increments/12-dem-to-mesh.md +++ b/docs/increments/12-dem-to-mesh.md @@ -68,7 +68,7 @@ fixture is decoded. triangles, 3.0 s. 6. **Without `imagecodecs`, `decode_dem` refuses the fixture** with a `GeoTiffError` that names Compression (259) = 5 and points at the `codecs` - extra (`src_python/tin_engine/io/geotiff.py:169`). Nothing new is needed + extra (`src_python/tin_engine/io/geotiff.py:207`). Nothing new is needed for the no-extra case beyond turning that error into a usage error. ## Rulings @@ -179,7 +179,7 @@ section), without changes: `keep_alive`. - The NoData value is converted to `T` exactly once. `decode_dem` already refuses a sentinel the cell type cannot hold - (`src_python/tin_engine/io/geotiff.py:307`), so the conversion is exact. + (`src_python/tin_engine/io/geotiff.py:345`), so the conversion is exact. - `sample(view, points)` takes a float64 `(N, 2)` array and returns `(z, valid)`. It releases the GIL. - `_core.pyi`: the class, the factory and `sample`. diff --git a/docs/increments/15-dem-mosaic.md b/docs/increments/15-dem-mosaic.md new file mode 100644 index 00000000..8736b157 --- /dev/null +++ b/docs/increments/15-dem-mosaic.md @@ -0,0 +1,1360 @@ +# Increment 15 — a DEM in many tiles, inputs in their own CRS, and the computation frame + +Status: **Q1-Q5 ruled by Ola; 15a implemented** on branch +`increment15-dem-mosaic` (red `2696bc2`, green `ff7cc8d`), in review. 15b-15d +are designed, not implemented; Q6-Q10 are open. Written by `@architect` before +`@tester`, per `docs/increments/README.md` step 1. + +## Ruled by Ola + +- **2026-09-27, priority:** "My priorities are to get the Norwegian cases + sorted first, so let's finish this and turn to the main topics." So the + increment is split so that **the Norwegian case ships first and on its own** + (15a, 15b): many aligned tiles in one projected CRS, as in Ola's DTM10 + archive, and a domain or features in their own CRS. The geographic DEM and + the basin-wide computation frame for the São Francisco basin (15c, 15d) are + designed here but come **after** the Norwegian sub-increments. Nothing in + 15a or 15b depends on them. +- **2026-09-27, Q1-Q5: "yes to all five, carry on"**, to the recommendations + in "Questions for Ola": + - **Q1:** two tiles that disagree at an overlapping node are refused, + naming both tiles, the node count and the largest difference. + - **Q2:** `--bbox` stays. + - **Q3:** the repository lives in `io/repository.py`, the one module in + `io/` that opens files. + - **Q4:** tiles with different NoData sentinels are refused. + - **Q5:** a request mixing the eight half-cell tiles with the main lattice + is refused; a request inside either lattice is meshed. + + Q6-Q10 (the basin) wait until after the Norwegian sub-increments. +- **2026-09-28, Q1 revised: overlaps that disagree are split down the middle + ("b", Ola).** Measured first, and widened after review with the committed + probe `15-probes/dtm10_dates.py` (120 random neighbour pairs, seed 1; a + tile's *export date* is the date of its side files, `.tif.aux.xml` or + `.tfw`, which agree on all 254 tiles; 143 of the 254 `.tif` files share one + date, 2021-12-10, later than their side files, so the `.tif` date does not + tell the exports apart, and the TIFF tags carry none). Of 120 sampled pairs, + 119 share a valid node: of 42 same-date pairs, 40 agree exactly and 41 + within 1 mm, but 7204_4 | 7304_3 differs by 1 mm or more at 156,379 nodes, + by up to 5.07 m; of 77 + different-date pairs, 49 agree within 1 mm and 28 do not, up to 52.1 m + (6500_1 | 6500_2). So a shared date makes agreement likely, not certain. + The differences have a mean near 0, are often exactly 0 on flat ground and + grow with slope (the first 60-pair sample, and three pairs examined node by + node). Kartverket's + metadata explains it: the current 10 m model is exported from NDH laser data + where it exists, supplemented by the 2013 contour-based DTM10 (±2-6 m), and + tiles were exported one by one as laser coverage grew. Not land uplift + (mm/yr) and not glaciers. The rule: + - each node of an overlap takes the value of the tile whose **interior it + lies deepest in** (the largest distance, in nodes, to that tile's own + nearest border), **ties by tile name**; NoData still loses to a valid + value; + - disagreement is **no longer refused**. The run **reports each disagreeing + seam** (the two tiles, the number of nodes that differ, the largest and + the median difference) in `--stats` and in a `dem_seams` file field, so a + 25 m jump cannot pass unnoticed; + - the result stays independent of tile order. + Ola will also download a fresh single-date set from hoydedata.no, whose + overlaps should mostly agree (the probe found one same-date pair that does + not); the fresh set is not blocking ("the data is data", Ola). +- **2026-09-28, the seam report ignores differences below 1 mm** (Ola: "Ignore + below 1mm"). A seam counts, and the report lists, only nodes where + |a − b| ≥ 1 mm; float noise such as 7807_1 | 7808_4 (31 nodes, 1.5e-5 m) + and 7910_2 | 7910_3 (4 nodes, 2.4e-7 m) no longer appears. The midline rule + itself is unchanged: which tile's value a node takes does not depend on the + threshold. +- **2026-09-27, Q5 as read after review ("Yes", to the main session's + proposal):** @reviewer found that DTM10's 51-node overlaps make a box that + only reaches into a half-cell tile's overlap strip select that tile and be + refused as mixed-lattice, although the main lattice covers every node (the + design's own 15a acceptance box was refused; 98 of 576 20-km boxes around + the eight tiles). The rule is now, per lattice, whether **its own tiles + cover every node the request needs**: + - exactly one lattice covers it: mesh on that lattice, and drop the other + lattices' tiles from the plan; + - none covers it: refuse, as before (the Q5 message); + - several cover it (a box wholly inside an overlap strip): use the lattice + with **the most tiles in the repository**, ties broken by the name of its + first tile. Header-only and order-independent. + +## Scope + +**15a and 15b (Norway, first).** Close `ROADMAP.md` gap 6 (a DEM in several +tiles) for tiles in one projected CRS, and "inputs in their own CRS" for a +projected DEM. After them, + +```sh +rasputin mesh --dem ../rasputin_data/DTM10_UTM33_20220924/ \ + --domain catchment_wgs84.geojson --tolerance 1 --out catchment.vtk +``` + +selects the tiles the catchment needs out of 254, stitches them, transforms the +catchment from EPSG:4326 into the DEM's EPSG:25833, and meshes it. `--dem +file.tif` alone behaves exactly as today. + +**15c and 15d (the basin, after Norway).** Geographic DEMs (ANADEM, EPSG:4326), +the computation frame, `--out-crs`, and decoding only the needed part of a +2.5 GiB tile. They settle the question Ola put on 2026-09-26, "The questions +will be in which coordinate system we shall do the math" +(`16-domain-polygon.md`, "Ruled by the user"). + +**Not in this increment.** Domain decomposition (increment 21's option D), +a block-sparse raster in C++, tiles in more than one CRS, and resampling. Each +is named in "Not in scope" with the reason it waits. + +## What this builds on + +The parked design, `git show f8cedbe:docs/increments/15-dem-repository.md` +(branch `increment15-dem-repository`, never merged), called "the parked design" +below: rulings R1-R8, questions U1-U4. It was written for Ola's DTM10 archive +before anyone had read the archive's headers. They have now been read (N1-N5), +and ANADEM's too (B1-B9). + +What is carried across, and what changes: + +| parked | here | why | +|---|---|---| +| R1: repository protocol, pure `plan_mosaic`, `assemble` | carried (R1) | still right; a test double is a dict | +| R2: paths only in `io/repository.py` | carried (R2), still a question (Q3) | plus a new `dem_input.py`, so `cli.py` (1101 lines) does not grow | +| R3: `read_meta` split out of `decode_dem` | carried (R3); window decoding added in 15d | an ANADEM tile is 2.5 GiB | +| R4: every tile in the directory on one lattice, or refuse | **changed** (R4): only the **selected** tiles must share a lattice | 8 of Ola's 254 tiles are half a cell off (N2); the parked rule refuses the whole archive | +| R4: canvas origin from the tiles as decoded | **changed** (R4): each lattice has a **reference node**, and every node has a global index | frame coordinates must not depend on the window, for domain decomposition (R12) | +| R5: valid beats NoData, disagreement split down the middle and reported (Q1 revised), order-independent | carried (R5), now **with real data behind it** | DTM10 overlaps between tiles exported on different dates often disagree, up to 52 m (Q1 revised); same-date ones nearly always agree; ANADEM's sampled seams agree (B4) | +| R5: gaps stay NaN | **changed** (R5): a node the request needs that no tile covers is refused | Ola's ruling, `18-row-span-scan.md` R6: "a missing tile inside the extent is a data error" | +| R6: `--bbox` in the DEM's CRS, no transform | carried (R6); the domain is transformed in 15b (R9) | "inputs in their own CRS" | +| R6: `GeoPolygon` lands with the clip | superseded: 16 shipped `DomainPolygon`; 15b gives it a transform | | +| R7: `MAX_MOSAIC_BYTES = 1 GiB`, the canvas copied twice | **changed** (R7): cap at half of physical memory; one canvas, not two | a Norwegian catchment's box passes 1 GiB (R7) | +| R7: window decoding later | 15d (basin) | DTM10 tiles are 100 MB decoded; ANADEM's are 2.5 GiB | +| R8: `--dem` repeatable, `--bbox`, `dem_tiles` | carried (R11) | | +| U1-U4 | carried to the questions (Q1-Q4), U1 now with measurements | | + +## What was measured + +On `increment15-dem-mosaic` at `59024e3`: tifffile 2026.9.20, pyproj 3.8.0 +(PROJ 9.8.1), numpy 2.5.3, Python 3.14.7, on Ola's Mac (32 GiB, +`sysctl hw.memsize` = 34359738368; 10 cores). The scripts are checked in, +so each number can be re-run: + +- `docs/increments/15-probes/dtm10_probe.py headers|seams DIR` (N1-N3), run + on `../rasputin_data/DTM10_UTM33_20220924`; +- `docs/increments/15-probes/anadem_probe.py headers|seams|resample|frame` + (B2-B7). It reads ANADEM by HTTP range requests, so no tile is downloaded + whole, and it needs network access. + +They are measurement scripts, not production code, and nothing imports them. + +### Norway: the DTM10 archive + +- **N1. What the archive is.** 254 `.tif` files (each with a `.tfw` and a + `.aux.xml` beside it), 19 GB. Every file passes `io/geotiff.py`'s own header + checks (the probe calls them, so a refusal would be `decode_dem`'s): + EPSG:25833, 10 m in both axes, area-registered, float32, LZW, tiled, NoData + −32767 from tag 42113. 243 are 5051 × 5051 nodes; the rest are 5052 or 5053 + on a side, and four are short on one side (7305_3 2881 × 5051, 7405_1 + 5051 × 3521, 7405_2 5051 × 3511, 7507_4 4103 × 5052; corrected in review: + the probe's per-axis tallies had been paired into two tiles that do not + exist). Header-only read of all + 254: 0.56 s wall. +- **N2. 246 tiles share one lattice; 8 do not.** Taking the north-west-most + node (x −100250, y 7950250) as reference, 246 tiles sit at integer node + offsets. **Eight are half a cell (5 m) off east-west**, on integer rows: + 7304_1, 7507_4, 7606_2, 7707_1, 7707_3, 7807_2, 7807_3, 7808_3. Their nodes + are at x = …745 instead of …750, and they are 5052 wide and mostly 5053 high. + They look like a second production run. Under the parked R4 (whole directory + on one lattice) `--dem DIR` would refuse the whole archive. +- **N3. Neighbours overlap by 51 nodes, and same-date overlaps agree.** + (Corrected 2026-09-28: tiles exported on different dates can disagree; see + "Ruled by Ola", Q1 revised.) A 5051-node + tile is 50 km plus 510 m, so each edge neighbour shares 51 rows or columns + (a few share 52 or 53; one pair 53). Over 4 neighbour pairs of the first + tile, every overlapping cell is equal: 257 601 of 257 601 on each edge + overlap, 2601 of 2601 on each corner. The control shifts the comparison by + one node: on the two edge overlaps it finds a maximum difference of 18.0 m + and 46.3 m, so the probe can fail there. On the two 51 × 51 corner overlaps + the shifted comparison is also all equal: those corners are flat (sea, most + likely), so for them the control is **not** able to fail. That is 4 pairs of + 869 overlapping pairs. +- **N4. Size.** The union of all 254 tiles spans 155 051 × 125 051 nodes: + 19.4 G nodes, 72 GiB as float32. The tiles hold 6.45 G nodes (24 GiB), so + most of the box is sea or outside Norway. A 2 × 2 block of tiles is + 10 051² = 101 M nodes, 404 MB. +- **N5. The committed benchmark tile is a different release from the + archive's.** `tests/fixtures/dem_archive/7908_3_10m_z33.tif` and the + archive's file of the same name have the same tie point and shape, but + 910 706 of 25.5 M cells differ, by up to 46.1 m, and 2868 cells are valid in + the archive and NoData in the fixture. So a real multi-tile fixture must be + cut from one release, and mixing the committed tile with archive neighbours + is a mixing of releases. + +### The basin: ANADEM + +- **B1. Where ANADEM lives.** `https://hge-iph.github.io/anadem/` lists 53 + tiles, `anadem_v1_.tif` under + `metadados.snirh.gov.br/files/anadem_v1_tiles/`, named by MGRS grid-zone + designator (17L … 25M). The server accepts range requests. Tiles are 0.58 to + 2.09 GB compressed (`curl -I`). The GitHub repository's `LICENSE` is MIT and + speaks of "the Software" (see Q10). +- **B2. What an ANADEM tile is.** Every tile near the basin: EPSG:4326 + (`GTModelTypeGeoKey` 2, `GeographicTypeGeoKey` 4326), **area-registered**, + float32, Deflate, tiled 512 × 512, NoData −9999 in tag 42113, and 4 to 6 + extra pages, all reduced-resolution overviews. Spacing + **0.00026949458523585647° in both axes**: 0.970″, not 1″. It is 30 m of + equator in degrees (`d·π·a/180 = 30.000000000000004`), so ANADEM is **not on + Copernicus's 1″ lattice**. A tile's header costs 64 KiB of reads. +- **B3. The tiles are huge and share one lattice.** They are 6° grid zones, + not 1° tiles: up to 674 M cells, 2.5 GiB as float32. The basin needs about + six, not the research note's "~150". Every tile's offset from 23L is an + integer number of cells to within 2.2e-11 cells, with identical spacing. + Neighbours overlap by 8 or 9 cells, and 24L/24K by 87 rows. Band edges sit + near 8.13°S and 16.26°S, not on MGRS's 8° and 16°. +- **B4. Overlapping cells agree bit for bit.** 23L|24L (the 42°W seam, 4096 + cells) and 23L|23K (the 16.26°S seam, 4088 cells): all equal. The control can + fail: shifted by one column, 34 of 3584 equal and a maximum difference of + 19.9 m; shifted by one row, 129 of 3577 and 3.4 m. In the 87-row 24L|24K + overlap, 40 776 of 44 544 cells are valid in one tile only, and the 3768 + valid in both are equal. So **valid-beats-NoData is needed on real data**. + Three samples of three seams, not the archive. +- **B5. Size.** ANADEM's cell at 7°S / 14°S / 21°S is 29.78 / 29.12 / 28.02 m + east-west and 29.80 / 29.82 / 29.84 m north-south. The basin (636 920 km², + OAS) is **734 M nodes**. Its bounding box (48–36°W, 21–7°S, the research + note's approximation) is **2.31 G nodes, 8.6 GiB as float32**. The basin is + mostly in 23K, 23L, 24K and 24L, with slivers of 23M and 24M. Whether any of + it lies west of 48°W (22K, 22L) is to be read from the BHO polygon; not + checked. +- **B6. What resampling costs.** A real 1536² window of 23K over the Serra do + Espinhaço (43.86–43.45°W, 19.16–18.74°S, heights 592–1682 m, no NoData) was + resampled bilinearly onto square grids in the research note's basin LCC. The + resampled surface was then compared with the source at every inner source + node: + + | target spacing | max | p99.9 | p99 | median | + |---:|---:|---:|---:|---:| + | 30 m | 13.38 m | 5.43 m | 3.07 m | 0.41 m | + | 20 m | 14.06 m | 3.77 m | 2.13 m | 0.28 m | + | 10 m | 8.91 m | 1.96 m | 1.11 m | 0.14 m | + + Control: resampling onto the source's own nodes gives 9.7e-10 m. So a mesh + built to `--tolerance 1` on a resampled grid can be 13 m off the DEM it came + from, and a finer target grid does not remove that. +- **B7. How far a straight edge bends between frames.** Take an edge that is + straight in longitude and latitude, and write it in the basin LCC. Its + midpoint lies off the straight LCC edge by 5 mm at 1 km, 0.14 m at 5 km, + 2.2 m at 20 km and 13.6 m at 50 km (worst direction, at 14°S; within 10 % of + that at 7°S and 21°S). The error grows with the square of the edge length. +- **B8. What refuses an ANADEM tile today.** `io/geotiff.py`'s private checks + on 23L's header: `_single_page` passes (the extra pages are reduced), + `_check_page` gives float32, `_placement` gives the node grid, `_nodata` + gives (−9999, "tag"), and `_projected_epsg` refuses: "GeographicTypeGeoKey + (2048) = 4326 is a Geographic 2D CRS; a projected CRS is required". Nothing + else stands in the way. +- **B9. The core's indices hold at basin size, by grep.** + `include/terrain/raster/` indexes with `std::size_t` (`geometry.hpp:60`), and + `RowSpan` carries `uint32` per axis (`mesh/row_spans.hpp:31`). A grep for + 32-bit flat node indices in `include/terrain/` found none. A run on 2.3 G + nodes is @perf's (15d acceptance). + +Not measured: Deflate or LZW decode throughput for a whole tile, the triangle +count at basin scale. The BHO polygon's CRS was checked afterwards by the main +session (web, 2026-09-27): ANA's metadata gives SIRGAS 2000, **EPSG:4674**, +geographic. ANADEM's *data* licence is still not found: no statement beyond the +repository's MIT software licence turned up, so Copernicus GLO-30's terms for +modified data apply at least (its notice, plus a notice of modification). + +## Prior art: legacy and literature + +### Literature + +- **The refinement method is unchanged**: greedy insertion, Garland and + Heckbert, "Fast polygonal approximation of terrains and height fields", + CMU-CS-95-181, 1995 (increments 14 and 14b). This increment changes which + grid refinement runs on and in which frame, not the method. +- **Barycentric coordinates are invariant under affine maps** (textbook; e.g. + Farin, *Curves and Surfaces for CAGD*, 5th ed., 2002; recalled, not reread). + R8 rests on it. At a given point, the vertical error of a linear triangle is + the same in any affine image of the plane. So a sup-norm guarantee computed + in an affine frame of the DEM's lattice is exactly the guarantee in the DEM's + own coordinates. +- **Bilinear interpolation error** is O(h²·|f''|) for smooth f (e.g. Ciarlet, + *The Finite Element Method for Elliptic Problems*, 1978; recalled). Terrain + is not smooth at 30 m, so the bound says little. B6 measures instead. +- **DEM resampling and reprojection change terrain.** Usery, Finn, Scheidt, + Ruhl, Beard and Bearden, "Geospatial data resampling and resolution effects + on watershed modeling", *J. Geographical Systems* 6:289-306, 2004; Kienzle, + "The effect of DEM raster resolution on first order, second order and + compound terrain derivatives", *Transactions in GIS* 8(1):83-111, 2004. Both + recalled, not reread. They study derived attributes (slope, flow). Our + concern is narrower: resampling changes the surface the sup-norm is measured + against. **Difference:** we do not resample (R8). +- **Mosaicking and warping practice, for reference only (no GDAL).** + `gdalbuildvrt` composites overlapping sources in order, later over earlier. + `gdalwarp` maps every destination pixel back through the inverse transform, + approximated by linear interpolation along rows within an error threshold + (default 0.125 pixel). GDAL documentation, recalled. **Differences:** no order + precedence by source order: a disagreeing overlap goes to the tile the node + lies deepest in, ties by name, and valid beats NoData in any order (R5, Q1 + revised); and no warp (R8). +- **Projection choice**: Snyder, *Map Projections — A Working Manual*, USGS + Professional Paper 1395, 1987, recalled. A Lambert conformal conic suits an + extent wide east-west at mid latitudes, with the standard parallels about one + sixth of the latitude range in from each edge. The research note's LCC (10°S + and 18.5°S over 7–21°S) is close to that rule (9.3°S, 18.7°S). Here it is + only a suggested **output** CRS (R10). +- **ANADEM**: Laipelt et al., "ANADEM: A Digital Terrain Model for South + America", *Remote Sensing* 16(13):2321, 2024. The grid facts used here come + from the tiles' headers (B2), not from the paper. +- **Domain decomposition** prior art (tile-parallel terrain simplification, + *Remote Sensing* 12(3):437, 2020; Linardakis and Chrisochoides 2006) is cited + in `21-parallel-refine.md`. Not designed here (R12). + +**Novelty: none claimed.** One claim could look tempting later: an exact +sup-norm guarantee against a geographic DEM's own nodes, with the mesh written +in a projected CRS. It is not made. Before anyone makes it, search Google +Scholar and IEEE Xplore for "TIN generation geographic coordinates DEM +reprojection error", "terrain simplification latitude longitude grid +projection" and "greedy insertion DEM geographic lattice". No web search tool +was available to this round; ANADEM's facts were read with `curl` and range +requests. + +### Legacy + +```sh +$ grep -rlE 'RasterRepository|get_intersections|add_raster|mosaic' legacy/ +legacy/bindings.cpp +legacy/rasputin/mesh.py +legacy/rasputin/application.py +legacy/rasputin/reader.py +legacy/rasputin/web_visualize.py +legacy/tests/test_gml_repository.py +legacy/tests/test_land_cover_repository.py +legacy/tests/test_raster_repository.py +$ grep -rliE 'reproject|resampl|Transformer|to_crs|transform\(' legacy/ +legacy/rasputin/globcov_repository.py +legacy/rasputin/reader.py +legacy/rasputin/application.py +legacy/rasputin/avalanche.py +legacy/rasputin/geometry.py +legacy/rasputin/gml_repository.py +legacy/tests/test_mesh.py +legacy/tests/test_land_cover_repository.py +legacy/tests/test_gml_repository.py +``` + +- **The repository** (`legacy/rasputin/reader.py:429-468`) was reported on in + `11-raster-ingestion-prior-art.md` §3.4, §4.12, §4.13, §5.8 and §8.4, and + the parked design ruled on it. That ruling is carried unchanged. Carried: + the repository over a directory, header-only footprints, selection of the + covering tiles. Not carried: `glob` order, the unchecked CRS, polygon + subtraction, the `1e-10` stop area, `not touches`. +- **The frame, new here.** `legacy/rasputin/application.py:101-123` transformed + the domain into the raster's CRS, meshed there, and transformed the finished + mesh's vertices into a caller-chosen `target_coordinate_system`. That is the + shape of R8-R10: compute in the DEM's own frame, bring vectors in, send the + result out. **Carried.** Not carried: every CRS was spelled `+init=` and no + `Transformer` passed `always_xy` (increment 11 §9); z went through + `proj.transform(x, y, z)`, which is harmless for a 2D CRS but says nothing; + the bending of edges (B7) was never measured or recorded; and the output CRS + was a proj4 string. +- `legacy/rasputin/geometry.py:235-241`, `GeoPolygon.transform`: the domain's + transform, with the same `+init=` caveat. R9 carries the operation into + `DomainPolygon` through the one reprojection helper. +- `globcov_repository.py`, `gml_repository.py` and `avalanche.py` transform + land cover and avalanche inputs point by point. That is 16b's concern. + +A new `@migration-expert` pass is not needed: no legacy constant is re-derived. + +## Rulings + +Sub-increment tags say where each piece lands: **[15a]** and **[15b]** are +Norway, **[15c]** and **[15d]** the basin. + +### R1. The interface: a narrow repository, pure planning, assembly beside it [15a] + +Carried from the parked R1 with two additions, the lattice key and the +reference node (R4). + +``` +DemRepository (Protocol) storage only + footprints() -> tuple[TileFootprint, ...] header-only, sorted by name + load(name: str) -> DemTile whole tile, decoded + [15d: load(name, window)] + +plan_mosaic(footprints, bounds, needed=None) -> MosaicPlan pure, no pixels +assemble(plan, load) -> Mosaic pixels, no files +``` + +- `TileFootprint`: `name: str`, `meta: RasterMeta`. +- `MosaicPlan` (frozen): the mosaic's `RasterMeta`; the lattice's reference + node and the window's global index offset `(row0, col0)` (R4); per selected + tile, its name, its expected `RasterMeta`, and two index windows (where it + sits in the canvas, and which part of it is used). Every refusal that + headers can decide fires in `plan_mosaic`, before any pixel is read. +- `Mosaic`: the assembled `DemTile` plus the plan, as provenance. +- The repository does not select or assemble. Those steps are the same grid + arithmetic for every storage backend. The protocol stays at two methods, and + sync: `load` is blocking I/O, and an async caller wraps it in + `asyncio.to_thread`. +- **A declarative front, `dem_input.py`.** `DemRequest` (frozen Pydantic: + `sources: tuple[Path, ...]`, `bounds: Bounds | None`, `nodata: float | None`; + 15b adds the domain) goes in, and `open_dem(request) -> DemInput` returns the + tile, the plan and the provenance strings. `cli.py` parses flags into a + `DemRequest` and calls `open_dem`. A GUI backend or an API worker builds the + same request without Typer. `cli.py` is 1101 lines today, and this keeps the + mosaic out of it. + +### R2. Homes, and where paths live [15a] + +| module | holds | imports | +|---|---|---| +| `io/geotiff.py` | `read_meta(stream)`, the header phase (R3) | as today | +| `io/repository.py` | `DemRepository`, `TileFootprint`, `TiffDemRepository` | `io.geotiff`, `io.models` | +| `tin_engine/mosaic.py` | `Bounds`, `IndexWindow`, `MosaicPlan`, `Mosaic`, `MosaicError`, `plan_mosaic`, `assemble` | `io.models`, numpy, shapely | +| `tin_engine/dem_input.py` | `DemRequest`, `DemInput`, `open_dem` | the three above | +| [15b] `tin_engine/crs.py` | `parse_crs`, `reprojector`: the one `Transformer.from_crs` site | pyproj | +| [15c] `tin_engine/frame.py` | `LatticeFrame`, `frame_for` | `io.models`, `crs` | + +- Paths live in `io/repository.py` and `dem_input.py` (which only builds the + repository), and nowhere else below `cli.py`. `read_meta` and `decode_dem` + still take streams. Where the repository lives is still the parked U3, now + Q3. +- `TiffDemRepository.from_directory` lists `*.tif` and `*.tiff` + (case-insensitive), not recursively. That skips DTM10's `.tfw` and + `.aux.xml` side files, which carry nothing the GeoTIFF tags do not already + hold. Paths are resolved, deduplicated and sorted by their string form. +- A file in the listing that fails `read_meta` is refused, naming the file, + and is not skipped. +- Footprints are read once, lazily, and cached. `assemble` checks that each + loaded tile's `meta` equals the planned one ("changed since it was listed"). + +### R3. The header phase [15a], and window decoding [15d] + +- [15a] Carried from the parked R3: `_header(tif, nodata) -> (RasterMeta, + dtype)` holds everything `decode_dem` does before `page.asarray()` + (now `_header` itself, `io/geotiff.py:105-140`). `read_meta(source, *, + nodata=None)` calls it (through `read_header`, which also returns the + decoded dtype, S2) and returns, and `decode_dem` calls it and then decodes. + The header phase keeps the codec refusal. +- [15d] `decode_dem(source, *, window=None)`: with an `IndexWindow`, decode + only the TIFF blocks (or strips) that meet it, and return a `DemTile` of the + window, with its own `x_min`/`y_max`. Both archives are tiled 512 × 512 (N1, + B2), so the blocks come from `page.dataoffsets` and `page.decode`, decoded on + a thread pool: zlib and LZW release the GIL. A window's `DemTile` must equal + the whole tile's `DemTile` sliced to the window, `meta` included (T-window). + +### R4. Planning: the selected tiles on one lattice, or refuse [15a] + +**What changes from the parked R4.** The parked design required every tile in +the repository to share one lattice. Ola's archive has two lattices (N2), so +that rule refuses the whole archive for a catchment that touches none of the +eight odd tiles. Here: + +1. **Footprints are grouped into lattices.** Two tiles are on the same lattice + when they have the same `epsg`, `delta_x`, `delta_y`, `pixel_is_area` and + `nodata`, and their node offsets are integers within `ALIGN_TOLERANCE = 1e-6` + cells. The measured noise is at most 2.2e-11 cells (B3), and exactly 0 for + DTM10. Grouping looks at headers only and is cheap: 254 headers take 0.56 s. +2. **Each lattice has a reference node**: its north-west-most node, the minimum + node `x_min` and maximum node `y_max` over its tiles, each taken as decoded. + A node's **global index** `(R, K)` counts from it. For DTM10's main lattice + that is (x −100250, y 7950250). The reference depends only on which tiles + the repository holds. It does not depend on the request or on file order. +3. **Selection** is by the request's box in the DEM's CRS. A tile is selected + when its node rectangle meets the box, closed on both sides (corrected by + the red suite: when its node *index range* meets the snapped window; see + "Pinned by the red suite (15a)"). **All selected + tiles must be on one lattice** (`refuses_mixed_lattice`). The message names + a tile from each lattice, and the difference: "7707_1 is 0.5 cell (5 m) + east-west off 7707_2's lattice". Different CRS, spacing, registration or + NoData are named as such (the parked `refuses_mixed_crs_mosaic`, + `refuses_mixed_spacing`, `refuses_mixed_registration`, + `refuses_mixed_nodata`). +4. **The window.** `c0 = floor((bx_min − X_ref) / dx)`, + `c1 = ceil((bx_max − X_ref) / dx)`, rows likewise (an edge within 1e-6 cell + of a node is first snapped onto it, so float noise in a spacing like 0.1 m + cannot add a node line), clamped to the lattice's + union. The mosaic's node grid is that index window, snapped outward, so it + covers the box. From here on everything is integer index arithmetic. The + mosaic's `x_min` is computed as `X_ref + c0 · dx`, and its `y_max` likewise. + For DTM10 (integers times 10) that is exact. The parked design's + measurement 4 (split and re-stitch reproduces the origin bit for bit) is + pinned again as M9. +5. **Coverage** (changed from the parked design's "gaps stay NaN", after Ola's + ruling in 18 R6). A node the request **needs** that no selected tile covers + is refused (`refuses_uncovered_nodes`), naming the uncovered area in the + DEM's CRS. The needed region is: + - with `bounds` alone, the whole window; + - with a domain [15b], the domain polygon grown by one cell. Only it is read + (bilinear z of a boundary vertex reads the four nodes around it; 16 R2). + Canvas cells outside the needed region that no tile covers stay NaN, as + filler, and are never read. + + The check is shapely's `covers(union of tile node rectangles, needed)`, in + the DEM's CRS. It is exact enough for rectangles on one lattice: every + corner is a node coordinate. (Corrected by the red suite: the check is per + node, since abutting area-registered tiles leave a gap between their node + rectangles; see "Pinned by the red suite (15a)".) +6. **Refused as well:** an empty repository; a box that meets no tile; a window + under 2 × 2 nodes; a canvas over the memory cap (R7). + +`plan_mosaic` never sees a path, a stream or a pixel. Its test double is a list +of `TileFootprint`. + +### R5. Assembly: stitching and overlaps [15a] + +Carried from the parked R5, with assembly reworked for Q1 revised. + +- The canvas is allocated once, NaN-filled, in `np.result_type` of the + selected tiles' dtypes (float32 for both archives). +- Tiles are loaded one at a time in plan order. Each tile's used window is + written whole into the canvas; where it meets another tile, a copy of its + values there (a strip) is kept; then the tile is dropped. Once all tiles + are in, each overlap is decided from the strips. +- **Overlaps**, per node: + - one value NoData (NaN or the sentinel), the other valid: **valid wins**; + - both NoData: NoData; + - both valid and equal (`==`): accepted; + - both valid and different: **the tile the node lies deepest in wins**, ties + by name, and the seam is reported (Q1 revised, 2026-09-28). Q1 first + refused this; real DTM10 overlaps from different export dates disagree, + so refusing blocked much of the archive. +- The result does not depend on tile order (M10). +- **One tile, no bounds: the loaded `DemTile` is returned as it is.** No + canvas, no copy, so `--dem file.tif` stays bit-identical to today, memory + included. +- The mosaic is one regular node grid, so `raster.to_core`, `grid_domain`, + `refine` and `elevation.trim` run unchanged. Refinement's lattice rule + (increment 14 R1-R2) holds. + +### R6. Which region to mesh [15a] + +The union of the selected lattice by default. `--bbox XMIN YMIN XMAX YMAX` in +the DEM's CRS, as in the parked R6 (Q2). With a domain [15b], the domain's +bounds in the DEM's CRS take `--bbox`'s place, and the two flags exclude each +other (16 R1 already said so). + +Without `--bbox` or `--domain`, `--dem DIR` on Ola's archive selects every +tile, the half-cell ones included, so the mixed-lattice refusal (Q5) fires +before the memory cap (R7) is reached; `test_no_bounds_selects_both_lattices_ +and_is_refused` pins that order. Both refusals are correct, and the lattice +message ends "but a --bbox inside one lattice is meshed". On one lattice alone, +the 72 GiB union box would be refused by the cap, whose message says +"narrow it with --bbox". + +### R7. Memory + +**[15a] One canvas, capped at half of physical memory.** + +- **No second canvas copy.** `DemTile`'s validator copies its array + (`io/models.py:82`), so the parked design peaked at two canvases. `assemble` + builds the canvas privately, so it can hand it over without a copy: a + private constructor in `io/models.py`, `_adopt(meta, array)`. It runs the + same shape and dtype checks, sets the array read-only, and stores it. Its one + caller is `assemble`, on a buffer that `assemble` allocated and never + returns writable. The concurrency rule (increment 11 §7) survives: nobody + else holds a writable reference. +- **The cap** is `physical memory / 2` (`os.sysconf("SC_PHYS_PAGES") * + os.sysconf("SC_PAGE_SIZE")`, which works on macOS and Linux), checked in + `plan_mosaic` before any pixel is read. On Ola's Mac that is 16 GiB, about + 4.3 G float32 nodes. The peak is then the canvas plus one decoded tile (100 + MB for DTM10) plus the mesh. The parked 1 GiB (268 M nodes) would refuse a + large Norwegian catchment's box outright, and the fixed number did not know + the machine. +- **What fits.** A 2 × 2 DTM10 block: 404 MB. A 164 km × 164 km box at 10 m: + 1 GiB. All of Norway at 10 m (72 GiB box, 24 GiB of tiles): no. That is DD's + or a block-sparse raster's problem (R12), not this increment's. + +**[15d] The basin.** Its box is 8.6 GiB at float32 (B5), under the 16 GiB cap +on Ola's Mac. So **the basin is meshable in one piece with a dense canvas on +a 32 GiB machine**, with two conditions: + +- **Window decoding (R3).** Without it, a request touching 23L decodes all + 2.5 GiB of it, and assembly holds the canvas plus a whole tile. With it, the + transient is one tile's window. +- **What must be held:** the canvas (8.6 GiB), the mesh, and refine's + per-triangle state. **What is streamed:** the tiles, one window at a time, + and within a window, blocks on a thread pool. Nothing streams during refine: + the scan runs on many threads over a read-only raster (14 R7), and 18 R6 + names lazy decode inside the scan as a hazard. Nothing here decodes lazily. +- On a 16 GiB machine the cap is 8 GiB and the basin box is refused. The + answers to that are a block-sparse raster that holds only the 512² blocks + meeting the basin (about 2.7 GiB, since the basin is 32 % of its box), or + domain decomposition. Neither is designed here (Q9). + +### R8. The computation frame [15c; basin, after Norway] + +**The question.** The core wants numbers in one Cartesian frame, roughly in +metres, with square cells if 21b's integer incircle is to apply (`dx == dy`, +`21-parallel-refine.md` QW2). ANADEM gives degrees, with cells 2.4 % taller +than wide at 14°S and 6.5 % at 21°S (B5). Three routes: + +- **(A) Resample onto a square grid in a projected CRS** (the basin LCC, UTM + 23S or Polyconic), in Python, chunked. Everything downstream is unchanged. + But the tolerance is then measured against the resampled grid, not the DEM: + on real terrain, 13 m off at the DEM's own nodes (B6). This also breaks 16 + R0 ("a DEM value is the exact height at a point") for the source, and it + costs a second full-size array plus 2.3 G inverse transforms. +- **(B) Mesh in a projected CRS and sample the geographic grid.** The DEM's + nodes are then a curved grid in the mesh's frame. Increment 18's row-span + scan, and 14's lattice rule that every inserted vertex is a DEM node, both + assume the nodes form a regular grid in the frame. This route rewrites the + scan. Rejected. +- **(C) The lattice frame. Recommended (Q6).** Mesh in an affine image of the + DEM's own lattice, bring vectors in by transform (R9), and send the mesh out + by transform (R10). Nothing is resampled. The legacy did this + (`legacy/rasputin/application.py:101-123`). + +**Why (C) keeps the guarantee exactly.** The frame is an affine map of +(longitude, latitude). Barycentric coordinates are affine-invariant, so at every +DEM node the vertical error of the mesh in the frame equals its error in +longitude and latitude. The sup-norm tolerance holds at the DEM's own nodes +with no loss. What the frame changes is only horizontal: triangle shape, and +which triangulation is Delaunay, are judged in the frame, not on the ground. + +**The frame, precisely.** `LatticeFrame` (frozen) holds the DEM's CRS, the +lattice's reference node in it, the DEM spacing `(du, dv)`, and the frame +spacing `(hx, hy)`: + +- **For a projected DEM it is the identity**: `hx = du`, `hy = dv`, and the + frame's origin is the DEM's origin. Everything Norway does is bit-identical + to having no frame at all. +- **For a geographic DEM:** `hx = round_to(du · π·a/180, 2⁻¹⁰)`, and `hy` from + `dv` likewise, where `a` is the CRS ellipsoid's semi-major axis. For ANADEM + that gives `hx = hy = 30.0` exactly (B2). The node with global index + `(R, K)` has frame coordinates `(K · hx, −R · hy)`, and `RasterMeta` handed + to `to_core` carries `x_min = c0 · hx`, `y_max = −r0 · hy`, `delta = h`. +- **Exact.** `h` has at most 15 significant bits and `|K|` stays below 2²¹ + (all of South America is 1.3 M columns). So `K · h` is an exact double, + whatever the window. The same node has the same frame coordinates in every + mosaic and every subdomain (R12). 21b's condition "significant bits of dx + plus `bit_width(max(rows, cols) − 1)` ≤ 53" holds: 4 + 16 for the basin. +- **Square when the DEM's cells are square in degrees**, as ANADEM's are + (`du == dv`). Then `hx == hy`, and 21b's integer incircle applies. The price + is anisotropy: a frame unit is `cos φ`-dependent on the ground. At 21°S it + is 0.934 m east-west against 0.995 m north-south, a ratio of 1.065. That + moves a 45° angle by at most 1.8°. Scale does not matter: the tolerance is + vertical. +- **Anisotropy limit** `MAX_FRAME_ANISOTROPY = 1.10` (Q8): the ratio of ground + metres per frame unit, east-west against north-south, over the request's + extent. Above it, `frame_for` refuses. The basin passes at 1.065. A + geographic DEM of Norway (60–71°N, ratio 2 to 3) is refused, and should be: + the triangles would be judged in a frame stretched two- to threefold. +- **A geographic DEM with `du ≠ dv`** gives `hx ≠ hy`. The kernel then takes + the filtered incircle, which is correct but slower. It is not refused. + +**Who chooses the frame: nobody.** It is a function of the DEM's lattice +alone. It does not depend on the domain, on the request, or on an option. That +is deliberate (R12). + +**The boundary rule, restated.** `project_structure.md`'s `raster` section +says "every number that crosses is metres in a projected CRS, because nothing +on either side can tell". Its reason was a 2:1 anisotropy nobody could see +(increment 11 §3). In 15c it becomes: **every number that crosses is in the +computation frame, which is Cartesian, affine to the DEM's lattice, and within +`MAX_FRAME_ANISOTROPY` of square on the ground over the request's extent.** A +projected DEM satisfies it as today. `to_core(tile, frame)` refuses a +geographic tile without a frame: that is the gate, and it is tested (T-gate). + +**Resampling is designed out, not forgotten.** It is the only route for tiles +in more than one CRS, and for N2's half-cell tiles if they must be meshed +together with their neighbours. If Ola wants either, it is its own increment, +and its guarantee is against the resampled grid, stated as such. + +### R9. Inputs in their own CRS + +**[15b] For a projected DEM (Norway).** The domain, and later 16b's features, +come in any CRS pyproj can parse. They are transformed into the DEM's CRS +before anything else sees them. + +- **One helper**, `crs.reprojector(src, dst) -> Callable[[xy], xy]`, holds the + only `Transformer.from_crs` in `src_python/`, always with `always_xy=True`. + Increment 11 §9's grep test already guards every `from_crs` call site; it + now has one real site to find. A GeoTIFF's model space is (x = easting or + longitude, y = northing or latitude) whatever the EPSG axis order, so + `always_xy` is the convention the tiles themselves use. +- `DomainPolygon` keeps its CRS (`epsg: int` becomes a `crs: str`, the text + pyproj parsed, since a domain may have no EPSG code). A new + `DomainPolygon.to_crs(dst) -> DomainPolygon` transforms **vertices only**, + and rings stay straight in the destination. This is Ola's "already discrete" + direction (16, direction 3): vertices are used as given, and an edge is the + straight segment between them in the frame the mesh is built in. Within one + UTM zone the edges bend by millimetres per kilometre of edge (B7's order of + magnitude). Not densified. +- `check_crs`'s must-match rule (`domain.py:97`) is replaced by the transform. + It was kept as "one replaceable function at the Python boundary" for exactly + this (16, "Ruled by the user"). The extent check (16 R1) now runs after the + transform, against the mosaic's coverage (R4 point 5), not one tile's + rectangle. +- A GeoJSON file without a `crs` member is EPSG:4326 (RFC 7946), and is now + transformed rather than refused. +- **Provenance:** the `.vtk` records `domain_crs` (the input's CRS) and + `domain_transform` (the pyproj transformer's `description`), so a datum + shift chosen by PROJ is on record. +- Snapping: the noder's 1 mm grid (16 U6) applies after the transform, in the + DEM's CRS, as today. + +**[15c] For a geographic DEM.** Vectors go through the same helper into the +DEM's geographic CRS (longitude, latitude), then through the frame's affine map +into the frame. A domain given in longitude and latitude, like BHO's is +likely to be, has edges straight in the frame, exactly. + +### R10. The output CRS [15c; basin, after Norway] + +- **`--out-crs TEXT`** (anything `pyproj.CRS.from_user_input` accepts). + - Projected DEM: optional, and the default is the DEM's CRS. The output is + unchanged from today, and nothing is transformed. + - Geographic DEM: **required** (Q7). The refusal message prints a suggested + LCC fitted to the domain's extent (Snyder's one-sixth rule, central + meridian at the middle), so the user can copy it. +- **What is transformed:** vertex (x, y) only, frame → DEM CRS → `--out-crs`. + z is untouched. Triangles and edges keep their indices. +- **What it costs, recorded rather than hidden.** An edge straight in the + frame is not straight in the output CRS (B7: 0.14 m on a 5 km edge). The + vertical effect at a point is about the horizontal displacement times the + triangle's slope. Per triangle, `δ_T = max over its edges of |T(midpoint in + frame) − midpoint of T(ends)|`, and `ε_T = δ_T · |∇z_T|` in the output CRS. + That is one extra transform per edge. The `.vtk` records + `max_reprojection_z_error_estimate` = max over triangles of `ε_T`. It is an + estimate (midpoints, not a bound), and it says so. When `--out-crs` is the + DEM's own CRS, the output map is affine, and the field is exactly 0. +- **Recorded CRS:** `EPSG:n` when pyproj finds an exact EPSG code, otherwise + single-line WKT2 (ASCII, which `write_vtk` requires). +- `--stats` (17) measures angles and areas in the output coordinates, since + that is the mesh the user gets. + +### R11. The CLI and what the file records [15a, 15b] + +Carried from the parked R8: + +- **`--dem` becomes repeatable**: exactly one directory, or one or more files. + A directory mixed with files, or two directories, is a usage error. +- **`--bbox XMIN YMIN XMAX YMAX`**, only with `--dem`; excludes `--domain` + [15b]. +- Refusals from `plan_mosaic` and `assemble` (`MosaicError`, a `ValueError`) + become `typer.BadParameter(param_hint="--dem")`, like `GeoTiffError` today. +- **Fields:** `crs` as today; for a mosaic, `elevation_source` is prefixed + with `mosaic of N tiles, R x C nodes; `, and `dem_tiles` lists the file names + (`; `-separated, escaped with `encode("ascii", "backslashreplace")`). [15b] + adds `domain_crs` and `domain_transform`. [15c] adds `computation_frame` + (for example `lattice of EPSG:4326, h = 30 m, reference lon -48.00089 lat + -8.12998`) and `max_reprojection_z_error_estimate`. +- stderr: one line, `mosaic of N tiles, R x C nodes`. + +### R12. What domain decomposition needs from this, and what must not block it + +Domain decomposition is not designed here (`21-parallel-refine.md` Q5 moved +it to the large-area work). What 15 settles so that it stays open: + +- **One frame for the whole DEM.** It is a function of the lattice alone + (R8), never of the domain, the window or an option. Every subdomain uses the + same frame. +- **Global node identity.** `(R, K)` from the lattice's reference node (R4). + Two subdomains that share a seam agree on every seam node's index, and, by + R8's exactness, on its frame coordinates bit for bit. +- **The raster for a subdomain is a plan.** `plan_mosaic(footprints, bounds)` + is pure. A subdomain's box gives its own window, under its own memory cap. +- **Overlaps resolve the same way in every subdomain**, because R5 does not + depend on order. +- **Vectors are transformed once**, before any partition, so a domain vertex on + a seam has one frame position. +- **The output CRS is explicit** (R10), not fitted per domain. A fitted + default would put two subdomains, or two catchments, in different CRSs. + +Choices rejected partly because they would block it: resampling per window +(neighbouring windows would disagree at seams); a computation CRS fitted to +each domain; order-dependent overlap resolution; a frame origin at each +mosaic's corner (the same node would get different coordinates); decoding +lazily inside the parallel scan. + +## Invariants + +- **I1. One lattice per mosaic.** Every node of an assembled mosaic lies on + the selected lattice. Any selected tile off it is a refusal, never resampled + or snapped. +- **I2. Order independence.** `plan_mosaic` and `assemble` give the same + result for every permutation of the footprints. +- **I3. Valid beats NoData; disagreement is decided and reported** (Q1 + revised, 2026-09-28; first "disagreement refuses"). No overlapping node is + ever chosen between two different valid values without the pair appearing + in the seam report. +- **I4. Coverage.** Every node the request needs comes from a tile. NaN filler + exists only outside the needed region. +- **I5. Split and re-stitch is the identity.** A grid cut into tiles + (any overlap, either registration) and planned without bounds reassembles to + the original `DemTile`, `meta` and array equal with `==`. +- **I6. Header before pixels.** Every refusal that headers can decide fires + before any `load`. +- **I7. Single-file path unchanged.** `--dem file.tif` output and memory are + bit-identical to before 15a. +- **I8. [15b] One reprojection site**, with `always_xy=True`. A vector's + vertices are transformed exactly once. +- **I9. [15c] The frame is a function of the lattice alone**, and exact: a + node's frame coordinates are `(K · hx, −R · hy)` bit for bit, in every + window. +- **I10. [15c] No degrees cross into `_core`.** `to_core` refuses a geographic + tile without a `LatticeFrame`. +- **I11. [15c] The guarantee is unchanged.** With the lattice frame, + `--tolerance` holds at every valid DEM node inside the domain, measured in + the DEM's own coordinates. + +## Degeneracy policy + +- **Overlap width.** Any width, including none (area-registered neighbours + that abut: disjoint node sets), one shared line (point-registered), 51 nodes + (DTM10), 87 rows (ANADEM 24L/24K), or a tile wholly inside another. One rule + covers all (R5). +- **The same file twice** under two names: every overlap is equal, so it is + accepted. The recorded tile list shows both. +- **A tile touching the box on one node line** is selected, and contributes + that line. +- **Float noise in tile offsets** up to `ALIGN_TOLERANCE = 1e-6` cells. Measured + noise is at most 2.2e-11 (B3). +- **Half-cell offsets** (N2): refused by name, not snapped. Snapping would move + every value of a tile by 5 m, which is resampling. +- **Spacings not exact in binary** (0.1 m, or ANADEM's degrees): the tolerance + above, then integer windows (parked M3). +- **A domain vertex on a tile seam, or a bilinear cell straddling a seam:** + nothing special. The canvas is one array. +- **[15b] Axis order.** Every transform is `always_xy`. A GeoJSON with + latitude first is a wrong file, not a case, and is refused by the extent + check. +- **[15b] Datum shifts** (SIRGAS 2000 or ETRS89 to WGS 84): whatever + transformation PROJ picks, recorded in `domain_transform`. +- **[15c] The antimeridian and the poles.** A lattice whose extent crosses + ±180° is refused. The poles are refused by the anisotropy limit. +- **[15c] `du ≠ dv`** (Copernicus above 50° uses wider longitude spacing): + allowed, with the filtered incircle. Tiles of different spacing in one + request are refused as mixed spacing. + +## Not in scope + +- **Tiles in more than one CRS, and the eight half-cell tiles meshed with + their neighbours.** Both need resampling, and R8 explains what that costs. + A catchment inside the eight's own lattice meshes; one straddling the two + lattices is refused by name. +- **Domain decomposition**, and all of Norway at 10 m in one mesh (R7, R12). +- **A block-sparse raster in C++** (18 R6's `row_segments` source). Needed + only to mesh the whole basin on a 16 GiB machine; Q9. +- **Recursive directory walks, remote storage, an async repository API.** +- **Vertical datums.** Heights pass through unchanged, as today. + +## Sub-increments and LOC + +Counted in `CLAUDE.md` §2's unit (production lines, comments and docstrings +excluded, tests excluded). Estimates. Increment 10 came in 39 % over its +estimate and 14 came in 17 % over; the worst case below applies 39 %. + +**Norway first (Ola, 2026-09-27).** 15a and 15b ship on their own and in this +order. 15c and 15d follow later. + +| | what | est. | worst | +|---|---|---:|---:| +| **15a** | **Norway: many tiles, one projected CRS** | | | +| | `io/geotiff.py`: `_header`, `read_meta` | 20 | | +| | `io/repository.py`: protocol, footprint, listing, cache, load | 60 | | +| | `mosaic.py` types: `Bounds`, `IndexWindow`, placements, plan, `Mosaic`, error | 40 | | +| | `plan_mosaic`: lattice grouping, reference node, selection, coverage, cap | 105 | | +| | `assemble`: canvas, overlap rule, single-tile fast path, meta check | 55 | | +| | `io/models.py`: `_adopt` | 15 | | +| | `dem_input.py`: `DemRequest`, `open_dem` | 35 | | +| | `cli.py`: `--dem` list, `--bbox`, refusals, fields | 45 | | +| | **15a total** | **375** | **520** | +| | *15a as built, measured at review (branch diff, CLAUDE.md §2 unit): 533, of which `mosaic.py` 325 against 200 estimated* | | | +| | *15a after Ola's Q1 revised (cba0073): 596 added against master, of which `mosaic.py` 372 against 200 estimated* | | | +| | *15a at the tip (a253a77): 597 added against master, blank lines excluded (`CLAUDE.md` §2 as ruled by Ola 2026-09-28); 554 net* | | | +| **15b** | **Norway: the domain in its own CRS** | | | +| | `crs.py`: `parse_crs`, `reprojector` | 30 | | +| | `domain.py`: `crs: str`, `to_crs`, extent against coverage; `check_crs` removed | 40 | | +| | `dem_input.py`: domain to bounds and the needed region | 20 | | +| | `cli.py`: `--domain` with a mosaic, `domain_crs`, `domain_transform` | 25 | | +| | **15b total** | **115** | **160** | +| **15c** | **Basin, after Norway: geographic DEMs and the frame** | | | +| | `io/geotiff.py`: accept geographic 2D, degree axes | 30 | | +| | `io/models.py`: `RasterMeta` CRS kind | 10 | | +| | `frame.py`: `LatticeFrame`, `frame_for`, anisotropy, affine maps | 95 | | +| | `raster.py`: `to_core(tile, frame)`, the gate | 15 | | +| | output: `--out-crs`, transform, bending estimate, WKT2 field, stats on output | 85 | | +| | **15c total** | **235** | **325** | +| **15d** | **Basin, after Norway: window decoding and basin memory** | | | +| | `io/geotiff.py`: window decode, block selection, thread pool | 55 | | +| | `io/repository.py`, `assemble`: windows through `load` | 20 | | +| | **15d total** | **75** | **105** | + +Each is under 700 even at its worst case. 15a and 15b together would be 490, +or 680 at the worst case: too close to the ceiling to merge, and Ola asked for +Norway to ship on its own anyway. No sub-increment touches C++, so the +refine/mesh acceptance rule (README, "Acceptance") does not formally apply. The +basin run in 15d's acceptance is @perf's all the same, because it is the first +run at 30× the benchmark. + +**Documentation changed in the same PRs** (not counted): +`project_structure.md` (layout; the `raster` boundary rule in 15c, R8); +`11-raster-ingestion.md` §3 and §10 (15c; `GeoPolygon` superseded); +`16-domain-polygon.md` U1 (15b: the must-match rule replaced); `io/__init__.py`'s +docstring (Q3); `ROADMAP.md` rows 15a-15d and the "inputs in their own CRS" +item. + +## Tests for `@tester` + +Micro-TIFFs come from `tests/python/geotiff_fixtures.py`. Planning and assembly +tests build `RasterMeta` and `DemTile` directly and use a dict as the +repository. The parked design's lists M1-M12, F1-F4, G1-G2 and C1-C6 are +carried as written there, with the changes below. + +**Invariant-critical, and mutation testing required:** + +- **15a: `test_mosaic.py`** (planning and assembly). The index arithmetic, the + lattice grouping and the overlap rule are where a wrong answer is a silently + wrong terrain. +- **15c: `test_frame.py`.** The frame's exactness and the no-degrees gate are + where a wrong answer is a silently wrong guarantee. + +15b and 15d are ordinary suites. 15b's axis-order cases (below) are the ones +that matter there. + +**15a, changed or new against the parked list:** + +- M2 becomes lattice grouping: two lattices in one repository are accepted; + a request inside either one plans; a request straddling both is refused, + naming a tile of each and the offset (0.5 cell, as in N2). +- New M13, the reference node: the same tile's global index is the same + whether planned alone, with neighbours, or under any `bounds`. The mosaic's + `x_min` equals `X_ref + c0 · dx` bit for bit. +- New M14, coverage: a hole in the tile set inside the box is refused, naming + the area. The same hole outside the needed region is NaN filler and is + accepted. +- M5 becomes the cap: set from a patched physical-memory function, not a + constant, and `load` is never called. +- New M15: `_adopt` is reachable only from `assemble`. The mosaic's array is + read-only, and `np.shares_memory` holds between the canvas and the result + (no copy), while the single-tile path still returns the loaded tile itself. +- T-real (the parked design's, `needs_codecs`): the committed benchmark tile + cut into quadrants, 1-pixel overlap, meshes exactly like the whole tile. + +**15b:** + +- The domain in EPSG:4326 GeoJSON (no `crs` member), in EPSG:25832, and in + WKT with `--domain-crs EPSG:3035`, each over a 25833 mosaic: the transformed + vertices equal pyproj's own `always_xy` transform, and the mesh is built. +- **Axis order, able to fail:** a test transform without `always_xy` puts a + vertex outside the DEM, and the extent check refuses it. Increment 11 §9's + grep test finds exactly one `from_crs` site. +- `domain_crs` and `domain_transform` are recorded. A domain already in the + DEM's CRS is not transformed, and the mesh is bit-identical to 16's. + +**15c:** + +- T-frame: for random lattices (spacing, offset, size), `(K · h, −R · h)` + equals the planned `RasterMeta`'s node coordinates bit for bit, in every + window. +- T-invariance: an analytic plane `z = a + b·lon + c·lat` on a geographic + micro-mosaic meshes to two triangles with zero error, at any tolerance. +- T-gate: `to_core` of a geographic tile without a frame raises. +- T-anisotropy: a geographic extent reaching 20°S passes, and one reaching 30°S + or 60°N is refused (the ratio is about 1/cos φ: 1.06, 1.15, 2.0). +- T-out: `--out-crs` equal to the DEM's CRS gives + `max_reprojection_z_error_estimate` exactly 0; an LCC gives a positive value + that grows with edge length. + +**15d:** + +- T-window: for every window, including windows on block edges and one-node + windows, `decode_dem(s, window=w)` equals `decode_dem(s)` sliced to `w`, + `meta` included. Tiled and stripped micro-TIFFs. +- The mosaic built with windows equals the one built from whole tiles. + +### Pinned by the red suite (15a) + +Names and behaviours the design left open, fixed by the red commit's suites +(`tests/python/test_mosaic.py`, `test_io_repository.py`, `test_io_read_meta.py`, +`test_dem_input.py`, `test_cli_mesh_mosaic.py`). All were run green against a +scratch implementation that was not committed; `test_mosaic.py` also killed +39 of 39 mutants of it. + +**Two corrections to R4.** Both were found by the scratch implementation +failing M1's area-registered case. + +- *Selection* (point 3) is by **index window**, not by the tile's node + rectangle meeting the box. A tile is selected when its node index range + meets the outward-snapped window, closed on both sides. The rectangle rule + leaves nodes uncovered: area-registered neighbours at x 0..50 and 60..110 + with `bx_max = 55` snap to column 60, which only the east tile holds, and + that tile's rectangle does not meet the box. +- *Coverage* (point 5) is **per node**: a node of the window that no selected + tile covers is uncovered. The design's check was shapely's `covers` over + the union of node rectangles, and it would refuse every pair of abutting + area-registered tiles: their node rectangles are one spacing apart, so the + union has a gap where no node lies. + +**`tin_engine/mosaic.py`.** + +- `ALIGN_TOLERANCE = 1e-6`; `MosaicError(ValueError)`; + `physical_memory() -> int`, which is `SC_PHYS_PAGES * SC_PAGE_SIZE`, looked + up at call time so a test can patch it. +- `Bounds(x_min=, y_min=, x_max=, y_max=)`: a non-finite, inverted or + zero-extent box raises `ValueError`. +- `IndexWindow(row0, col0, rows, cols)`. `TilePlacement(name, meta, canvas, + source)`: `canvas` is in canvas indices, `source` in the tile's own. + Without bounds, `source` is the whole tile. +- `MosaicPlan(meta, reference, window, tiles)` is a frozen **Pydantic** model, + because the tests reorder `tiles` with `model_copy`. `reference` is + `(X_ref, Y_ref)`. `window` is the mosaic's first node as a global index, + plus its shape. `tiles` is sorted by name. A tile's global origin is + `window.row0 + canvas.row0 - source.row0`, and the same for columns. +- `Mosaic(tile, plan)`. `plan_mosaic(footprints, bounds=None, needed=None)` + takes exactly those parameters. `needed` is a shapely geometry in the DEM's + CRS, and it is closed: a node on its boundary is needed. Growing a domain by + one cell is the caller's job (15b). +- **The cap** counts **the planned decoded dtype**: the itemsize of + `np.result_type` over the *selected* footprints' `dtype`, and refuses when + `rows * cols * itemsize > physical_memory() // 2`; the refusal names that + dtype (`float64`) and `--bbox`. A float64 tile outside the window does not + count. The cap is on the window, not on the lattice's union. (Amended after + review, S2; the red suite had pinned 4 bytes a node, which under-counted a + float64 mosaic by 2×.) +- **The mosaic's `RasterMeta`**: `nodata_source` comes from the first selected + tile by name, and `vertical_unit_assumed` is true if any selected tile's is. +- **NoData against NoData:** two NaNs give NaN, and two sentinels give the + sentinel (I5 needs both). NaN against the sentinel is NoData, and which of + the two is not ruled, only that every order gives the same answer. +- **What messages name** (a case-insensitive substring match): + - a lattice straddle: both tile names, `0.5 cell`, and `east-west` or + `north-south`; + - a CRS difference: both EPSG codes; + - a spacing difference: `spacing` and the odd value; + - a registration difference: `registration`; + - a NoData difference: `nodata` and both values, with `None` for an absent + sentinel; + - an overlap disagreement: *no longer a refusal* (Q1 revised); what it + named — the two tiles whose *values* disagree, not a tile that merely + covers the node (B2), the count and the largest difference — is now the + seam report's (below); + - uncovered nodes: the node bounding box of the uncovered nodes, with each + coordinate written out, not in scientific notation; + - a changed tile: `changed since it was listed`; + - the cap: `--bbox`, and the planned dtype. + +**Amended after review (tests after green, S1-S3, B1, B2).** Pinned by the +test amendment that follows green `ff7cc8d`: + +- **Q5 as read after review (B1),** in `TestB1LatticeByCoverage` + (`test_mosaic.py`), the real-extract cases in `test_dem_input.py`'s + `TestRealDtm10`, one CLI case in `test_cli_mesh_mosaic.py`, and the + acceptance box against Ola's archive in `test_io_repository.py`'s + `TestB1RealArchive` (headers only; skipped where the archive is absent). When + one lattice's tiles are selected, nothing changes. When several are, **the + nodes a lattice must cover** are its nodes in the request's box, snapped + outward, clamped to the bounding box of every selected tile (any lattice) — + *not* to that lattice's own union, or a box running past it into the other + lattice's tiles would be silently cut short. Without a box, that is the + bounding box of every tile. So a bare `--dem DIR` over two lattices is still + refused as mixed-lattice (`test_no_bounds_selects_both_lattices_and_is_refused` + keeps its outcome), as are all the other two-lattice refusals in M2 and C4. + The chosen lattice's plan equals the plan of its tiles alone. The count is of + a lattice's tiles in the repository, not of those the box selects; "its first + tile" is its tile whose name sorts first. +- **The design's acceptance box selects nine tiles, not four.** With the + 51-node overlaps, the index-window selection (the first correction above) + also selects the main-lattice neighbours 7807_1, 7808_2, 7809_3, 7809_4 and + 7909_3, whose overlap strips reach into the 2 × 2 block. The archive test + asserts the four are in the plan, 7807_2 is not, and the window is + 10051 × 10051; the Acceptance's "`dem_tiles` listing four files" is not + pinned. Whether to drop tiles that only duplicate covered nodes is open. +- **The coverage refusal's memory (S1).** Refusing a mostly uncovered window + with no `needed` peaks, by tracemalloc, below the window's float32 canvas + (index and coordinate arrays of the uncovered nodes cost 8× it). The + `needed` path is not pinned. +- **The edge snap (S3)** has its test: at spacing 0.1, the box edges 0.3, 0.7 + and 0.9 add no node line. + +**Amended for Ola's Q1 revised (2026-09-28).** Pinned by the test amendment +that follows it (`15a tests: overlaps split down the middle and reported`): +`TestQ1DeepestInterior` and `TestQ1SeamReport` in `test_mosaic.py`, four +rewritten tests of its `TestM8Overlaps`, `TestQ1Seams` in +`test_cli_mesh_mosaic.py`, and the real-seam cases of `test_dem_input.py` and +`test_cli_mesh_mosaic.py`. All were run green against a scratch implementation +that was not committed; `test_mosaic.py` killed 16 of 16 mutants of it. + +- **Depth** of a node in a tile is `min(r, c, rows - 1 - r, cols - 1 - c)` in + the tile's own indices, `rows x cols` being the **whole tile's** `meta` — + not the window a `--bbox` uses of it, and not the mosaic's. 0 on the + border. Of the tiles holding a **valid** value at a node, the deepest + gives it; equal depths go to the tile whose name sorts **first** (Python + `str` order, as the plan's). NoData against NoData is unchanged (two NaNs + NaN, two sentinels the sentinel, NaN against the sentinel either, the same + in every order). So one shared line (point-registered neighbours) is all + ties, and a tile wholly inside a bigger one takes only the ties. +- **`Seam`** in `tin_engine.mosaic`, with `first`, `second` (names, + `first < second`), `nodes` (int), `largest` and `median` (float). + `Mosaic.seams` and `DemInput.seams` are tuples of them, sorted by + `(first, second)`, empty for one tile and when every overlap agrees. +- A seam is a **pair** of tiles. `nodes` counts the mosaic's nodes (inside the + window, never a node a `--bbox` leaves out) where both hold a valid value + and the two differ (`!=`, so one ulp counts; superseded by the 1 mm + threshold below). `largest` and `median` are of + `|a - b|` over those nodes only, in float64; the median of an even count is + the mean of the middle two (`np.median`). A node in three tiles counts once + in each disagreeing pair. Pairs that agree are not listed. +- **`dem_seams`** (file field) is recorded exactly when `dem_tiles` is: + `none` when no pair disagrees, else one entry per pair in `seams` order, + `; `-joined and escaped like `dem_tiles`: + ` | : nodes , max , median `, for + example `ne.tif | nw.tif: nodes 1, max 4, median 4`. One entry per + disagreeing pair, not per node, so it stays small. +- **`--stats`** has a `## DEM seams` section, between `## Sizes` and + `## Quality (plan view, x/y)`, only when some pair disagrees: a table + `| tile | tile | nodes | max | median |`, one row per entry, numbers + formatted as in `dem_seams`. +- Not pinned: a stderr line for a disagreeing seam, a cap on the number of + entries, and where the report is computed (the scratch kept each pair's + overlap strips, which needs no second load). + +**Amended for Ola's 1 mm threshold (2026-09-28).** Pinned by the test +amendment `15a tests: the seam report ignores differences below 1 mm; table +cells escaped`: `TestSeamThreshold` in `test_mosaic.py`, the oracle +`seams_of` in `mosaic_fixtures.py` (`SEAM_THRESHOLD`), and two tests of +`test_cli_mesh_mosaic.py`'s `TestQ1Seams`. Run green against a scratch +implementation that was not committed; three mutants of it killed (`>` for +`>=`, the threshold consulted by the midline decision, the median over every +differing node). + +- A seam counts a node where both tiles hold a valid value and + `|a - b| >= 0.001` in float64 (the DEM's units), replacing `!=`. `nodes`, + `largest` and `median` are over those nodes only; a pair with none is not + listed, so `dem_seams` is `none` and `--stats` has no `## DEM seams` + section when no pair qualifies. The boundary is pinned with float64 tiles + holding 0.0 against `0.001` (counts) and against + `np.nextafter(0.001, 0.0)` (does not), in either tile. +- The midline decision does not look at the threshold: 0.5 mm apart, a node + still takes the deeper tile's value, ties by name, in every assembly order. + `TestM8Overlaps.test_one_ulp_is_a_disagreement` became + `test_one_ulp_is_below_the_threshold_and_still_decided_by_depth` (no seam; + the value is still the tie's `e.tif`). +- `TestRealDtm10.test_q1_the_seam_shifted_by_one_cell_is_reported_and_split` + (`test_dem_input.py`) is unchanged but follows the oracle: 13 of the + shifted real seam's 12,800 differing nodes are below 1 mm. +- **Table cells:** a `|` in a tile name is written `\|` in the + `## DEM seams` table, so the row keeps five cells. The `dem_seams` field is + not a table and keeps the name as listed (`ne.tif | n|w.tif: ...`). Which + layer escapes is not pinned. + +**`io/models.py`.** `DemTile._adopt(meta, array)` is a classmethod. It raises +`ValueError` on a shape, dtype, ndim or non-C-contiguous mismatch, sets the +**passed** buffer read-only, and keeps it without a copy. Only `io/models.py` +and `mosaic.py` contain the string `_adopt`. + +**`io/repository.py` and `io/geotiff.py`.** + +- `read_meta(source, *, nodata=None)`. +- `TileFootprint(name=, meta=, dtype=)`, where `name` is the file's name + (`path.name`) and `dtype` the numpy dtype the tile decodes to (a `np.dtype`, + default float32), filled by the repository from the header through + `geotiff.PROMOTION` (S2). +- `TiffDemRepository(paths, *, nodata=None)` and + `TiffDemRepository.from_directory(directory, *, nodata=None)`. Construction + reads no file. +- Refusals: + - two paths with one file name: a `ValueError` naming the name; + - an empty directory: a `ValueError` naming the directory; + - a file that fails `read_meta`: a `GeoTiffError` naming the file; + - `load` of an unknown name: `KeyError`. +- Every `open` in `repository.py` uses mode `"rb"`. +- No other `io/` module calls a file opener. The guard is an AST scan whose + own scanner is tested on planted source. +- The docstring of `io/__init__.py` names `repository.py` and no longer says + "Nothing here opens a file." + +**`tin_engine/dem_input.py`.** `DemRequest(sources=, bounds=, nodata=)` is +frozen. A directory mixed with files, or two directories, raises a +`ValueError` whose message contains "director". No sources also raises. +`DemInput` has `tile`, `plan` and `label`. `label` is the directory's name, or +the stem of the first file as given. + +**`cli.py`.** + +- `--bbox` needs a short `metavar` (the scratch used `BOX`). Typer's default + `` widens the help's type column until + `--no-constraint-feet` is cut off at 80 columns, which fails the existing, + unedited `test_cli_constraint_feet.py::TestTheFlag::test_the_help_names_the_flag`. +- `--bbox` without `--dem`, or with an invalid box, is a usage error naming + `--bbox`. `--bbox` on a single file meshes a window of that file. +- **`dem_tiles`** holds the selected tiles' names, sorted and `; `-joined. It + is recorded whenever `--dem` is a directory or several files, even when only + one tile is selected, because the label is then the directory's and the file + used must be on record. It is never recorded for a single file. +- The `mosaic of N tiles, R x C nodes; ` prefix and the stderr line appear only + for N ≥ 2. + +**Fixtures.** `tests/fixtures/dtm10/`, cut by `tests/fixtures/dtm10/extract.py` +from Ola's archive (`DTM10_UTM33_20220924`), one release, © Kartverket, CC BY +4.0 (checked by the main session, 2026-09-27, against Geonorge's metadata API +for dataset `dddbb667-1303-4ac5-8640-7ec04c0e3918`: "Åpne data", CC BY 4.0): + +- `seam/`: 6400_4 | 6400_1, rows 3072-3327, 307 columns each, with a 51-column + overlap. It agrees bit for bit, and a one-column shift disagrees; the script + asserts both. +- `lattices/`: 7707_1 over 7707_2, 96 × 96 each, 5 m east-west apart. + +Deflate, 0.6 MB together. + +## Test data + +**Norway (15a, 15b).** + +- *Synthetic:* micro-TIFF mosaics, 2 × 2, overlap 0, 1 and 3, point- and + area-registered; a two-lattice repository with one tile shifted half a cell. +- *Real, split:* the committed benchmark tile cut into quadrants (above). +- *Real, seam:* an extract of a real DTM10 edge overlap from Ola's archive, + for example 6400_1 | 6400_4 (N3's first edge pair). Two windows of about + 256 × 307 nodes (256 plus the 51-node overlap), with their real + georeferencing, Deflate-compressed: well under 1 MB. **Cut both from the + archive, never one from the committed tile** (N5). Same attribution as the + committed tile. The extraction script is test support and goes under + `tests/`, as @tester's. +- *Real, two lattices:* an extract of one of N2's eight tiles and an aligned + neighbour, for `refuses_mixed_lattice`. @tester picks the pair with + `dtm10_probe.py headers`. +- *@perf, Norway:* the 2 × 2 block 7908_3, 7908_2, 7808_4, 7808_1 from the + archive (all on the main lattice): 10 051² = 101 M nodes, 404 MB float32, + 4× the benchmark tile. Local only, not committed. + +**The basin (15c, 15d; after Norway).** + +- *Synthetic:* a geographic micro-TIFF with ANADEM's exact header numbers + (EPSG:4326, area-registered, spacing 0.00026949458523585647, tie point + (−48.00102804928458, −8.129843152810082), NoData −9999). `micro_tiff` needs + a geographic-CRS parameter, which is test code. +- *Real, seam:* extracts of 23L | 24L (the 42°W seam, bit-equal overlap) and + 24L | 24K (the 87-row overlap with NoData on one side), cut by range reads + as in `anadem_probe.py seams`. Four 512² blocks, about 2.4 MB compressed. + Whether they may be committed is Q10. +- *@perf, a real piece of the basin:* **46–44°W × 17.26–15.26°S**, across + the 23L/23K seam, over the São Francisco near Pirapora–Januária and the + Paracatu. It is believed to lie inside the basin; @perf checks that against + BHO. 7422 × 7422 = **55 M nodes, 210 MiB float32**, about 2.2× the + benchmark. About 240 blocks by range reads, roughly 150 MB of download + (from the measured 0.5–0.85 MB per block). The Espinhaço window of B6 (2.4 M + nodes, 45 km, steep) is a quick steep-terrain case. +- *@perf, the whole basin:* the four main tiles plus slivers, about 8 GB of + download; the box is 8.6 GiB at float32 (B5). + +## Acceptance + +- **15a:** `rasputin mesh --dem ../rasputin_data/DTM10_UTM33_20220924/ --bbox + … --tolerance 1` on the 2 × 2 block writes a `.vtk` that ParaView opens, + with `dem_tiles` listing the four block tiles among the main-lattice tiles + the index window selects (nine: with 51-node overlaps it also takes 7807_1, + 7808_2, 7809_3, 7809_4 and 7909_3; corrected at the 15a test amendment), and + not 7807_2, whose lattice does not cover the box (Ola's Q5 reading). A box + that straddles one of N2's odd tiles and a neighbour so that neither lattice + covers it is refused, naming both. The quadrant split of the benchmark + tile meshes identically to the tile. `--dem file.tif` output unchanged. +- **15b:** a catchment polygon in EPSG:4326 over the archive meshes, with + `domain_crs` and `domain_transform` recorded. +- **15c:** the basin piece above meshes with `--out-crs` set to the basin + LCC, and records `computation_frame` and the reprojection estimate. +- **15d:** @perf meshes the basin piece and then the whole basin on Ola's Mac, + recording peak memory, decode time and refine time, under + `docs/benchmarks//`, with the power state. +- For all: every gate in `CLAUDE.md` §4 green, and CI green. + +## Questions for Ola (Q1-Q5 ruled 2026-09-27, see "Ruled by Ola"; Q6-Q10 open) + +Q1-Q5 are about Norway and are needed before 15a starts. Q6-Q10 are about +the basin and can wait until after Norway. + +**Q1 (the parked U1). Two tiles give different valid values at the same node.** +Now measured: DTM10 overlaps agree bit for bit on 4 of 869 pairs (N3; but see +Q1 revised: different-date pairs disagree), and +ANADEM's on 3 seams (B4). But the committed benchmark tile and the archive's +tile of the same name differ at 910 706 nodes (N5), so two releases in one +directory is a real case. +- **(a) Refuse, naming both tiles, the count and the largest difference. + Recommended.** Valid still beats NoData, and the result does not depend on + tile order. +- (b) First tile by sorted name wins, silently. That is what the legacy did in + effect. +- (c) Refuse by default, and add a `--overlap first` flag. About 15 lines more. + +**Q2 (the parked U2). `--bbox`.** +- **(a) The union by default, plus an optional `--bbox` in the DEM's CRS. + Recommended.** Without it, `--dem DIR` on your archive is refused by the cap + (72 GiB box), and the only way round would be copying files into a smaller + directory. It is about 20 lines. +- (b) The union only. + +**Q3 (the parked U3). Where the repository and its paths live.** `io/__init__.py` +says nothing in `io/` opens a file. +- **(a) `io/repository.py`, with `io/`'s rule narrowed to "decoders and + encoders take streams; `io/repository.py` is the one module that opens + files, read-only". Recommended.** Storage access sits next to the formats it + reads. +- (b) A top-level `tin_engine/dem_repository.py`, leaving `io/`'s rule as it is. +- (c) Paths in `cli.py`. Not recommended: `cli.py` is 1101 lines, and a GUI + or API worker could not reuse it. + +**Q4 (the parked U4). Tiles with different NoData sentinels.** Your ruling +C2 (b) in increment 18 already assumed (a): "differing sentinels are refused". +Both real archives have one sentinel each (N1, B2). +- **(a) Refuse. Recommended; please confirm.** +- (b) Rewrite each tile's sentinel to NaN and give the mosaic no sentinel. + About 10 lines. + +**Q5 (new). The eight half-cell tiles in your archive (N2).** 7304_1, +7507_4, 7606_2, 7707_1, 7707_3, 7807_2, 7807_3 and 7808_3 sit 5 m east-west +off the other 246, and are one or two nodes larger. +- **(a) Refuse a request that mixes them with the main lattice; a request + inside their own lattice meshes. Recommended.** Nothing is resampled. You + may know where they came from: a re-delivery of those sheets on a shifted + grid would explain it, and a fresh download of them might be on the main + lattice. +- (b) Resample them onto the main lattice (bilinear, a half-cell shift, so + every value becomes the average of two). That is resampling (R8), and its + own increment. + +**Q6 (the basin). The computation frame.** +- **(a) The lattice frame: mesh in an affine image of the DEM's own grid, + transform the domain in and the mesh out. Recommended.** The tolerance holds + exactly at the DEM's own nodes. Triangle shapes are judged in a frame up to + 6.5 % out of square on the ground at the basin's south edge. +- (b) Resample onto a square grid in a projected CRS first. Shapes are judged + on the ground, but the tolerance then refers to the resampled grid: 13 m off + the DEM at worst on real Espinhaço terrain (B6). It also costs a second + array the size of the basin's box. + +**Q7 (the basin). The output CRS for a geographic DEM.** +- **(a) Required as `--out-crs`, with a suggested basin-fitted LCC in the + refusal message. Recommended.** The output CRS is a contract with whatever + reads the mesh, and a CRS fitted to each domain would put two catchments, or + two subdomains, in different CRSs. +- (b) Fit an LCC to the domain automatically, and record it as WKT2. +- (c) Write longitude and latitude. Exact, with no bending, but in degrees. + +**Q8 (the basin). The anisotropy limit and the square frame.** +- **(a) A square frame (21b's integer incircle applies) and a limit of 10 %: + the basin passes (6.5 %); geographic DEMs above about 25° latitude are + refused. Recommended.** +- (b) A frame fitted to the extent's mean latitude: shapes truer (±3 % over + the basin), cells not square, so the filtered incircle is used and 21b's + gain (−15 % refine at 8 threads) is lost for geographic DEMs. + +**Q9 (the basin). Memory for the whole basin.** Its box at 30 m is 8.6 GiB, +and the design caps the canvas at half of physical memory. +- **(a) A dense canvas, as designed: the basin fits on your 32 GiB Mac. + Recommended for now.** Is 32 GiB the machine the basin must run on, or must + it also run on a smaller one? +- (b) A block-sparse raster in C++ holding only the 512² blocks that meet the + basin (about 2.7 GiB). Its own increment: a new `RasterSource`, the concept + change 18 R6 describes, and a binding. +- (c) Go to domain decomposition, which also solves memory. + +**Q10 (the basin). May an ANADEM extract be committed as a test fixture?** +ANADEM's repository is MIT-licensed, but that licence names "the Software", +and the data derives from Copernicus GLO-30, which carries its own attribution +terms. +- **(a) Commit about 2.4 MB of extracts with both notices, after you or the + authors confirm the data licence. Recommended.** +- (b) Download on demand in a test marked `network`, skipped in CI. + +Still open from the research note, for @perf's basin run: the tolerance +intended for the basin (it sets the triangle count more than anything else), +and whether commercial use matters. diff --git a/docs/increments/15-probes/anadem_probe.py b/docs/increments/15-probes/anadem_probe.py new file mode 100644 index 00000000..ec508c49 --- /dev/null +++ b/docs/increments/15-probes/anadem_probe.py @@ -0,0 +1,222 @@ +"""Increment 15's measurements on ANADEM v1, by HTTP range reads (no full download). + + python docs/increments/15-probes/anadem_probe.py headers # B2, B3: CRS, lattice, overlaps + python docs/increments/15-probes/anadem_probe.py seams # B4: do overlaps agree? + python docs/increments/15-probes/anadem_probe.py resample # B6: resampling error + python docs/increments/15-probes/anadem_probe.py frame # B5, B7: sizes, edge bending + +Needs network access to metadados.snirh.gov.br, plus tifffile, numpy and pyproj. +It is a measurement script, not production code, and nothing imports it. +""" + +from __future__ import annotations + +import io +import math +import sys +import urllib.request + +import numpy as np +import pyproj +import tifffile + +URL = "https://metadados.snirh.gov.br/files/anadem_v1_tiles/anadem_v1_{}.tif" +BASIN_TILES = ("22L", "23K", "23L", "23M", "24K", "24L", "24M", "25L", "25M") +NODATA = -9999.0 +LCC = "+proj=lcc +lat_1=-10 +lat_2=-18.5 +lat_0=-14 +lon_0=-42 +ellps=WGS84 +units=m +no_defs" + + +class RangeFile(io.RawIOBase): + """A seekable read-only file over HTTP range requests, cached in 64 KiB blocks.""" + + BLOCK = 1 << 16 + + def __init__(self, url: str) -> None: + self.url, self.pos, self.cache, self.fetched = url, 0, {}, 0 + head = urllib.request.urlopen(urllib.request.Request(url, method="HEAD"), timeout=30) + self.size = int(head.headers["Content-Length"]) + + def seekable(self) -> bool: + return True + + def readable(self) -> bool: + return True + + def tell(self) -> int: + return self.pos + + def seek(self, offset: int, whence: int = 0) -> int: + self.pos = {0: offset, 1: self.pos + offset, 2: self.size + offset}[whence] + return self.pos + + def _block(self, b: int) -> bytes: + if b not in self.cache: + start, end = b * self.BLOCK, min((b + 1) * self.BLOCK, self.size) - 1 + req = urllib.request.Request(self.url, headers={"Range": f"bytes={start}-{end}"}) + self.cache[b] = urllib.request.urlopen(req, timeout=60).read() + self.fetched += end - start + 1 + return self.cache[b] + + def readinto(self, buf) -> int: # type: ignore[no-untyped-def] + n, out = min(len(buf), self.size - self.pos), bytearray() + while len(out) < n: + b, o = divmod(self.pos + len(out), self.BLOCK) + out += self._block(b)[o : o + n - len(out)] + buf[:n] = out + self.pos += n + return n + + +def open_tile(name: str) -> tuple[io.BufferedReader, tifffile.TiffFile]: + f = io.BufferedReader(RangeFile(URL.format(name)), buffer_size=1 << 16) + return f, tifffile.TiffFile(f) + + +def header(name: str) -> tuple[float, float, int, int, float]: + """(lon, lat) of the upper-left corner, rows, cols, spacing.""" + _, tif = open_tile(name) + page = tif.pages.first + tie, scale = page.tags[33922].value, page.tags[33550].value + return tie[3], tie[4], page.imagelength, page.imagewidth, scale[0] + + +def block(name: str, block_row: int, block_col: int) -> np.ndarray: + """One 512 x 512 TIFF block, decoded, as float64.""" + f, tif = open_tile(name) + page = tif.pages.first + index = block_row * -(-page.imagewidth // 512) + block_col + f.seek(page.dataoffsets[index]) + data = f.read(page.databytecounts[index]) + return np.asarray(page.decode(data, index)[0], dtype=np.float64).reshape(512, 512) + + +def headers() -> None: + rows = {} + for name in BASIN_TILES: + f, tif = open_tile(name) + page, keys = tif.pages.first, tif.geotiff_metadata or {} + rows[name] = header(name) + print( + name, page.shape, page.dtype, page.compression.name, "tiled", page.tilewidth, + "pages", len(tif.pages), "reduced", [p.is_reduced for p in tif.pages[1:]], + "nodata", page.tags[42113].value if 42113 in page.tags else None, + "model", keys.get("GTModelTypeGeoKey"), "raster", keys.get("GTRasterTypeGeoKey"), + "geographic", keys.get("GeographicTypeGeoKey"), "bytes read", f.raw.fetched, + ) + x0, y0, _, _, d = rows["23L"] + print(f"spacing {d!r} deg = {d * 3600:.6f} arc-seconds") + boxes = {} + for name, (x, y, r, c, dd) in rows.items(): + kx, ky = (x - x0) / d, (y0 - y) / d + boxes[name] = (round(ky), round(ky) + r, round(kx), round(kx) + c) + print( + f"{name}: same spacing {dd == d}; offset cols {kx:.6f} rows {ky:.6f} " + f"(off-integer {abs(kx - round(kx)):.1e}, {abs(ky - round(ky)):.1e}); " + f"lon [{x:.4f}, {x + c * d:.4f}] lat [{y - r * d:.4f}, {y:.4f}]; " + f"{r * c / 1e6:.0f} M cells, {r * c * 4 / 2**30:.2f} GiB float32" + ) + for a, b in (("23L", "24L"), ("23L", "23K"), ("23L", "23M"), ("24L", "24K"), ("24L", "24M"), + ("23K", "24K"), ("22L", "23L"), ("24L", "25L"), ("23M", "24M")): + ra, rb = boxes[a], boxes[b] + print(f"{a}/{b}: overlap rows {min(ra[1], rb[1]) - max(ra[0], rb[0])}, " + f"cols {min(ra[3], rb[3]) - max(ra[2], rb[2])}") + + +def compare(label: str, a: np.ndarray, b: np.ndarray) -> None: + va, vb = a != NODATA, b != NODATA + both = va & vb + diff = np.abs(a[both] - b[both]) + worst = diff.max() if diff.size else None + print(f"{label}: {a.size} cells, both valid {int(both.sum())}, " + f"one valid {int((va ^ vb).sum())}, equal {int((a[both] == b[both]).sum())}, " + f"max |diff| {worst}") + + +def seams() -> None: + # 23L | 24L at 42 W: 24L starts 22264 columns east of 23L, same rows. Row block 29. + a, b = block("23L", 29, 43), block("24L", 29, 0) + c0 = 22264 - 43 * 512 + compare("23L|24L aligned ", a[:, c0 : c0 + 8], b[:, 0:8]) + compare("23L|24L shifted 1 column ", a[:, c0 + 1 : c0 + 8], b[:, 0:7]) + # 23L | 23K: 23K starts 30163 rows south and 1 column west of 23L. + tr = 30163 // 512 + r0 = 30163 - tr * 512 + a, b = block("23L", tr, 20), block("23K", 0, 20) + compare("23L|23K aligned ", a[r0 : r0 + 8, 0:511], b[0:8, 1:512]) + compare("23L|23K shifted 1 row ", a[r0 + 1 : r0 + 8, 0:511], b[0:7, 1:512]) + # 24L | 24K: 87-row overlap, same columns. + a = np.vstack([block("24L", tr, 5), block("24L", tr + 1, 5)])[r0 : r0 + 87] + compare("24L|24K 87 rows ", a, block("24K", 0, 5)[0:87]) + + +def bilinear(z: np.ndarray, fx: np.ndarray, fy: np.ndarray) -> np.ndarray: + c = np.clip(np.floor(fx).astype(int), 0, z.shape[1] - 2) + r = np.clip(np.floor(fy).astype(int), 0, z.shape[0] - 2) + tx, ty = fx - c, fy - r + return (z[r, c] * (1 - tx) * (1 - ty) + z[r, c + 1] * tx * (1 - ty) + + z[r + 1, c] * (1 - tx) * ty + z[r + 1, c + 1] * tx * ty) + + +def resample() -> None: + """3 x 3 blocks of 23K over the Serra do Espinhaco, resampled onto square LCC grids.""" + r_block, c_block = 18, 30 + s = np.vstack([np.hstack([block("23K", r_block + i, c_block + j) for j in range(3)]) + for i in range(3)]) + x, y, _, _, d = header("23K") + lon0 = x + d / 2 + c_block * 512 * d # area-registered: nodes at cell centres + lat0 = y - d / 2 - r_block * 512 * d + print(f"window lon [{lon0:.4f}, {lon0 + 1535 * d:.4f}] " + f"lat [{lat0 - 1535 * d:.4f}, {lat0:.4f}], " + f"NoData cells {int((s == NODATA).sum())}, z [{s.min():.1f}, {s.max():.1f}]") + fwd = pyproj.Transformer.from_crs("EPSG:4326", LCC, always_xy=True) + inv = pyproj.Transformer.from_crs(LCC, "EPSG:4326", always_xy=True) + rr, cc = np.mgrid[0:1536, 0:1536] + lon, lat = lon0 + cc * d, lat0 - rr * d + px, py = fwd.transform(lon, lat) + m = 40 + inner = (slice(2 * m, -2 * m), slice(2 * m, -2 * m)) + for h in (30.0, 20.0, 10.0): + xs = np.arange(px[m:-m, m:-m].min(), px[m:-m, m:-m].max(), h) + ys = np.arange(py[m:-m, m:-m].max(), py[m:-m, m:-m].min(), -h) + gx, gy = np.meshgrid(xs, ys) + glon, glat = inv.transform(gx, gy) + resampled = bilinear(s, (glon - lon0) / d, (lat0 - glat) / d) + back = bilinear(resampled, (px[inner] - xs[0]) / h, (ys[0] - py[inner]) / h) + e = np.abs(back - s[inner]) + print(f"h = {h:4.0f} m: {resampled.size / 1e6:.2f} M target nodes; |resampled - source| at " + f"source nodes: max {e.max():.2f} m, p99.9 {np.quantile(e, 0.999):.2f}, " + f"p99 {np.quantile(e, 0.99):.2f}, median {np.median(e):.3f}") + ident = bilinear(s, (lon[100:-100, 100:-100] - lon0) / d, (lat0 - lat[100:-100, 100:-100]) / d) + control = np.abs(ident - s[100:-100, 100:-100]).max() + print(f"control, resampled onto its own nodes: max {control:.1e}") + + +def frame() -> None: + d = 0.00026949458523585647 + geod = pyproj.Geod(ellps="WGS84") + for lat in (-7, -14, -21): + ew = geod.inv(-42, lat, -42 + d, lat)[2] + ns = geod.inv(-42, lat, -42, lat + d)[2] + print(f"lat {lat}: cell E-W {ew:.3f} m, N-S {ns:.3f} m, ratio {ns / ew:.4f}") + cols, rows = 12 / d, 14 / d + print(f"basin bbox 48-36 W, 21-7 S: {cols:.0f} x {rows:.0f} = {cols * rows / 1e9:.2f} G nodes, " + f"{cols * rows * 4 / 2**30:.1f} GiB float32") + cell = geod.inv(-42, -14, -42 + d, -14)[2] * geod.inv(-42, -14, -42, -14 + d)[2] + print(f"basin 636 920 km2 at the 14 S cell area: {636920e6 / cell / 1e6:.0f} M nodes") + print(f"frame spacing d * pi * a / 180 = {d * math.pi * 6378137.0 / 180!r}") + fwd = pyproj.Transformer.from_crs("EPSG:4326", LCC, always_xy=True) + for lat in (-7, -14, -21): + for km in (1, 5, 20, 50): + worst = 0.0 + for deg in range(0, 180, 15): + a, half = math.radians(deg), 0.5 * km * 1000 / 111320 + lo = [-42 - half * math.cos(a), -42 + half * math.cos(a), -42.0] + la = [lat - half * math.sin(a), lat + half * math.sin(a), float(lat)] + (x0, x1, xm), (y0, y1, ym) = fwd.transform(lo, la) + worst = max(worst, math.hypot(xm - (x0 + x1) / 2, ym - (y0 + y1) / 2)) + print(f"lat {lat}, edge {km:2d} km: straight lattice edge vs straight LCC edge, " + f"midpoints {worst:.4f} m apart") + + +if __name__ == "__main__": + {"headers": headers, "seams": seams, "resample": resample, "frame": frame}[sys.argv[1]]() diff --git a/docs/increments/15-probes/dtm10_dates.py b/docs/increments/15-probes/dtm10_dates.py new file mode 100644 index 00000000..ca7edd71 --- /dev/null +++ b/docs/increments/15-probes/dtm10_dates.py @@ -0,0 +1,101 @@ +"""Do DTM10 overlaps agree when the two tiles share an export date? + +Evidence for "Ruled by Ola", Q1 revised (docs/increments/15-dem-mosaic.md). +The export date of a tile is the modification date of its side files +(`.tif.aux.xml`, else `.tfw`; the two agree on all 254 tiles). 143 of the +254 `.tif` files share one date, 2021-12-10, later than their side files, so +the `.tif` date does not tell the exports apart. The TIFF tags carry no date. + +Usage: python dtm10_dates.py [ARCHIVE] [N_PAIRS] [SEED] +Prints, for N random neighbour pairs on one lattice, whether they share an +export date and how their jointly valid overlap nodes compare: exact, within +1 mm, or beyond. +""" + +import datetime +import glob +import os +import random +import sys + +import numpy as np +import tifffile + +NODATA = -32767.0 + + +def export_date(tif): + for side in (tif + ".aux.xml", tif[:-4] + ".tfw"): + if os.path.exists(side): + return datetime.date.fromtimestamp(os.path.getmtime(side)) + return None + + +def origin(tif): + with tifffile.TiffFile(tif) as f: + page = f.pages[0] + tie = page.tags["ModelTiepointTag"].value + return tie[3], tie[4], page.shape + + +def overlap(a, b, info): + (xa, ya, sa), (xb, yb, sb) = info[a], info[b] + x0, x1 = max(xa, xb), min(xa + sa[1] * 10, xb + sb[1] * 10) + y0, y1 = max(ya - sa[0] * 10, yb - sb[0] * 10), min(ya, yb) + if x1 <= x0 or y1 <= y0: + return None + arr_a, arr_b = tifffile.imread(a), tifffile.imread(b) + + def window(arr, xm, ym): + r0, r1 = round((ym - y1) / 10), round((ym - y0) / 10) + c0, c1 = round((x0 - xm) / 10), round((x1 - xm) / 10) + return arr[r0:r1, c0:c1].astype(np.float64) + + wa, wb = window(arr_a, xa, ya), window(arr_b, xb, yb) + ok = (wa != NODATA) & (wb != NODATA) + return np.abs(wa[ok] - wb[ok]) + + +def main(): + archive = sys.argv[1] if len(sys.argv) > 1 else "../rasputin_data/DTM10_UTM33_20220924" + n = int(sys.argv[2]) if len(sys.argv) > 2 else 60 + seed = int(sys.argv[3]) if len(sys.argv) > 3 else 1 + tiles = sorted(glob.glob(f"{archive}/*_10m_z33.tif")) + info = {t: origin(t) for t in tiles} + pairs = [ + (a, b) + for i, a in enumerate(tiles) + for b in tiles[i + 1 :] + if (info[a][0] - info[b][0]) % 10 == 0 + and (info[a][1] - info[b][1]) % 10 == 0 + and abs(info[a][0] - info[b][0]) <= 50510 + and abs(info[a][1] - info[b][1]) <= 50510 + ] + random.seed(seed) + rows = [] + for a, b in random.sample(pairs, min(n, len(pairs))): + d = overlap(a, b, info) + if d is None or d.size == 0: + continue + same = export_date(a) == export_date(b) + rows.append((same, int((d > 0).sum()), int((d >= 1e-3).sum()), float(d.max()), a, b)) + for same in (True, False): + group = [r for r in rows if r[0] == same] + exact = sum(r[1] == 0 for r in group) + within = sum(r[2] == 0 for r in group) + worst = max((r[3] for r in group), default=0.0) + label = "same date" if same else "different dates" + print( + f"{label:16s} pairs {len(group):3d} exact {exact:3d}" + f" within 1 mm {within:3d} worst {worst:.3f} m" + ) + same_bad = [r for r in rows if r[0] and r[2] > 0] + worst_rows = sorted(rows, key=lambda r: -r[3])[:8] + for same, _n_any, n_mm, worst, a, b in same_bad + worst_rows: + tag = "same" if same else "diff" + name_a, name_b = os.path.basename(a)[:6], os.path.basename(b)[:6] + print(f" {tag} {name_a} | {name_b} >1mm {n_mm:7d} max {worst:.3f}") + + +if __name__ == "__main__": + main() diff --git a/docs/increments/15-probes/dtm10_probe.py b/docs/increments/15-probes/dtm10_probe.py new file mode 100644 index 00000000..f5cd42e8 --- /dev/null +++ b/docs/increments/15-probes/dtm10_probe.py @@ -0,0 +1,111 @@ +"""Increment 15's measurements on Ola's DTM10 archive (254 UTM33 tiles). + + python docs/increments/15-probes/dtm10_probe.py headers DIR # N1, N2, N4: headers, lattice + python docs/increments/15-probes/dtm10_probe.py seams DIR # N3: do overlaps agree? + +Header-only for `headers`, through io/geotiff.py's own private checks, so a +refusal here is the refusal `decode_dem` would raise. A measurement script, not +production code; nothing imports it. +""" + +from __future__ import annotations + +import collections +import sys +from pathlib import Path + +import numpy as np +import tifffile + +from tin_engine.io import geotiff as g + + +def metas(directory: Path) -> dict[str, tuple]: + out, refused = {}, collections.Counter() + for path in sorted(directory.glob("*.tif")): + with path.open("rb") as stream, tifffile.TiffFile(stream) as tif: + page, pages = tif.pages.first, tuple(tif.pages) + try: + g._single_page(pages) + dtype = g._check_page(page) + tie, scale = g._georeferencing(page) + keys = tif.geotiff_metadata or {} + x, y, dx, dy, area = g._placement(tie, scale, keys) + epsg = g._projected_epsg(keys) + nodata, source = g._nodata(page, dtype, None) + except g.GeoTiffError as error: + refused[str(error)[:90]] += 1 + continue + out[path.name] = (x, y, dx, dy, page.imagelength, page.imagewidth, epsg, area, + nodata, source, str(dtype), page.compression.name, page.is_tiled) + for text, n in refused.items(): + print(f"REFUSED x{n}: {text}") + return out + + +def headers(directory: Path) -> None: + m = metas(directory) + print(f"{len(m)} tiles read") + for i, label in ((6, "epsg"), (2, "dx"), (3, "dy"), (7, "area"), (8, "nodata"), + (9, "nodata source"), (10, "dtype"), (11, "compression"), + (12, "tiled"), (4, "rows"), (5, "cols")): + print(f"{label}: {collections.Counter(v[i] for v in m.values()).most_common(6)}") + x0 = min(v[0] for v in m.values()) + y0 = max(v[1] for v in m.values()) + dx = next(iter(m.values()))[2] + off = [((v[0] - x0) / dx, (y0 - v[1]) / dx) for v in m.values()] + worst = max(max(abs(a - round(a)), abs(b - round(b))) for a, b in off) + print(f"reference node x {x0}, y {y0}; worst off-integer offset {worst:.1e} cells") + boxes = {k: (round((y0 - v[1]) / dx), round((y0 - v[1]) / dx) + v[4], + round((v[0] - x0) / dx), round((v[0] - x0) / dx) + v[5]) for k, v in m.items()} + rows = collections.Counter() + names = sorted(boxes) + for i, a in enumerate(names): + for b in names[i + 1 :]: + ra, rb = boxes[a], boxes[b] + h = min(ra[1], rb[1]) - max(ra[0], rb[0]) + w = min(ra[3], rb[3]) - max(ra[2], rb[2]) + if h > 0 and w > 0: + rows[(min(h, w), "corner" if h < 1000 and w < 1000 else "edge")] += 1 + print("overlapping pairs by (overlap width in nodes, kind):", sorted(rows.items())) + span = (max(b[1] for b in boxes.values()), max(b[3] for b in boxes.values())) + print(f"union bbox {span[0]} x {span[1]} nodes = {span[0] * span[1] / 1e9:.2f} G, " + f"{span[0] * span[1] * 4 / 2**30:.1f} GiB float32") + covered = sum(v[4] * v[5] for v in m.values()) + print(f"sum of tile nodes {covered / 1e9:.2f} G ({covered * 4 / 2**30:.1f} GiB float32)") + + +def seams(directory: Path) -> None: + m = metas(directory) + x0 = min(v[0] for v in m.values()) + y0 = max(v[1] for v in m.values()) + dx = next(iter(m.values()))[2] + boxes = {k: (round((y0 - v[1]) / dx), round((y0 - v[1]) / dx) + v[4], + round((v[0] - x0) / dx), round((v[0] - x0) / dx) + v[5]) for k, v in m.items()} + names, done = sorted(boxes), 0 + for i, a in enumerate(names): + for b in names[i + 1 :]: + ra, rb = boxes[a], boxes[b] + r0, r1 = max(ra[0], rb[0]), min(ra[1], rb[1]) + c0, c1 = max(ra[2], rb[2]), min(ra[3], rb[3]) + if r1 - r0 <= 0 or c1 - c0 <= 0 or min(r1 - r0, c1 - c0) > 1000 or done >= 4: + continue + za = tifffile.imread(directory / a)[r0 - ra[0] : r1 - ra[0], c0 - ra[2] : c1 - ra[2]] + zb = tifffile.imread(directory / b)[r0 - rb[0] : r1 - rb[0], c0 - rb[2] : c1 - rb[2]] + nd = m[a][8] + va, vb = za != nd, zb != nd + both = va & vb + diff = np.abs(za[both].astype(float) - zb[both]) + print(f"{a} | {b}: {za.shape} overlap, both valid {int(both.sum())}, one valid " + f"{int((va ^ vb).sum())}, equal {int((za[both] == zb[both]).sum())}, " + f"max |diff| {diff.max() if diff.size else None}") + if za.shape[0] > 1 and za.shape[1] > 1: # the probe can fail: shift by one + s = np.abs(za[1:, :].astype(float) - zb[:-1, :]) if za.shape[0] <= za.shape[1] \ + else np.abs(za[:, 1:].astype(float) - zb[:, :-1]) + print(f" shifted by one node: max |diff| {s.max():.2f}, " + f"equal {(s == 0).mean():.1%}") + done += 1 + + +if __name__ == "__main__": + {"headers": headers, "seams": seams}[sys.argv[1]](Path(sys.argv[2])) diff --git a/docs/increments/20b-min-insertion-distance.md b/docs/increments/20b-min-insertion-distance.md index d3d83419..105ec11c 100644 --- a/docs/increments/20b-min-insertion-distance.md +++ b/docs/increments/20b-min-insertion-distance.md @@ -421,7 +421,8 @@ below included (3 of them); 90 in `refine.hpp` against ~70. Method: for each production file the branch touches (`include/`, `bindings/`, `src_python/`), count the lines at the branch tip and at `d32ce59` (the last increment-20 commit), skipping comments, docstrings and raw-literal bodies per `CLAUDE.md` -§2; blank lines count. The figure is the difference. Settled at green: `--no-constraint-feet` needs +§2; blank lines count (superseded 2026-09-28: `CLAUDE.md` §2 now excludes +blank lines). The figure is the difference. Settled at green: `--no-constraint-feet` needs `--tolerance` and `--dem`, like `--start-min-angle`; only the first constrained edge closer than ε is tried; a refusal is counted only when `N` is then inserted; when `G` is 0, ε is the cap (otherwise 0/0 at tolerance 0 on diff --git a/docs/research/sao-francisco-basin.md b/docs/research/sao-francisco-basin.md new file mode 100644 index 00000000..7cea5708 --- /dev/null +++ b/docs/research/sao-francisco-basin.md @@ -0,0 +1,140 @@ +# The São Francisco basin as rasputin's target + +Status: research note, 2026-09-27, main session. Ola's aim (2026-09-27): +"sub second is very good for smaller catchments, and of course faster is +better. But the important thing is huge catchments. I want to construct the +San Fransisco Basin in Brazil eventually, which is about the size of Norway. +And I think we can get 50m rasters for the DEM." + +Facts below are from the sources listed at the end, checked by web search on +2026-09-27. Figures marked *computed* were computed here with pyproj; figures +marked *inferred* are extrapolations, not measurements. + +## The basin + +- **Area: 636,920 km²** (OAS). Norway is about 385,000 km² with Svalbard and + Jan Mayen and about 324,000 km² without (from memory, not checked here), so + the basin is roughly 1.65 to 2 times Norway. +- It drains parts of Minas Gerais, Goiás, Bahia, Pernambuco, Alagoas, Sergipe + and the Federal District. +- Roughly latitude 7°S to 21°S and longitude 36°W to 48°W (approximate + bounding box, used for the computations below). *Computed:* about + 1,280 km × 1,620 km in a projected CRS. + +## Elevation data + +| DEM | spacing | what it is | licence | +|---|---|---|---| +| **ANADEM** (ANA / UFRGS, 2024) | 30 m (0.970″, measured from its headers in increment 15's B2; not Copernicus's 1″) | Copernicus GLO-30 with vegetation bias removed for South America (Landsat-8, Sentinel-2, GEDI lidar). Mean bias 9.6 m (COPDEM) → 1.5 m; in forest 14.3 m → 0.4 m | free and open source per its authors; exact licence to be read from the repository | +| Copernicus GLO-30 | 30 m (1″) | a surface model (tree canopy and buildings included) | free (Copernicus licence) | +| Copernicus GLO-90 | 90 m (3″) | the same at 3″ | free | +| FABDEM | 30 m | Copernicus with forests and buildings removed (Bristol / Fathom) | CC BY-NC-SA 4.0: **non-commercial only** | +| IBGE MDE | 1:25,000 and 1:50,000 scale | national mapping agency, aerial restitution | coverage is partial | + +- **No Brazil-wide 50 m DEM turned up.** A 50 m grid would most likely be + ANADEM or Copernicus resampled from 30 m, or IBGE's 1:50,000 material where + it exists. Where Ola's 50 m source is from is a question for Ola. +- **For hydrology, ANADEM is the obvious first choice.** It is a terrain model + rather than a surface model, it was built for South America by the national + water agency's partners, and it is free. It comes in MGRS tiles (GitHub) and + on Google Earth Engine. +- All of these are in **geographic coordinates** (EPSG:4326, arc-seconds), not + a projected grid. *Computed* (pyproj, WGS84): at 15°S, 1″ is 30.7 m + north–south and 29.9 m east–west, so the cells are not square in metres. + +## Other inputs for the basin + +- **Basin outline:** ANA's *Base Hidrográfica Ottocodificada* (BHO) has + Otto-Pfafstetter basins at several levels as GeoPackage, including the São + Francisco. That gives the confining polygon directly, without auto-catchment. + BHO's drainage lines (`GEOFT_BHO_TRECHO_DRENAGEM`) are candidate river + polylines for 16b. +- **Land cover:** CORINE does not cover Brazil. **MapBiomas** (annual, 30 m, + 1985 onward, CC BY 4.0) is the Brazilian equivalent. It is a **raster**, not + vector polygons like CORINE's GeoPackage, so 16b would need a + raster-to-polygon step, or a separate path, for it. + +## Size, against rasputin today + +| grid | nodes inside the basin | float32 DEM | +|---|---:|---:| +| 30 m | 708 M | 2.8 GB | +| 50 m | 255 M | 1.0 GB | +| 90 m | 79 M | 0.3 GB | + +*Computed* from the area. The bounding box holds more (about 830 M nodes at +50 m). Today's 1 m benchmark tile is 25.5 M nodes, so the basin at 50 m is +about **10×** the tile in nodes and at 30 m about **28×**. + +*Inferred, not measured:* the scan visits every domain node about 5.6 times +over a refine (serial profile) at roughly 9 ns per visit on one thread. At +50 m that is on the order of 10-15 s of scan on one thread and a few seconds on +8. The triangle count, and so the serial phase and output size, depends on +relief and on the tolerance, not on the node count. It cannot be extrapolated +from the tile, which is a coastal fjord tile at 10 m with 1 m tolerance. +Measuring it on a real piece of the basin is the first thing to do. + +The DEM fits in memory on a 32 GB machine at 30 m and at 50 m. Memory does not +force streaming at this size; parallel scaling and the serial phase set the +time. + +## What this means for the roadmap + +In roughly the order the basin needs them: + +1. **A DEM in several tiles (ROADMAP gap 6).** ANADEM comes as MGRS tiles and + Copernicus as 1°×1° tiles. The basin spans about 14° × 12°, so on the order + of 150 1° tiles. Gap 6 today assumes aligned tiles in one CRS. +2. **Geographic DEMs and the computation CRS** (ROADMAP's "inputs in their own + CRS"). rasputin's core expects a projected grid. Two routes: + - resample the DEM onto a projected grid in Python before meshing (pyproj + plus NumPy, chunked; no GDAL), or + - mesh in a projected CRS while sampling the geographic grid. + + **One CRS for the whole basin is a real choice.** *Computed* with pyproj + over the whole 7-21°S, 36-48°W box (corrected in review; the first figures + came from a few sample points): UTM zone 23S reaches a scale error of + 1.20 %, and the Brazil Polyconic (EPSG:5880) 4.89 %, both at 36°W 7°S. A + Lambert conformal conic fitted to the basin (standard parallels 10°S and + 18.5°S, central meridian 42°W) stays within **0.52 %**. +3. **21b's integer incircle does not apply to non-square cells.** QW2 needs + `dx == dy`. On a geographic grid, or a projected grid resampled with + non-square cells, refine falls back to the filtered kernel. That is correct + but loses 21b's gain. Resampling to a square projected grid keeps it. +4. **Parallel refine (21d) matters more here than on the benchmark.** At + 10-28× the nodes, the scan and the serial phase both grow. Ola's + determinism ruling of 2026-09-27 ("very large areas … we should not rely on + bit-identical outputs") was made with exactly this case in mind. +5. **Domain decomposition** (21's option D, deferred to the mosaic work) is + the natural fit for a basin of this size: split the basin into parts, mesh + them in parallel, and keep shared boundaries. The tile-parallel greedy + insertion prior art (Remote Sensing 12(3):437, 2020) is the reference. +6. **16b with the basin's own data:** BHO rivers as polylines, and MapBiomas + land cover, which needs the raster step above. + +## Open questions for Ola + +1. ~~Where does the 50 m DEM come from?~~ **Answered by Ola, 2026-09-27:** + the 50 m "was probably referring to 30m". The working assumption is a 30 m + DEM (ANADEM first): 708 M nodes inside the basin, 2.8 GB as float32. +2. What tolerance is intended for the basin? It sets the triangle count more + than anything else. +3. Commercial use? FABDEM is non-commercial only; ANADEM and Copernicus are + the safe choices if that matters. + +## Sources + +- OAS, *São Francisco River Basin* brochure: + http://www.oas.org/en/sedi/dsd/iwrm/Past_Projects/Documents/Sao_Francisco_Brochure.pdf +- "ANADEM: A Digital Terrain Model for South America", Remote + Sensing 16(13):2321, 2024: https://www.mdpi.com/2072-4292/16/13/2321 ; + data: https://github.com/HGE-IPH/anadem , https://hge-iph.github.io/anadem/ +- Copernicus DEM: https://dataspace.copernicus.eu/explore-data/data-collections/copernicus-contributing-missions/collections-description/COP-DEM ; + https://registry.opendata.aws/copernicus-dem/ +- FABDEM licence: https://www.fathom.global/insight/fabdem-download/ +- IBGE digital elevation model: + https://www.ibge.gov.br/en/geosciences/digital-surface-models/digital-surface-models/19081-digital-elevation-model.html +- ANA BHO: https://metadados.snirh.gov.br/geonetwork/srv/api/records/0c698205-6b59-48dc-8b5e-a58a5dfcc989 +- MapBiomas Collection 9 (FAO catalogue): + https://data.apps.fao.org/catalog/iso/416a2581-b5e2-43a5-bb93-a2fc26cd1d68 +- Tile-parallel terrain simplification: https://www.mdpi.com/2072-4292/12/3/437 diff --git a/project_structure.md b/project_structure.md index bfba4743..a9008825 100644 --- a/project_structure.md +++ b/project_structure.md @@ -83,6 +83,9 @@ src_python/tin_engine/ # public Python API (distribution name: rasputin) raster.py # the ONLY adapter from decoded data into _core grid_domain.py # DEM extent -> stride-subsampled nodes + outer ring; # pure numpy, never imports _core + mosaic.py # plan_mosaic / assemble: select, group by lattice, + # check overlaps and coverage, stitch; no files (15a) + dem_input.py # --dem/--bbox -> DemInput(tile, plan, label) (15a) domain.py # --domain: reads one polygon (GeoJSON or WKT), checks # CRS and extent; shapely + pyproj, never imports _core elevation.py # drops mesh vertices the DEM has no data for; @@ -110,6 +113,8 @@ src_python/tin_engine/ # public Python API (distribution name: rasputin) # ParaView; takes no path and opens nothing geotiff.py # TIFF container + GeoKey decoding -> DemTile models.py # Pydantic RasterMeta / DemTile + repository.py # TiffDemRepository: the ONE io/ module that opens + # files ("rb"); lists headers, loads tiles (15a) tests/ cpp/ # C++ tests (Catch2; unit/ and property/) diff --git a/src_python/tin_engine/cli.py b/src_python/tin_engine/cli.py index 5e2afbb6..34fb2b31 100644 --- a/src_python/tin_engine/cli.py +++ b/src_python/tin_engine/cli.py @@ -1,9 +1,11 @@ """Command-line interface for the rasputin terrain engine. -This module is the **single composition root**. It is the only Python module -that has a path, and joining the file system to the engine is its whole job -(``tin_engine.raster`` also imports ``_core``, to build the one core raster, per -``project_structure.md``): ``viz/`` is written against protocols and never +This module is the **single composition root**: joining the file system to the +engine is its whole job. Paths from the command line also reach +``tin_engine.domain``, ``tin_engine.dem_input`` and ``io/repository.py``, which +read the files they name (``tin_engine.raster`` also imports ``_core``, to +build the one core raster, per ``project_structure.md``): ``viz/`` is written +against protocols and never names a core type, while the core never sees a file, a path or a CRS. Everything that has to know both sides lives here. (``project_structure.md``'s rule that exactly one module constructs a core *raster* is about ``tin_engine.raster``.) @@ -52,6 +54,7 @@ import numpy as np import numpy.typing as npt import typer +from pydantic import ValidationError from tin_engine._core import ( ChainRole, @@ -65,14 +68,15 @@ sample, triangulate, ) +from tin_engine.dem_input import DemInput, DemRequest, open_dem from tin_engine.domain import DomainError, DomainPolygon, read_domain from tin_engine.elevation import Trimmed, trim from tin_engine.features import DEFAULT_VOCABULARY from tin_engine.grid_domain import default_stride, refine_start_stride, subsample -from tin_engine.io.geotiff import decode_dem -from tin_engine.io.models import GeoTiffError, RasterMeta +from tin_engine.io.models import DemTile, RasterMeta from tin_engine.io.ply import write_ply from tin_engine.io.vtk_legacy import write_vtk +from tin_engine.mosaic import Bounds, Seam from tin_engine.raster import to_core from tin_engine.stats import PhaseClock, Refinement, Report, Sizes, _exact, quality, render from tin_engine.viz.fixtures import GALLERY, Fixture @@ -526,8 +530,21 @@ def mesh( str | None, typer.Argument(help="Gallery fixture to write; or give --dem instead.") ] = None, dem: Annotated[ - Path | None, - typer.Option("--dem", help="A GeoTIFF DEM: mesh its extent with z sampled from it."), + list[Path] | None, + typer.Option( + "--dem", + help="A GeoTIFF DEM, several (repeat --dem), or one directory of tiles: mesh " + "its extent with z sampled from it.", + ), + ] = None, + bbox: Annotated[ + tuple[float, float, float, float] | None, + typer.Option( + "--bbox", + metavar="BOX", + help="With --dem, mesh only XMIN YMIN XMAX YMAX in the DEM's CRS, " + "snapped outward to its nodes.", + ), ] = None, stride: Annotated[ int | None, @@ -619,6 +636,8 @@ def mesh( the inside is meshed. Vertices where the DEM has no data are dropped with their triangles, and the count is reported. The file records the DEM's CRS, so ``--crs`` and ``--flat`` are refused. + ``--dem DIR`` or several ``--dem`` files are stitched into one grid first, + cut to ``--bbox`` if given (increment 15a, ``tin_engine.mosaic``). ``.vtk`` is one file for ParaView: triangles, constraint lines, their feature masks, one 0/1 array per feature that occurs, and the vocabulary @@ -643,13 +662,16 @@ def mesh( """ clock = PhaseClock() dem_run: _DemMesh | None = None - if (name is None) == (dem is None): + seams: tuple[Seam, ...] = () + if (name is None) == (not dem): raise typer.BadParameter( "give a gallery fixture name or --dem PATH, exactly one of the two", param_hint="--dem", ) - if dem is None and (domain is not None or domain_crs is not None): + if not dem and (domain is not None or domain_crs is not None): raise typer.BadParameter("applies only with --dem", param_hint="--domain") + if not dem and bbox is not None: + raise typer.BadParameter("applies only with --dem", param_hint="--bbox") if domain is None and domain_crs is not None: raise typer.BadParameter("applies only with --domain", param_hint="--domain-crs") if name is not None and name not in GALLERY: @@ -663,7 +685,7 @@ def mesh( "a .vtk file already carries the constraint edges", param_hint="--out-edges" ) - if dem is not None: + if dem: if flat: raise typer.BadParameter("--dem samples z from the DEM", param_hint="--flat") if crs: @@ -698,9 +720,11 @@ def mesh( raise typer.BadParameter( "--no-constraint-feet needs --tolerance", param_hint="--no-constraint-feet" ) - label = dem.stem + opened = _open_dem(dem, bbox, clock) + label = opened.label dem_run = _dem_mesh( - dem, + opened.tile, + ", ".join(map(str, dem)), stride, delaunay, snap_spacing, @@ -713,8 +737,19 @@ def mesh( ) surface_mesh = dem_run.trimmed epsg, sentence, described = dem_run.epsg, dem_run.sentence, dem_run.described + names = [t.name for t in opened.plan.tiles] + if len(names) > 1: + mosaic = f"mosaic of {len(names)} tiles, {opened.tile.meta.rows} x " + mosaic += f"{opened.tile.meta.cols} nodes" + typer.echo(mosaic, err=True) + sentence = f"{mosaic}; {sentence}" fields = [("crs", f"EPSG:{epsg}"), ("elevation_source", sentence)] comments = [f"crs EPSG:{epsg}", f"elevation {sentence}"] + if len(dem) > 1 or dem[0].is_dir(): # R11: the files used, named + fields.append(("dem_tiles", _ascii("; ".join(names)))) + seams = opened.seams + listed = "; ".join(s.entry() for s in seams) or "none" + fields.append(("dem_seams", _ascii(listed))) # Ola's Q1 revised if described: fields.append(("domain", described)) comments.append(f"domain {described}") @@ -800,7 +835,7 @@ def mesh( target.write_bytes(data) typer.echo(f"{target}") if stats is not None: - _write_report(clock, report_target, surface_mesh, dem_run, targets) + _write_report(clock, report_target, surface_mesh, dem_run, targets, seams) def _report_target( @@ -824,6 +859,7 @@ def _write_report( trimmed: Trimmed, dem_run: _DemMesh | None, files: list[Path], + seams: tuple[Seam, ...], ) -> None: """Build the report from what ran and write it, or print it for ``-``. The total stops here; the quality pass is timed on its own line (R4).""" @@ -855,6 +891,7 @@ def _write_report( total=total, stats_seconds=(time.perf_counter_ns() - t0) / 1e9, threads=os.cpu_count() if refinement else None, + seams=[s.cells() for s in seams], ) ) if target is None: @@ -883,6 +920,43 @@ def _fixture_mesh(name: str, delaunay: bool, spacing: float, clock: PhaseClock) ) +def _open_dem( + dem: list[Path], bbox: tuple[float, float, float, float] | None, clock: PhaseClock +) -> DemInput: + """``--dem`` and ``--bbox`` to one tile (increment 15a, R11); every refusal, + the reader's or the mosaic's, is a usage error in its own words.""" + try: + bounds = ( + None + if bbox is None + else Bounds(**dict(zip(("x_min", "y_min", "x_max", "y_max"), bbox, strict=True))) + ) + except ValidationError as exc: + raise typer.BadParameter(_words(exc), param_hint="--bbox") from exc + try: + with clock.phase("decode"): + return open_dem(DemRequest(sources=tuple(dem), bounds=bounds)) + except OSError as exc: + where = exc.filename or ", ".join(map(str, dem)) + raise typer.BadParameter( + f"cannot read {where}: {exc.strerror or exc}", param_hint="--dem" + ) from exc + except ValueError as exc: + raise typer.BadParameter(_words(exc), param_hint="--dem") from exc + + +def _ascii(text: str) -> str: + """A file field's text with non-ASCII escaped, as `dem_tiles` records names.""" + return text.encode("ascii", "backslashreplace").decode("ascii") + + +def _words(exc: ValueError) -> str: + """A refusal's own words: Pydantic's messages without its wrapper.""" + if isinstance(exc, ValidationError): + return "; ".join(str(e["msg"]) for e in exc.errors()) + return str(exc) + + @dataclass(frozen=True, slots=True) class _DemMesh: """What ``_dem_mesh`` made: the mesh and its file fields, then what @@ -901,7 +975,8 @@ class _DemMesh: def _dem_mesh( - dem: Path, + tile: DemTile, + dem: str, stride: int | None, delaunay: bool, spacing: float, @@ -912,7 +987,7 @@ def _dem_mesh( min_angle: float = 0.0, feet: bool = False, ) -> _DemMesh: - """Decode, subsample, triangulate, sample or refine, and trim. + """Subsample, triangulate, sample or refine, and trim ``tile``. Without ``tolerance`` this is increment 12's R6: z sampled bilinearly at the stride grid. With it, increment 14's R9: the stride grid is the start @@ -922,19 +997,9 @@ def _dem_mesh( Returns the mesh, the ``elevation`` sentence for the file, the EPSG code, the ``domain`` field (empty without one), and the ``--stats`` inputs; ``clock`` gets R5's phases. - Every refusal is a usage error in the reader's or the engine's own words, - and no file is written. + ``dem`` names the source in messages. Every refusal is a usage error in the + engine's own words, and no file is written. """ - try: - with clock.phase("decode"), dem.open("rb") as stream: - tile = decode_dem(stream) - except OSError as exc: - raise typer.BadParameter( - f"cannot read {dem}: {exc.strerror or exc}", param_hint="--dem" - ) from exc - except GeoTiffError as exc: - raise typer.BadParameter(str(exc), param_hint="--dem") from exc - meta = tile.meta described = "" domain_vertices = domain_holes = None diff --git a/src_python/tin_engine/dem_input.py b/src_python/tin_engine/dem_input.py new file mode 100644 index 00000000..43a8cd37 --- /dev/null +++ b/src_python/tin_engine/dem_input.py @@ -0,0 +1,70 @@ +"""The DEM a run meshes, from a declarative request (increment 15a, R1). + +`docs/increments/15-dem-mosaic.md` R1 and R11. A `DemRequest` names the +sources, the box and the caller's NoData; `open_dem` lists the tiles, plans the +mosaic and assembles it. `cli.py` parses flags into a request and calls +`open_dem`; a GUI backend or an API worker builds the same request without +Typer. Besides `io/repository.py`, this is the one module below `cli.py` +with paths, and it only hands them to the repository. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from pathlib import Path +from typing import Self + +from pydantic import BaseModel, ConfigDict, model_validator + +from tin_engine.io.models import DemTile +from tin_engine.io.repository import TiffDemRepository +from tin_engine.mosaic import Bounds, MosaicPlan, Seam, assemble, plan_mosaic + + +class DemRequest(BaseModel): + """Exactly one directory of tiles, or one or more tile files (R11).""" + + model_config = ConfigDict(frozen=True) + + sources: tuple[Path, ...] + bounds: Bounds | None = None + nodata: float | None = None + + @model_validator(mode="after") + def _one_form(self) -> Self: + if not self.sources: + raise ValueError("no DEM source given") + directories = sum(p.is_dir() for p in self.sources) + if directories and len(self.sources) > 1: + raise ValueError( + f"give exactly one directory, or one or more files; got {len(self.sources)} " + f"sources of which {directories} are directories" + ) + return self + + +@dataclass(frozen=True, slots=True) +class DemInput: + """The tile to mesh, the plan it was assembled by, a name for the run (the + directory's name, or the stem of the first file as given), and the + mosaic's disagreeing seams (Ola's Q1 revised).""" + + tile: DemTile + plan: MosaicPlan + label: str + seams: tuple[Seam, ...] = () + + +def open_dem(request: DemRequest) -> DemInput: + """List, plan and assemble. Every refusal is a `ValueError` (`GeoTiffError`, + `MosaicError`) or, for a file that cannot be opened, an `OSError`.""" + first = request.sources[0] + if first.is_dir(): + repository = TiffDemRepository.from_directory(first, nodata=request.nodata) + label = first.resolve().name + else: + repository = TiffDemRepository(request.sources, nodata=request.nodata) + label = first.stem + plan = plan_mosaic(repository.footprints(), request.bounds, None) + mosaic = assemble(plan, repository.load) + return DemInput(tile=mosaic.tile, plan=plan, label=label, seams=mosaic.seams) diff --git a/src_python/tin_engine/io/__init__.py b/src_python/tin_engine/io/__init__.py index 13213b6a..e5922d55 100644 --- a/src_python/tin_engine/io/__init__.py +++ b/src_python/tin_engine/io/__init__.py @@ -3,11 +3,14 @@ `project_structure.md` marked this package as the home of "all file decoding"; increment 10 makes it the first *encoding* module too, and widens that line. The rule the package keeps either way is the one that matters: `io/` knows -formats and knows nothing about `_core`, so everything in it is testable with -no compiled extension in the process and no filesystem. +formats and knows nothing about `_core`, so no module in it needs a compiled +extension in the process. -Nothing here opens a file. A path belongs to `cli.py`, which is the only module -that has one (`06-cdt-viewer.md`, "no file is written below `cli.py`"). +Files are opened in exactly one module, `repository.py`, and only for reading +(increment 15a, Ola's Q3 ruling): a DEM in many tiles has to be listed and +read tile by tile, below `cli.py`. Every other module here takes or returns +streams and bytes, so it is testable with no filesystem, and nothing in `io/` +writes a file (`06-cdt-viewer.md`, "no file is written below `cli.py`"). """ from __future__ import annotations diff --git a/src_python/tin_engine/io/geotiff.py b/src_python/tin_engine/io/geotiff.py index 23838e66..490a3508 100644 --- a/src_python/tin_engine/io/geotiff.py +++ b/src_python/tin_engine/io/geotiff.py @@ -45,6 +45,27 @@ NodataSource = Literal["tag", "caller", "absent"] +def read_meta(source: BinaryIO, *, nodata: float | None = None) -> RasterMeta: + """The header phase of `decode_dem` alone (increment 15a, R3): no pixel is read. + + Every refusal `decode_dem` makes from the header fires here with the same + message, because both run the same `_header`. A file whose pixels are + damaged still gives its `RasterMeta`; `decode_dem` then refuses it at the + "pixel data" stage. `source` is read but not closed. + """ + return read_header(source, nodata=nodata)[0] + + +def read_header( + source: BinaryIO, *, nodata: float | None = None +) -> tuple[RasterMeta, np.dtype[Any]]: + """`read_meta`, and the dtype `decode_dem` will return: the file's through + `PROMOTION` (15a S2, so a mosaic's cap can count the canvas it will build).""" + with _tiff(source, nodata) as tif: + meta, dtype = _header(tif, nodata) + return meta, PROMOTION[dtype] + + def decode_dem(source: BinaryIO, *, nodata: float | None = None) -> DemTile: """Decode page 0 of the GeoTIFF in `source` into a `DemTile`. @@ -63,43 +84,60 @@ def decode_dem(source: BinaryIO, *, nodata: float | None = None) -> DemTile: while `DemTile` takes its read-only copy (§7). `source` is read but not closed; closing it is the caller's. """ + with _tiff(source, nodata) as tif: + meta, dtype = _header(tif, nodata) + with _stage("pixel data"): + array = tif.pages.first.asarray().astype(PROMOTION[dtype], copy=False) + return DemTile(meta=meta, array=array) + + +@contextmanager +def _tiff(source: BinaryIO, nodata: float | None) -> Iterator[tifffile.TiffFile]: + """The caller-argument check and the TIFF structure, shared by both phases.""" if isinstance(nodata, bool | np.bool_): raise TypeError(f"nodata= must be a number or None, not {type(nodata).__name__}") with _stage("TIFF structure"): tif = tifffile.TiffFile(source) with tif: - with _stage("TIFF structure"): - page, pages = tif.pages.first, tuple(tif.pages) # parses every IFD - _single_page(pages) - dtype = _check_page(page) - tie, scale = _georeferencing(page) - with _stage("GeoKey directory"): - geokeys: dict[str, Any] = tif.geotiff_metadata or {} - x_min, y_max, delta_x, delta_y, area = _placement(tie, scale, geokeys) - epsg = _projected_epsg(geokeys) - vertical = geokeys.get("VerticalUnitsGeoKey") - if vertical is not None and int(vertical) != METRE: - raise GeoTiffError( - f"VerticalUnitsGeoKey (4099) = {int(vertical)}; only metres ({METRE}) are read" - ) - sentinel, source_of = _nodata(page, dtype, nodata) - with _stage("pixel data"): - array = page.asarray().astype(PROMOTION[dtype], copy=False) - rows, cols = array.shape + yield tif + + +def _header(tif: tifffile.TiffFile, nodata: float | None) -> tuple[RasterMeta, np.dtype[Any]]: + """Everything before the pixels (R3): the §5 header refusals, then the meta. + + Returns the file dtype too, which `decode_dem` promotes by. `rows` and + `cols` come from ImageLength and ImageWidth, which `_check_page` has + bounded; `DemTile` checks them against the decoded array's shape. + """ + with _stage("TIFF structure"): + page, pages = tif.pages.first, tuple(tif.pages) # parses every IFD + _single_page(pages) + dtype = _check_page(page) + tie, scale = _georeferencing(page) + with _stage("GeoKey directory"): + geokeys: dict[str, Any] = tif.geotiff_metadata or {} + x_min, y_max, delta_x, delta_y, area = _placement(tie, scale, geokeys) + epsg = _projected_epsg(geokeys) + vertical = geokeys.get("VerticalUnitsGeoKey") + if vertical is not None and int(vertical) != METRE: + raise GeoTiffError( + f"VerticalUnitsGeoKey (4099) = {int(vertical)}; only metres ({METRE}) are read" + ) + sentinel, source_of = _nodata(page, dtype, nodata) meta = RasterMeta( x_min=x_min, y_max=y_max, delta_x=delta_x, delta_y=delta_y, - cols=cols, - rows=rows, + cols=int(page.imagewidth), + rows=int(page.imagelength), epsg=epsg, nodata=sentinel, nodata_source=source_of, pixel_is_area=area, vertical_unit_assumed=vertical is None, ) - return DemTile(meta=meta, array=array) + return meta, dtype @contextmanager @@ -320,4 +358,4 @@ def _same(a: float, b: float) -> bool: return a == b or (math.isnan(a) and math.isnan(b)) -__all__ = ["decode_dem"] +__all__ = ["decode_dem", "read_header", "read_meta"] diff --git a/src_python/tin_engine/io/models.py b/src_python/tin_engine/io/models.py index 387dc8bd..0c6793e1 100644 --- a/src_python/tin_engine/io/models.py +++ b/src_python/tin_engine/io/models.py @@ -83,6 +83,29 @@ def _read_only_c_contiguous(cls, value: npt.NDArray[Any]) -> npt.NDArray[Any]: copy.flags.writeable = False return copy.view() + @classmethod + def _adopt(cls, meta: RasterMeta, array: npt.NDArray[Any]) -> DemTile: + """Take `array` as the tile's own, without the copy (increment 15a, R7). + + Private: its one caller is `mosaic.assemble`, on a canvas it allocated + and never hands out writable, so a mosaic peaks at one canvas, not two. + The same checks as the public constructor, plus C-contiguity, which + the constructor gets from its copy. The **passed** buffer is set + read-only, so the concurrency rule (§7) holds: nobody keeps a writable + reference. + """ + if array.ndim != 2 or array.dtype not in (np.float32, np.float64): + raise ValueError( + f"need a 2-D float32 or float64 array, got {array.dtype} {array.shape}" + ) + if not array.flags.c_contiguous or array.shape != (meta.rows, meta.cols): + raise ValueError( + f"need a C-contiguous array of shape {(meta.rows, meta.cols)}, " + f"got {array.shape}, C-contiguous {array.flags.c_contiguous}" + ) + array.flags.writeable = False + return cls.model_construct(meta=meta, array=array) + @model_validator(mode="after") def _shape_agrees(self) -> Self: if (self.meta.rows, self.meta.cols) != self.array.shape: diff --git a/src_python/tin_engine/io/repository.py b/src_python/tin_engine/io/repository.py new file mode 100644 index 00000000..ed0d87cb --- /dev/null +++ b/src_python/tin_engine/io/repository.py @@ -0,0 +1,105 @@ +"""A DEM held as tiles on disk: headers first, pixels on request (increment 15a). + +`docs/increments/15-dem-mosaic.md` R1-R3 and Ola's Q3 ruling: this is the one +module in `io/` that opens files, and it opens them read-only (`"rb"`). It +only stores: which tiles there are (`footprints`, header-only) and one tile's +pixels (`load`). Choosing tiles and stitching them is `tin_engine.mosaic`'s, +the same grid arithmetic for any storage. +""" + +from __future__ import annotations + +from collections.abc import Callable, Iterable +from pathlib import Path +from typing import Any, Protocol + +import numpy as np +from pydantic import BaseModel, ConfigDict + +from .geotiff import decode_dem, read_header +from .models import DemTile, GeoTiffError, RasterMeta + +#: `from_directory` lists these suffixes, in any case (R2). +TILE_SUFFIXES = frozenset({".tif", ".tiff"}) + + +class TileFootprint(BaseModel): + """One tile as its header describes it: the file's name, its node grid, and + the dtype it decodes to (float32 unless given; S2).""" + + model_config = ConfigDict(frozen=True, arbitrary_types_allowed=True) + + name: str + meta: RasterMeta + dtype: np.dtype[Any] = np.dtype(np.float32) + + +class DemRepository(Protocol): + """Storage only (R1). Sync: `load` is blocking I/O, and an async caller + wraps it in `asyncio.to_thread`.""" + + def footprints(self) -> tuple[TileFootprint, ...]: + """Every tile's header, sorted by name.""" + ... + + def load(self, name: str) -> DemTile: + """The whole tile `name`, decoded. `KeyError` for a name not listed.""" + ... + + +class TiffDemRepository: + """GeoTIFF tiles given as paths. Construction reads no file. + + Paths are resolved and deduplicated, so two spellings of one file are one + tile. A footprint's name is the file name, so two files sharing one are + refused. `nodata` is the caller's sentinel for every tile (`decode_dem`). + """ + + def __init__(self, paths: Iterable[Path], *, nodata: float | None = None) -> None: + self._paths: dict[str, Path] = {} + for path in sorted({Path(p).resolve() for p in paths}, key=str): + if path.name in self._paths: + raise ValueError( + f"two tiles named {path.name}: {self._paths[path.name]} and {path}" + ) + self._paths[path.name] = path + self._nodata = nodata + self._footprints: tuple[TileFootprint, ...] | None = None + + @classmethod + def from_directory(cls, directory: Path, *, nodata: float | None = None) -> TiffDemRepository: + """The `*.tif` and `*.tiff` files directly in `directory`, not recursively. + + Side files (`.tfw`, `.aux.xml`) carry nothing the GeoTIFF tags do not. + """ + paths = [ + p for p in directory.iterdir() if p.suffix.lower() in TILE_SUFFIXES and p.is_file() + ] + if not paths: + raise ValueError(f"no .tif or .tiff file in the directory {directory}") + return cls(paths, nodata=nodata) + + def footprints(self) -> tuple[TileFootprint, ...]: + """Read every header once, on the first call, and keep them (R2).""" + if self._footprints is None: + headers = {name: self._read(name, read_header) for name in sorted(self._paths)} + self._footprints = tuple( + TileFootprint(name=name, meta=meta, dtype=dtype) + for name, (meta, dtype) in headers.items() + ) + return self._footprints + + def load(self, name: str) -> DemTile: + return self._read(name, decode_dem) + + def _read[T](self, name: str, reader: Callable[..., T]) -> T: + """Open tile `name` read-only and run `reader`; a refusal names the file.""" + path = self._paths[name] + with path.open("rb") as stream: + try: + return reader(stream, nodata=self._nodata) + except GeoTiffError as exc: + raise GeoTiffError(f"{path.name}: {exc}") from exc + + +__all__ = ["DemRepository", "TiffDemRepository", "TileFootprint"] diff --git a/src_python/tin_engine/mosaic.py b/src_python/tin_engine/mosaic.py new file mode 100644 index 00000000..ff8d46b2 --- /dev/null +++ b/src_python/tin_engine/mosaic.py @@ -0,0 +1,569 @@ +"""Many DEM tiles as one node grid: planning from headers, then assembly (increment 15a). + +`docs/increments/15-dem-mosaic.md` R1, R4, R5 and R7, with the two +corrections in its "Pinned by the red suite (15a)": tiles are selected by +index window, and coverage is checked per node. + +`plan_mosaic` is pure: footprints and a request in, a `MosaicPlan` out, and +every refusal a header can decide fires there, before any pixel (I6). +`assemble` reads pixels through a `load` callable and never sees a path. + +Positions are integers from here on. Each lattice has a reference node, its +north-west-most node over all its tiles, and a node's global index `(R, K)` +counts from it; the mosaic's first node is `X_ref + c0 * dx`, never a sum of +steps (I1, R12's global node identity). +""" + +from __future__ import annotations + +import math +import os +from collections.abc import Callable, Sequence +from dataclasses import dataclass +from typing import TYPE_CHECKING, Any, Self + +import numpy as np +import numpy.typing as npt +import shapely +from pydantic import BaseModel, ConfigDict, model_validator + +from tin_engine.io.models import DemTile, RasterMeta + +if TYPE_CHECKING: + from tin_engine.io.repository import TileFootprint + +#: DEM units (metres for DTM10). A seam counts a node only where the two +#: tiles differ by at least this (Ola, 2026-09-28: "Ignore below 1mm"). +SEAM_THRESHOLD = 0.001 + +#: Cells. Two tiles share a lattice when their node offsets are integers to +#: within this. Measured noise is at most 2.2e-11 cells (B3), and 0 for DTM10. +ALIGN_TOLERANCE = 1e-6 + + +class MosaicError(ValueError): + """A request the tiles cannot answer. A `ValueError`, like `GeoTiffError`.""" + + +def physical_memory() -> int: + """Bytes of physical memory, looked up at call time (macOS and Linux).""" + return os.sysconf("SC_PHYS_PAGES") * os.sysconf("SC_PAGE_SIZE") + + +class Bounds(BaseModel): + """A box in the DEM's CRS: finite, with `x_min < x_max` and `y_min < y_max`.""" + + model_config = ConfigDict(frozen=True) + + x_min: float + y_min: float + x_max: float + y_max: float + + @model_validator(mode="after") + def _a_box(self) -> Self: + corners = (self.x_min, self.y_min, self.x_max, self.y_max) + if not all(map(math.isfinite, corners)) or not ( + self.x_min < self.x_max and self.y_min < self.y_max + ): + raise ValueError(f"need finite x_min < x_max and y_min < y_max, got {corners}") + return self + + +class IndexWindow(BaseModel): + """`rows x cols` nodes starting at `(row0, col0)`.""" + + model_config = ConfigDict(frozen=True) + + row0: int + col0: int + rows: int + cols: int + + +class TilePlacement(BaseModel): + """One selected tile: `canvas` in canvas indices, `source` in the tile's own, + and the dtype its footprint says it decodes to.""" + + model_config = ConfigDict(frozen=True, arbitrary_types_allowed=True) + + name: str + meta: RasterMeta + canvas: IndexWindow + source: IndexWindow + dtype: np.dtype[Any] = np.dtype(np.float32) + + +class MosaicPlan(BaseModel): + """What `assemble` will build. `window` is the mosaic's first node as a + global index, and its shape; `tiles` is sorted by name.""" + + model_config = ConfigDict(frozen=True) + + meta: RasterMeta + reference: tuple[float, float] + window: IndexWindow + tiles: tuple[TilePlacement, ...] + + +@dataclass(frozen=True, slots=True) +class Seam: + """Two tiles whose valid values differ by at least `SEAM_THRESHOLD` at + `nodes` of the mosaic's nodes, by `largest` and `median` there (float64); + `first < second` by name.""" + + first: str + second: str + nodes: int + largest: float + median: float + + def entry(self) -> str: + """The `dem_seams` entry, unescaped.""" + _, _, n, largest, median = self.cells() + return f"{self.first} | {self.second}: nodes {n}, max {largest}, median {median}" + + def cells(self) -> tuple[str, str, str, str, str]: + """The `--stats` table row, numbers formatted as in `entry`.""" + return self.first, self.second, f"{self.nodes}", f"{self.largest:g}", f"{self.median:g}" + + +@dataclass(frozen=True, slots=True) +class Mosaic: + """The assembled tile, the plan it came from, as provenance, and each pair + of tiles whose overlap disagrees, sorted by name (Ola's Q1 revised).""" + + tile: DemTile + plan: MosaicPlan + seams: tuple[Seam, ...] = () + + +@dataclass(frozen=True, slots=True) +class _Lattice: + """One lattice's tiles (sorted by name), its reference node, each tile's + global index, and the box's window on it with the tiles that meet it.""" + + group: list[TileFootprint] + reference: tuple[float, float] + placed: list[_Placed] + window: tuple[int, int, int, int] + selected: list[_Placed] + + +@dataclass(frozen=True, slots=True) +class _Placed: + """A footprint and its global index on its lattice.""" + + footprint: TileFootprint + row: int + col: int + + def meets(self, r0: int, c0: int, r1: int, c1: int) -> bool: + """Its node index range meets `[r0, r1] x [c0, c1]`, closed on both sides.""" + m = self.footprint.meta + return ( + self.row <= r1 and self.row + m.rows > r0 and self.col <= c1 and self.col + m.cols > c0 + ) + + +def plan_mosaic( + footprints: Sequence[TileFootprint], bounds: Bounds | None = None, needed: Any = None +) -> MosaicPlan: + """Select the tiles the request needs, on one lattice, or refuse (R4). + + `bounds` is the box in the DEM's CRS; without it, the lattice's union. + `needed` is a shapely geometry in the DEM's CRS, closed: a node on its + boundary is needed. Without it, every node of the window is. + + When the request selects tiles on several lattices, the plan is on the one + whose own tiles cover every needed node (Ola's Q5 reading, `_covering`). + """ + if not footprints: + raise MosaicError("no tiles to plan a mosaic from") + groups: list[list[TileFootprint]] = [] + for footprint in sorted(footprints, key=lambda f: f.name): + group = next((g for g in groups if _aligned(g[0].meta, footprint.meta)), None) + if group is None: + groups.append([footprint]) + else: + group.append(footprint) + + chosen: list[_Lattice] = [] + for group in groups: + reference, placed = _placed(group) + window = _window(bounds, reference, group[0].meta, placed) + selected = [p for p in placed if window is not None and p.meets(*window)] + if window is not None and selected: + chosen.append(_Lattice(group, reference, placed, window, selected)) + if not chosen: + raise MosaicError(f"the box {bounds} meets no tile") + if len(chosen) > 1: + return plan_mosaic(_covering(chosen, bounds, needed), bounds, needed) + + lattice = chosen[0] + (x_ref, y_ref), (r0, c0, r1, c1), selected = lattice.reference, lattice.window, lattice.selected + rows, cols = r1 - r0 + 1, c1 - c0 + 1 + if rows < 2 or cols < 2: + raise MosaicError(f"the request is {rows} x {cols} nodes; a mesh needs at least 2 x 2") + dtype = np.result_type(*(p.footprint.dtype for p in selected)) + cap, size = physical_memory() // 2, rows * cols * dtype.itemsize + if size > cap: + raise MosaicError( + f"the request is {rows} x {cols} nodes, {size} bytes at {dtype}, over the cap " + f"of half the physical memory ({cap} bytes); narrow it with --bbox" + ) + vertical = any(p.footprint.meta.vertical_unit_assumed for p in selected) + meta = _grid(selected[0].footprint.meta, lattice.reference, lattice.window, vertical) + tiles = tuple(_placement(p, r0, c0, r1, c1) for p in selected) + uncovered = _uncovered(meta, tiles, needed) + if uncovered: + raise MosaicError(uncovered) + return MosaicPlan( + meta=meta, + reference=(x_ref, y_ref), + window=IndexWindow(row0=r0, col0=c0, rows=rows, cols=cols), + tiles=tiles, + ) + + +def assemble(plan: MosaicPlan, load: Callable[[str], DemTile]) -> Mosaic: + """Load the plan's tiles one at a time, in plan order, into one canvas (R5). + + One tile whose grid is the mosaic's is returned as loaded: no canvas, no + copy (I7). Otherwise the canvas is allocated once, NaN-filled, in the + planned dtype (the placements' result dtype, from the headers), and handed + to `DemTile` without a copy (R7). A hand-built footprint may under-state + its dtype, so a loaded tile that promotes further re-casts the canvas. + + Each tile is written whole, then dropped before the next load; what it + holds where it meets another tile's placement is kept as a strip. Once all + are in, every overlap is decided from the strips (Ola's Q1 revised, + `_decide`) and each pair that disagrees is reported (`_seam`). + + Memory: the peak is the canvas, the strips, and one load's own peak, which + is not one tile: decoding a DTM10 tile peaks at 2.0 to 3.0 tiles, + depending on the tile (the decoder's buffers and `DemTile`'s read-only + copy). Measured with + tracemalloc on the 15a acceptance box (nine DTM10 tiles, 404 MB canvas, + 102 MB tiles): 720 MB, the canvas plus 3.1 tiles. + """ + if len(plan.tiles) == 1 and plan.tiles[0].meta == plan.meta: + return Mosaic(tile=_loaded(plan.tiles[0], load), plan=plan) + if not plan.tiles: + raise MosaicError("the plan has no tiles") + planned = np.result_type(*(t.dtype for t in plan.tiles)) + canvas: npt.NDArray[Any] = np.full((plan.meta.rows, plan.meta.cols), np.nan, dtype=planned) + ordered = sorted(plan.tiles, key=lambda t: t.name) + strips: dict[tuple[str, str], npt.NDArray[Any]] = {} + for placement in plan.tiles: + array = _loaded(placement, load).array + if np.result_type(canvas.dtype, array.dtype) != canvas.dtype: + canvas = canvas.astype(np.result_type(canvas.dtype, array.dtype)) + c, s = placement.canvas, placement.source + incoming = array[s.row0 : s.row0 + s.rows, s.col0 : s.col0 + s.cols] + canvas[c.row0 : c.row0 + c.rows, c.col0 : c.col0 + c.cols] = incoming + for other in ordered: + box = _meet(c, other.canvas) + if other.name != placement.name and box is not None: + strips[placement.name, other.name] = incoming[_within(box, c)].copy() + del array, incoming # before the next load, or two tiles outlive this one + seams = [] + for i, a in enumerate(ordered): + for b in ordered[i + 1 :]: + box = _meet(a.canvas, b.canvas) + if box is None: + continue + _decide(canvas, box, ordered, strips, (a.name, b.name), plan.meta.nodata) + seam = _seam(a.name, b.name, strips[a.name, b.name], strips[b.name, a.name], plan) + if seam is not None: + seams.append(seam) + return Mosaic(tile=DemTile._adopt(plan.meta, canvas), plan=plan, seams=tuple(seams)) + + +def _aligned(a: RasterMeta, b: RasterMeta) -> bool: + """Same CRS, spacing, registration and NoData, and integer node offsets (R4.1).""" + return _key(a) == _key(b) and all(abs(v - round(v)) <= ALIGN_TOLERANCE for v in _cells(a, b)) + + +def _key(m: RasterMeta) -> tuple[int, float, float, bool, float | None]: + return m.epsg, m.delta_x, m.delta_y, m.pixel_is_area, m.nodata + + +def _cells(a: RasterMeta, b: RasterMeta) -> tuple[float, float]: + """`b`'s first node from `a`'s, in `a`'s cells east and south.""" + return (b.x_min - a.x_min) / a.delta_x, (a.y_max - b.y_max) / a.delta_y + + +def _placed(group: list[TileFootprint]) -> tuple[tuple[float, float], list[_Placed]]: + """The lattice's reference node (R4.2), and each tile's global index from it.""" + x_ref = min(f.meta.x_min for f in group) + y_ref = max(f.meta.y_max for f in group) + dx, dy = group[0].meta.delta_x, group[0].meta.delta_y + return (x_ref, y_ref), [ + _Placed(f, round((y_ref - f.meta.y_max) / dy), round((f.meta.x_min - x_ref) / dx)) + for f in group + ] + + +def _window( + bounds: Bounds | None, reference: tuple[float, float], m: RasterMeta, placed: list[_Placed] +) -> tuple[int, int, int, int] | None: + """`(r0, c0, r1, c1)`, inclusive: the box snapped outward, clamped to the + lattice's union (R4.4); None when the box misses the union.""" + last_row = max(p.row + p.footprint.meta.rows for p in placed) - 1 + last_col = max(p.col + p.footprint.meta.cols for p in placed) - 1 + return _clamp(_snapped(bounds, reference, m), (0, 0, last_row, last_col)) + + +def _snapped( + bounds: Bounds | None, reference: tuple[float, float], m: RasterMeta +) -> tuple[int, int, int, int] | None: + """The box snapped outward to `m`'s lattice, unclamped; None for no box.""" + if bounds is None: + return None + (x_ref, y_ref), dx, dy = reference, m.delta_x, m.delta_y + return ( + math.floor(_snap((y_ref - bounds.y_max) / dy)), + math.floor(_snap((bounds.x_min - x_ref) / dx)), + math.ceil(_snap((y_ref - bounds.y_min) / dy)), + math.ceil(_snap((bounds.x_max - x_ref) / dx)), + ) + + +def _clamp( + window: tuple[int, int, int, int] | None, limits: tuple[int, int, int, int] +) -> tuple[int, int, int, int] | None: + """`window` (None: unbounded) inside `limits`, or None when they miss.""" + r0, c0, r1, c1 = limits if window is None else window + r0, c0 = max(r0, limits[0]), max(c0, limits[1]) + r1, c1 = min(r1, limits[2]), min(c1, limits[3]) + return (r0, c0, r1, c1) if r0 <= r1 and c0 <= c1 else None + + +def _covering(chosen: list[_Lattice], bounds: Bounds | None, needed: Any) -> list[TileFootprint]: + """Ola's Q5 reading: of the lattices the request selects tiles on, the one + whose own tiles cover every node it needs; several, the most tiles, ties by + the first tile's name; none, the Q5 refusal. + + A lattice needs its nodes in the box snapped outward, clamped to the + bounding box of every selected tile on any lattice, not to its own union, + or a box running past it into another lattice's tiles would be cut short. + """ + corners = [(p.footprint.meta, lattice) for lattice in chosen for p in lattice.selected] + x_lo = min(m.x_min for m, _ in corners) + x_hi = max(m.x_min + (m.cols - 1) * m.delta_x for m, _ in corners) + y_hi = max(m.y_max for m, _ in corners) + y_lo = min(m.y_max - (m.rows - 1) * m.delta_y for m, _ in corners) + covering = [] + for lattice in chosen: + (x_ref, y_ref), m = lattice.reference, lattice.group[0].meta + inside = ( + math.ceil(_snap((y_ref - y_hi) / m.delta_y)), + math.ceil(_snap((x_lo - x_ref) / m.delta_x)), + math.floor(_snap((y_ref - y_lo) / m.delta_y)), + math.floor(_snap((x_hi - x_ref) / m.delta_x)), + ) + window = _clamp(_snapped(bounds, lattice.reference, m), inside) + if window is None: + continue + tiles = tuple(_placement(p, *window) for p in lattice.placed if p.meets(*window)) + if not _uncovered(_grid(m, lattice.reference, window, False), tiles, needed): + covering.append(lattice) + if not covering: + raise MosaicError(_mixed(chosen[0].selected[0].footprint, chosen[1].selected[0].footprint)) + return min(covering, key=lambda c: (-len(c.group), c.group[0].name)).group + + +def _grid( + first: RasterMeta, + reference: tuple[float, float], + window: tuple[int, int, int, int], + vertical_unit_assumed: bool, +) -> RasterMeta: + """The node grid of `window` on `first`'s lattice.""" + r0, c0, r1, c1 = window + return RasterMeta( + x_min=reference[0] + c0 * first.delta_x, + y_max=reference[1] - r0 * first.delta_y, + delta_x=first.delta_x, + delta_y=first.delta_y, + cols=c1 - c0 + 1, + rows=r1 - r0 + 1, + epsg=first.epsg, + nodata=first.nodata, + nodata_source=first.nodata_source, + pixel_is_area=first.pixel_is_area, + vertical_unit_assumed=vertical_unit_assumed, + ) + + +def _snap(cells: float) -> float: + """A box edge within `ALIGN_TOLERANCE` of a node is on it, so float noise + in a spacing like 0.1 does not grow the window by a line.""" + nearest = round(cells) + return float(nearest) if abs(cells - nearest) <= ALIGN_TOLERANCE else cells + + +def _placement(p: _Placed, r0: int, c0: int, r1: int, c1: int) -> TilePlacement: + """Where tile `p` sits in the canvas, and which part of it is used.""" + m = p.footprint.meta + rs, re = max(r0, p.row), min(r1, p.row + m.rows - 1) + cs, ce = max(c0, p.col), min(c1, p.col + m.cols - 1) + rows, cols = re - rs + 1, ce - cs + 1 + return TilePlacement( + name=p.footprint.name, + meta=m, + canvas=IndexWindow(row0=rs - r0, col0=cs - c0, rows=rows, cols=cols), + source=IndexWindow(row0=rs - p.row, col0=cs - p.col, rows=rows, cols=cols), + dtype=p.footprint.dtype, + ) + + +def _uncovered(meta: RasterMeta, tiles: tuple[TilePlacement, ...], needed: Any) -> str | None: + """Why a needed node no tile covers refuses the request (R4.5, per node), + or None. A mask, a count and per-axis reductions: no per-node index or + coordinate array unless `needed` must be tested node by node (S1).""" + uncovered = np.ones((meta.rows, meta.cols), dtype=bool) + for t in tiles: + c = t.canvas + uncovered[c.row0 : c.row0 + c.rows, c.col0 : c.col0 + c.cols] = False + if needed is not None and uncovered.any(): + rows, cols = np.nonzero(uncovered) + outside = ~shapely.intersects_xy( + needed, meta.x_min + cols * meta.delta_x, meta.y_max - rows * meta.delta_y + ) + uncovered[rows[outside], cols[outside]] = False + count = np.count_nonzero(uncovered) + if not count: + return None + rows_hit = np.flatnonzero(uncovered.any(axis=1)) + cols_hit = np.flatnonzero(uncovered.any(axis=0)) + x0, x1 = (meta.x_min + cols_hit[i] * meta.delta_x for i in (0, -1)) + y1, y0 = (meta.y_max - rows_hit[i] * meta.delta_y for i in (0, -1)) + return ( + f"{count} nodes the request needs are in no tile: x {_num(x0)} to {_num(x1)}, " + f"y {_num(y0)} to {_num(y1)} (EPSG:{meta.epsg})" + ) + + +def _loaded(placement: TilePlacement, load: Callable[[str], DemTile]) -> DemTile: + tile = load(placement.name) + if tile.meta != placement.meta: + raise MosaicError( + f"{placement.name} changed since it was listed: {tile.meta} is not {placement.meta}" + ) + return tile + + +def _meet(a: IndexWindow, b: IndexWindow) -> IndexWindow | None: + """The canvas nodes both windows hold, or None.""" + r0, c0 = max(a.row0, b.row0), max(a.col0, b.col0) + r1, c1 = min(a.row0 + a.rows, b.row0 + b.rows), min(a.col0 + a.cols, b.col0 + b.cols) + if r0 >= r1 or c0 >= c1: + return None + return IndexWindow(row0=r0, col0=c0, rows=r1 - r0, cols=c1 - c0) + + +def _within(inner: IndexWindow, outer: IndexWindow) -> tuple[slice, slice]: + """`inner`'s slices in an array holding `outer` (both in canvas indices).""" + r, c = inner.row0 - outer.row0, inner.col0 - outer.col0 + return slice(r, r + inner.rows), slice(c, c + inner.cols) + + +def _valid(values: npt.NDArray[Any], nodata: float | None) -> npt.NDArray[np.bool_]: + valid = ~np.isnan(values) + if nodata is not None: + valid &= values != nodata + return valid + + +def _decide( + canvas: npt.NDArray[Any], + box: IndexWindow, + ordered: list[TilePlacement], + strips: dict[tuple[str, str], npt.NDArray[Any]], + pair: tuple[str, str], + nodata: float | None, +) -> None: + """Ola's Q1 revised over `box`, the overlap of `pair`: each node takes the + valid value of the tile it lies deepest in, depth being + `min(r, c, rows - 1 - r, cols - 1 - c)` in the whole tile's own indices; + ties to the name that sorts first (`ordered` is by name, and only a deeper + tile replaces). No valid value: the sentinel if any tile holds it, else + NaN, in every order. A third tile's values come from its strip with one + of the pair, which holds every node of `box` it covers.""" + best = np.full((box.rows, box.cols), -1, dtype=np.int64) + value = np.full((box.rows, box.cols), np.nan, dtype=canvas.dtype) + sentinel = np.zeros((box.rows, box.cols), dtype=bool) + for t in ordered: + part = _meet(t.canvas, box) + if part is None: + continue + other = next(p for p in ordered if p.name == (pair[1] if t.name == pair[0] else pair[0])) + held = _meet(t.canvas, other.canvas) + assert held is not None # it holds `part` + values = strips[t.name, other.name][_within(part, held)] + row = np.arange(part.rows) + part.row0 - t.canvas.row0 + t.source.row0 + col = np.arange(part.cols) + part.col0 - t.canvas.col0 + t.source.col0 + depth = np.minimum.outer( + np.minimum(row, t.meta.rows - 1 - row), np.minimum(col, t.meta.cols - 1 - col) + ) + here = _within(part, box) + take = _valid(values, nodata) & (depth > best[here]) + value[here][take] = values[take] + best[here][take] = depth[take] + if nodata is not None: + sentinel[here] |= values == nodata + if nodata is not None: + value[(best < 0) & sentinel] = nodata + canvas[box.row0 : box.row0 + box.rows, box.col0 : box.col0 + box.cols] = value + + +def _seam( + first: str, second: str, a: npt.NDArray[Any], b: npt.NDArray[Any], plan: MosaicPlan +) -> Seam | None: + """The pair's report over its overlap in the mosaic: nodes where both hold + a valid value and `|a - b| >= SEAM_THRESHOLD`, and the largest and median + `|a - b|` over those, in float64. None when no node qualifies. The + threshold is the report's only: `_decide` never consults it.""" + both = _valid(a, plan.meta.nodata) & _valid(b, plan.meta.nodata) + gaps = np.abs(a[both].astype(np.float64) - b[both].astype(np.float64)) + gaps = gaps[gaps >= SEAM_THRESHOLD] + if not gaps.size: + return None + return Seam(first, second, int(gaps.size), float(gaps.max()), float(np.median(gaps))) + + +def _mixed(a: TileFootprint, b: TileFootprint) -> str: + """Why `b` is not on `a`'s lattice (R4.3, Q5), in the terms that differ.""" + ma, mb = a.meta, b.meta + if ma.epsg != mb.epsg: + why = f"is in EPSG:{mb.epsg}, not EPSG:{ma.epsg}" + elif (ma.delta_x, ma.delta_y) != (mb.delta_x, mb.delta_y): + why = f"has spacing {mb.delta_x!r} x {mb.delta_y!r}, not {ma.delta_x!r} x {ma.delta_y!r}" + elif ma.pixel_is_area != mb.pixel_is_area: + why = f"has registration {_registration(mb)}, not {_registration(ma)}" + elif ma.nodata != mb.nodata: + why = f"has nodata {mb.nodata}, not {ma.nodata}" + else: + offsets = [ + f"{abs(v - round(v)):.6g} cell ({abs(v - round(v)) * h:.6g} m) {axis}" + for v, h, axis in zip( + _cells(ma, mb), (ma.delta_x, ma.delta_y), ("east-west", "north-south"), strict=True + ) + if abs(v - round(v)) > ALIGN_TOLERANCE + ] + why = f"is {' and '.join(offsets)} off {a.name}'s lattice" + return ( + f"the request selects tiles on two lattices, {a.name} and {b.name}: {b.name} {why}; " + "tiles are not resampled onto one another, but a --bbox inside one lattice is meshed" + ) + + +def _registration(m: RasterMeta) -> str: + return "pixel-is-area" if m.pixel_is_area else "pixel-is-point" + + +def _num(value: float | np.floating[Any]) -> str: + """A coordinate written out, never in scientific notation.""" + return f"{value:f}".rstrip("0").rstrip(".") diff --git a/src_python/tin_engine/stats.py b/src_python/tin_engine/stats.py index 6d52a00e..367835ca 100644 --- a/src_python/tin_engine/stats.py +++ b/src_python/tin_engine/stats.py @@ -153,6 +153,9 @@ class Report: total: float stats_seconds: float threads: int | None = None + #: One row per disagreeing pair of DEM tiles: two names, nodes, max and + #: median, already formatted (`mosaic.Seam.cells`; Ola's Q1 revised). + seams: Sequence[Sequence[str]] = () def _exact(value: float) -> str: @@ -175,8 +178,9 @@ def _bytes(n: int) -> str: def _table(header: Sequence[str], rows: Sequence[Sequence[str]]) -> list[str]: + """A Markdown table; a `|` in a cell (a tile name) is escaped `\\|`.""" lines = ["| " + " | ".join(header) + " |", "|" + "---|" * len(header)] - return lines + ["| " + " | ".join(row) + " |" for row in rows] + return lines + ["| " + " | ".join(c.replace("|", "\\|") for c in row) + " |" for row in rows] def _sizes(s: Sizes) -> list[str]: @@ -279,6 +283,9 @@ def render(report: Report) -> str: """The Markdown report (R4). Pure: no clock, no I/O.""" lines = ["# rasputin mesh — statistics", "", f"`{report.command}`", ""] lines += ["## Sizes", "", *_sizes(report.sizes), ""] + if report.seams: + header = ("tile", "tile", "nodes", "max", "median") + lines += ["## DEM seams", "", *_table(header, report.seams), ""] lines += ["## Quality (plan view, x/y)", "", *_quality(report.quality), ""] if report.refinement is not None: lines += ["## Refinement", "", *_refinement(report.refinement), ""] diff --git a/tests/fixtures/dtm10/extract.py b/tests/fixtures/dtm10/extract.py new file mode 100644 index 00000000..3206b0ee --- /dev/null +++ b/tests/fixtures/dtm10/extract.py @@ -0,0 +1,110 @@ +"""Cut increment 15a's real multi-tile fixtures out of Ola's DTM10 archive. + + python tests/fixtures/dtm10/extract.py ../rasputin_data/DTM10_UTM33_20220924 + +Test support, not production code (`15-dem-mosaic.md`, "Test data"). Every +window is cut from the **archive**, never from the committed benchmark tile, +which is a different release (N5). Source: Kartverket's DTM10, UTM 33, the +archive's 2022-09-24 release, © Kartverket, CC BY 4.0. + +Two fixture directories, each usable as ``--dem DIR``: + +- ``seam/``: 6400_4 | 6400_1, N3's first edge pair. East-west neighbours whose + last and first 51 columns are the same nodes. Rows 3072-3327 of both, and + the 307 columns nearest the seam of each (256 own plus the 51-column + overlap). Every overlapping node is valid in both and equal (asserted here). +- ``lattices/``: 7707_1, one of N2's eight tiles half a cell east-west off the + main lattice, over 7707_2, its aligned southern neighbour. 7707_1's last 96 + rows and 7707_2's first 96, so they share 51 node rows (in y), in columns + that meet in x but are 5 m apart. + +Written with the tiles' own georeferencing (tie point shifted to the window, +area-registered, EPSG:25833, NoData -32767 in tag 42113), float32, Deflate, so +reading them needs no `imagecodecs`. The file names are the source tiles'. +""" + +from __future__ import annotations + +import sys +from pathlib import Path + +import numpy as np +import tifffile + +HERE = Path(__file__).resolve().parent +sys.path.insert(0, str(HERE.parents[1] / "python")) + +from geotiff_fixtures import ( # noqa: E402 + GEOGRAPHIC_TYPE, + GT_MODEL_TYPE, + GT_RASTER_TYPE, + METRE, + PIXEL_IS_AREA, + PROJ_LINEAR_UNITS, + PROJECTED_CS_TYPE, + micro_tiff, +) + +EPSG = 25833 +ETRS89 = 4258 +NODATA = -32767 +KEYS = { + GT_MODEL_TYPE: 1, + GT_RASTER_TYPE: PIXEL_IS_AREA, + GEOGRAPHIC_TYPE: ETRS89, + PROJECTED_CS_TYPE: EPSG, + PROJ_LINEAR_UNITS: METRE, +} + + +def cut(source: Path, rows: slice, cols: slice, target: Path) -> np.ndarray: + """Write `source[rows, cols]` to `target` with the window's own tie point.""" + with tifffile.TiffFile(source) as tif: + page = tif.pages.first + tie = page.tags[33922].value + scale = page.tags[33550].value + assert tif.geotiff_metadata["ProjectedCSTypeGeoKey"] == EPSG + assert int(tif.geotiff_metadata["GTRasterTypeGeoKey"]) == PIXEL_IS_AREA + assert page.tags[42113].value.strip() == str(NODATA) + array = page.asarray() + window = np.ascontiguousarray(array[rows, cols]) + x0 = tie[3] + cols.start * scale[0] + y0 = tie[4] - rows.start * scale[1] + stream = micro_tiff( + window, + tiepoint=(0.0, 0.0, 0.0, x0, y0, 0.0), + scale=tuple(scale), + geokeys=KEYS, + nodata=str(NODATA), + compression="deflate", + ) + target.parent.mkdir(parents=True, exist_ok=True) + target.write_bytes(stream.getvalue()) + return window + + +def main(archive: Path) -> None: + name = "{}_10m_z33.tif".format + rows = slice(3072, 3328) + west = cut( + archive / name("6400_4"), rows, slice(5051 - 307, 5051), HERE / "seam" / name("6400_4") + ) + east = cut(archive / name("6400_1"), rows, slice(0, 307), HERE / "seam" / name("6400_1")) + overlap_west, overlap_east = west[:, -51:], east[:, :51] + assert (overlap_west != NODATA).all() and (overlap_east != NODATA).all() + assert np.array_equal(overlap_west, overlap_east), "the seam must agree bit for bit" + # The control: shifted by one column, the overlap must disagree somewhere. + assert not np.array_equal(west[:, -52:-1], overlap_east) + + cols = slice(2000, 2096) + north = cut( + archive / name("7707_1"), slice(5053 - 96, 5053), cols, HERE / "lattices" / name("7707_1") + ) + south = cut(archive / name("7707_2"), slice(0, 96), cols, HERE / "lattices" / name("7707_2")) + assert (north != NODATA).mean() > 0.9 and (south != NODATA).mean() > 0.9 + for path in sorted(HERE.glob("*/*.tif")): + print(f"{path.relative_to(HERE)}: {path.stat().st_size} bytes") + + +if __name__ == "__main__": + main(Path(sys.argv[1])) diff --git a/tests/fixtures/dtm10/lattices/7707_1_10m_z33.tif b/tests/fixtures/dtm10/lattices/7707_1_10m_z33.tif new file mode 100644 index 00000000..88c43a0a Binary files /dev/null and b/tests/fixtures/dtm10/lattices/7707_1_10m_z33.tif differ diff --git a/tests/fixtures/dtm10/lattices/7707_2_10m_z33.tif b/tests/fixtures/dtm10/lattices/7707_2_10m_z33.tif new file mode 100644 index 00000000..7cd215b1 Binary files /dev/null and b/tests/fixtures/dtm10/lattices/7707_2_10m_z33.tif differ diff --git a/tests/fixtures/dtm10/seam/6400_1_10m_z33.tif b/tests/fixtures/dtm10/seam/6400_1_10m_z33.tif new file mode 100644 index 00000000..3c5310d6 Binary files /dev/null and b/tests/fixtures/dtm10/seam/6400_1_10m_z33.tif differ diff --git a/tests/fixtures/dtm10/seam/6400_4_10m_z33.tif b/tests/fixtures/dtm10/seam/6400_4_10m_z33.tif new file mode 100644 index 00000000..2c799ff1 Binary files /dev/null and b/tests/fixtures/dtm10/seam/6400_4_10m_z33.tif differ diff --git a/tests/python/mosaic_fixtures.py b/tests/python/mosaic_fixtures.py new file mode 100644 index 00000000..ee6c9539 --- /dev/null +++ b/tests/python/mosaic_fixtures.py @@ -0,0 +1,271 @@ +"""Tiles for increment 15a's suites, built from `RasterMeta` and `DemTile` directly. + +`15-dem-mosaic.md`, "Tests for @tester": planning and assembly tests need no +file, and a dict is the repository. Everything here imports only what +increment 11 already shipped (`io.models`), so a missing 15a module fails the +tests that use it, not the collection of this helper. + +The whole grids are asymmetric (rows != cols, `delta_x` != `delta_y`, every +node a distinct value), so a transposed index or a swapped axis cannot pass. +Values are small integers plus halves, exact in float32. +""" + +from __future__ import annotations + +from collections.abc import Callable, Iterable, Mapping +from typing import Any + +import numpy as np + +from tin_engine.io.models import DemTile, RasterMeta + +X0 = 500_000.0 +Y0 = 6_600_000.0 +DX = 10.0 +DY = 5.0 +EPSG = 25833 +SENTINEL = -32767.0 + + +def meta( + *, + rows: int, + cols: int, + x_min: float = X0, + y_max: float = Y0, + dx: float = DX, + dy: float = DY, + epsg: int = EPSG, + nodata: float | None = None, + area: bool = False, + vertical_unit_assumed: bool = True, + nodata_source: str | None = None, +) -> RasterMeta: + source = nodata_source or ("absent" if nodata is None else "tag") + return RasterMeta( + x_min=x_min, + y_max=y_max, + delta_x=dx, + delta_y=dy, + cols=cols, + rows=rows, + epsg=epsg, + nodata=nodata, + nodata_source=source, + pixel_is_area=area, + vertical_unit_assumed=vertical_unit_assumed, + ) + + +def values(rows: int, cols: int, dtype: Any = np.float32) -> np.ndarray: + """Node (r, c) holds `100 r + c + 0.5`: distinct, and exact in float32.""" + r, c = np.indices((rows, cols)) + return np.asarray(100 * r + c + 0.5).astype(dtype) + + +def whole(rows: int = 9, cols: int = 13, **kwargs: Any) -> DemTile: + """One tile to cut up. Default 9 x 13, point-registered, dx 10, dy 5.""" + dtype = kwargs.pop("dtype", np.float32) + array = kwargs.pop("array", None) + grid = values(rows, cols, dtype) if array is None else array + return DemTile(meta=meta(rows=rows, cols=cols, **kwargs), array=grid) + + +def piece(source: DemTile, r0: int, r1: int, c0: int, c1: int, **changes: Any) -> DemTile: + """`source[r0:r1, c0:c1]` as its own tile, placed where a producer would put it. + + `x_min` is computed as `x_min + c0 * dx`, the way a producer's tie point + would give it; `changes` override `RasterMeta` fields (to plant a defect), + and `array=` replaces the values. + """ + m = source.meta + array = changes.pop("array", None) + fields = m.model_dump() + fields.update( + x_min=m.x_min + c0 * m.delta_x, + y_max=m.y_max - r0 * m.delta_y, + rows=r1 - r0, + cols=c1 - c0, + ) + fields.update(changes) + grid = source.array[r0:r1, c0:c1] if array is None else array + return DemTile(meta=RasterMeta(**fields), array=grid) + + +def quadrants(source: DemTile, row_cut: int, col_cut: int, overlap: int) -> dict[str, DemTile]: + """Four tiles, named by compass, sharing `overlap` node lines at each cut. + + `overlap` 0 is area-registered neighbours that abut (disjoint node sets), + 1 is point-registered neighbours sharing a row and a column, more is a + true overlap. North and west tiles extend `overlap` lines past the cut. + """ + rows, cols = source.array.shape + north, south = (0, row_cut + overlap), (row_cut, rows) + west, east = (0, col_cut + overlap), (col_cut, cols) + return { + "nw.tif": piece(source, *north, *west), + "ne.tif": piece(source, *north, *east), + "sw.tif": piece(source, *south, *west), + "se.tif": piece(source, *south, *east), + } + + +def blocks( + source: DemTile, rows: int, cols: int, skip: Iterable[tuple[int, int]] = () +) -> dict[str, DemTile]: + """`source` cut into abutting `rows x cols`-node blocks, named `b.tif`. + + Blocks at the (block row, block column) pairs in `skip` are left out, to + make a hole in the tile set. + """ + out: dict[str, DemTile] = {} + omitted = set(skip) + n_rows, n_cols = source.array.shape + for i in range(n_rows // rows): + for j in range(n_cols // cols): + if (i, j) not in omitted: + out[f"b{i}{j}.tif"] = piece( + source, i * rows, (i + 1) * rows, j * cols, (j + 1) * cols + ) + return out + + +class Loads: + """A `load` for `assemble`: a dict lookup that records every call.""" + + def __init__(self, tiles: Mapping[str, DemTile]) -> None: + self.tiles = dict(tiles) + self.calls: list[str] = [] + + def __call__(self, name: str) -> DemTile: + self.calls.append(name) + return self.tiles[name] + + +def never(name: str) -> DemTile: + """A `load` that must not be reached (I6: header refusals fire before pixels).""" + raise AssertionError(f"load({name!r}) was called") + + +def footprints(footprint: Callable[..., Any], tiles: Mapping[str, DemTile]) -> list[Any]: + """`TileFootprint`s for `tiles`, built with the class the test fixture imported.""" + return [footprint(name=name, meta=tile.meta) for name, tile in tiles.items()] + + +def same_array(a: np.ndarray, b: np.ndarray) -> bool: + """Equal bit for bit: dtype, shape and bytes (so NaN matches NaN, -0 != 0).""" + return a.dtype == b.dtype and a.shape == b.shape and a.tobytes() == b.tobytes() + + +# --------------------------------------------------------------------------- +# Ola's Q1 revised (2026-09-28): the oracle for overlaps and seams +# --------------------------------------------------------------------------- + + +def own_depth(rows: int, cols: int) -> np.ndarray: + """Each node's distance, in nodes, to its own tile's nearest border. + + `min(r, c, rows - 1 - r, cols - 1 - c)`: 0 on the border. The tile's own + border is its whole grid (`meta.rows x meta.cols`), never the window a + request uses of it and never the mosaic's. + """ + r, c = np.indices((rows, cols)) + return np.minimum.reduce([r, c, rows - 1 - r, cols - 1 - c]) + + +def on_canvas( + tiles: Mapping[str, DemTile], grid: RasterMeta +) -> dict[str, tuple[np.ndarray, np.ndarray]]: + """Each tile's valid values (float64) and own depths on `grid`'s nodes. + + NaN and -1 where the tile does not cover the node or holds NoData there + (NaN, or its sentinel). Tiles are placed by their coordinates, so this + shares no index arithmetic with `plan_mosaic`. + """ + out: dict[str, tuple[np.ndarray, np.ndarray]] = {} + for name, tile in tiles.items(): + m = tile.meta + row0 = round((grid.y_max - m.y_max) / grid.delta_y) + col0 = round((m.x_min - grid.x_min) / grid.delta_x) + value = np.full((grid.rows, grid.cols), np.nan) + depth = np.full((grid.rows, grid.cols), -1) + array = np.asarray(tile.array, dtype=np.float64) + valid = ~np.isnan(array) + if m.nodata is not None: + valid &= array != m.nodata + tile_depth = own_depth(m.rows, m.cols) + for i, j in zip(*np.nonzero(valid), strict=True): + r, c = row0 + int(i), col0 + int(j) + if 0 <= r < grid.rows and 0 <= c < grid.cols: + value[r, c], depth[r, c] = array[i, j], tile_depth[i, j] + out[name] = (value, depth) + return out + + +def deepest_interior(tiles: Mapping[str, DemTile], grid: RasterMeta) -> np.ndarray: + """The Q1-revised mosaic, node by node: of the tiles holding a valid value, + the one the node lies deepest in; ties to the name that sorts first. + NaN where no tile holds a valid value.""" + placed = on_canvas(tiles, grid) + expected = np.full((grid.rows, grid.cols), np.nan) + for r in range(grid.rows): + for c in range(grid.cols): + held = [(-d[r, c], name) for name, (v, d) in placed.items() if d[r, c] >= 0] + if held: + expected[r, c] = placed[min(held)[1]][0][r, c] + return expected + + +def winners(tiles: Mapping[str, DemTile], grid: RasterMeta) -> list[list[str]]: + """The name of the tile each node's value is taken from, '' for none.""" + placed = on_canvas(tiles, grid) + return [ + [ + min(((-d[r, c], n) for n, (_, d) in placed.items() if d[r, c] >= 0), default=(0, ""))[1] + for c in range(grid.cols) + ] + for r in range(grid.rows) + ] + + +#: Ola, 2026-09-28: "Ignore below 1mm". A seam counts a node only where +#: |a - b| >= 1 mm, 0.001 in the DEM's (metre) units, compared in float64. +SEAM_THRESHOLD = 0.001 + + +def seams_of( + tiles: Mapping[str, DemTile], grid: RasterMeta +) -> list[tuple[str, str, int, float, float]]: + """The seam report the rule asks for, pair by pair, sorted by name: + `(first, second, nodes, largest, median)` over the mosaic's nodes where + both tiles hold a valid value and `|a - b| >= SEAM_THRESHOLD`; + `|difference|` in float64, the median of an even count the mean of the + middle two. Pairs with no such node are left out.""" + placed = on_canvas(tiles, grid) + out = [] + names = sorted(placed) + for i, a in enumerate(names): + for b in names[i + 1 :]: + va, vb = placed[a][0], placed[b][0] + both = ~np.isnan(va) & ~np.isnan(vb) + differ = both & (np.abs(np.where(both, va - vb, 0.0)) >= SEAM_THRESHOLD) + if differ.any(): + gaps = np.abs(va[differ] - vb[differ]) + out.append((a, b, int(differ.sum()), float(gaps.max()), float(np.median(gaps)))) + return out + + +def seams(result: Any) -> list[tuple[str, str, int, float, float]]: + """A `Mosaic`'s (or `DemInput`'s) seam report as plain tuples.""" + return [(s.first, s.second, s.nodes, s.largest, s.median) for s in result.seams] + + +def shifted_by(tile: DemTile, offset: float) -> DemTile: + """`tile` with `offset` added to every value, NoData kept: a tile that + disagrees with its neighbours everywhere, by exactly `offset`.""" + array = np.asarray(tile.array).copy() + valid = ~np.isnan(array) + if tile.meta.nodata is not None: + valid &= array != tile.meta.nodata + array[valid] += np.asarray(offset, dtype=array.dtype) + return DemTile(meta=tile.meta, array=array) diff --git a/tests/python/test_cli_mesh_mosaic.py b/tests/python/test_cli_mesh_mosaic.py new file mode 100644 index 00000000..c39eccb4 --- /dev/null +++ b/tests/python/test_cli_mesh_mosaic.py @@ -0,0 +1,550 @@ +"""`rasputin mesh --dem DIR | --dem F... [--bbox ...]`: increment 15a through the CLI. + +`docs/increments/15-dem-mosaic.md` R6 and R11, and the parked design's C1-C6 +and T-real, carried. The oracle for equivalence is the single-file path, which +the existing `test_cli_mesh_dem.py` pins and this increment must not move +(I7): a grid cut into tiles must mesh exactly like the grid as one file. + +HOW THIS FILE GOES RED. It imports nothing new: until 15a lands, `--dem DIR` +is refused as "not a file" and `--bbox` is "No such option", so every test here +fails on its assertions, not at collection. + +Micro-TIFFs are written to `tmp_path`. Two real extracts of Ola's DTM10 +archive are read from `tests/fixtures/dtm10/` (see `extract.py` there), and +T-real cuts the committed benchmark tile into quadrants (`needs_codecs`). +""" + +from __future__ import annotations + +import io +import re +from pathlib import Path + +import numpy as np +import pytest +import tifffile +from typer.testing import CliRunner + +from geotiff_fixtures import ( + EPSG_UTM33, + GT_RASTER_TYPE, + KARTVERKET, + METRE, + PIXEL_IS_AREA, + PROJECTED_CS_TYPE, + VERTICAL_UNITS, + micro_tiff, + needs_codecs, + with_keys, +) +from mosaic_fixtures import piece, quadrants, whole +from test_cli_mesh import plain +from tin_engine.cli import app +from tin_engine.io.models import DemTile +from vtkread import VtkFile, read_vtk + +runner = CliRunner(env={"NO_COLOR": "1", "TERM": "dumb"}) +USAGE = 2 +DTM10 = Path(__file__).resolve().parents[1] / "fixtures" / "dtm10" + + +def invoke(*args: str) -> tuple[int, str]: + result = runner.invoke(app, ["mesh", *args]) + return result.exit_code, plain(result.output) + + +def terrain(rows: int, cols: int) -> np.ndarray: + """Not a plane, so `--tolerance` has work to do; exact in float32.""" + r, c = np.indices((rows, cols)) + return np.asarray(((7 * r + 3 * c) % 11) * 1.5 + 0.25 * r * c).astype(np.float32) + + +def tiff_of(tile: DemTile, compression: str | None = None) -> io.BytesIO: + """`tile` as a GeoTIFF with the registration its meta says.""" + m = tile.meta + half = 0.5 if m.pixel_is_area else 0.0 + keys = with_keys({GT_RASTER_TYPE: PIXEL_IS_AREA}) if m.pixel_is_area else None + return micro_tiff( + np.asarray(tile.array), + tiepoint=(0.0, 0.0, 0.0, m.x_min - half * m.delta_x, m.y_max + half * m.delta_y, 0.0), + scale=(m.delta_x, m.delta_y, 0.0), + geokeys=keys, + nodata=None if m.nodata is None else f"{m.nodata:g}", + compression=compression, + ) + + +def write_tiles(directory: Path, tiles: dict[str, DemTile]) -> list[Path]: + directory.mkdir(parents=True, exist_ok=True) + for name, tile in tiles.items(): + (directory / name).write_bytes(tiff_of(tile).getvalue()) + return sorted(directory.iterdir()) + + +def report(tmp_path: Path, *args: str) -> str: + """`--stats -`'s Markdown, read from stdout apart from stderr and unflattened.""" + result = runner.invoke(app, ["mesh", *args, "--out", str(tmp_path / "m.vtk"), "--stats", "-"]) + assert result.exit_code == 0, result.output + return result.stdout + + +def write_one(path: Path, tile: DemTile) -> Path: + path.parent.mkdir(parents=True, exist_ok=True) + path.write_bytes(tiff_of(tile).getvalue()) + return path + + +def run_vtk(out: Path, *args: str) -> VtkFile: + code, output = invoke(*args, "--out", str(out)) + assert code == 0, output + return read_vtk(out.read_bytes()) + + +def field(vtk: VtkFile, name: str) -> str: + (value,) = vtk.field_data[name].values + return str(value) + + +def same_mesh(a: VtkFile, b: VtkFile) -> None: + assert np.array_equal(a.points, b.points) + assert len(a.polygons) == len(b.polygons) + assert all(np.array_equal(p, q) for p, q in zip(a.polygons, b.polygons, strict=True)) + assert len(a.lines) == len(b.lines) + assert all(np.array_equal(p, q) for p, q in zip(a.lines, b.lines, strict=True)) + assert a.scalars.keys() == b.scalars.keys() + for name in a.scalars: + assert np.array_equal( + np.asarray(a.scalars[name].values), np.asarray(b.scalars[name].values) + ), name + + +@pytest.fixture +def source() -> DemTile: + return whole(9, 13, array=terrain(9, 13)) + + +@pytest.fixture +def mosaic_dir(tmp_path: Path, source: DemTile) -> Path: + write_tiles(tmp_path / "tiles", quadrants(source, row_cut=4, col_cut=6, overlap=1)) + return tmp_path / "tiles" + + +class TestC1Fields: + """C1 and R11: what a mosaic's `.vtk` records, and the stderr line.""" + + def test_a_directory_writes_the_mosaic_fields(self, tmp_path: Path, mosaic_dir: Path) -> None: + code, output = invoke("--dem", str(mosaic_dir), "--out", str(tmp_path / "m.vtk")) + assert code == 0, output + vtk = read_vtk((tmp_path / "m.vtk").read_bytes()) + assert field(vtk, "crs") == "EPSG:25833" + assert field(vtk, "elevation_source").startswith("mosaic of 4 tiles, 9 x 13 nodes; ") + assert field(vtk, "dem_tiles") == "ne.tif; nw.tif; se.tif; sw.tif" + assert "mosaic of 4 tiles, 9 x 13 nodes" in output + + def test_one_file_records_no_tile_list(self, tmp_path: Path, source: DemTile) -> None: + vtk = run_vtk(tmp_path / "one.vtk", "--dem", str(write_one(tmp_path / "one.tif", source))) + assert "dem_tiles" not in vtk.field_data + assert not field(vtk, "elevation_source").startswith("mosaic") + + +class TestC2Equivalence: + """C2 (T-equiv) and I5, I7: a tile split into four meshes exactly like the tile.""" + + @pytest.mark.parametrize( + ("area", "overlap", "shape"), + [(False, 1, (9, 13)), (True, 0, (8, 12)), (True, 3, (8, 12))], + ids=["point-shared-line", "area-abutting", "area-overlap-3"], + ) + @pytest.mark.parametrize("extra", [(), ("--tolerance", "0.5")], ids=["stride", "tolerance"]) + def test_split_meshes_like_whole( + self, + tmp_path: Path, + area: bool, + overlap: int, + shape: tuple[int, int], + extra: tuple[str, ...], + ) -> None: + grid = whole(*shape, array=terrain(*shape), area=area) + single = write_one(tmp_path / "whole.tif", grid) + tiles = write_tiles( + tmp_path / "split", quadrants(grid, row_cut=4, col_cut=6, overlap=overlap) + ) + assert len(tiles) == 4 + expected = run_vtk(tmp_path / "whole.vtk", "--dem", str(single), *extra) + got = run_vtk(tmp_path / "split.vtk", "--dem", str(tmp_path / "split"), *extra) + same_mesh(got, expected) + + +class TestC3RepeatedDem: + """C3: `--dem a --dem b ...` writes the same mesh as `--dem DIR`.""" + + def test_files_equal_directory(self, tmp_path: Path, mosaic_dir: Path) -> None: + files = [str(p) for p in sorted(mosaic_dir.glob("*.tif"), reverse=True)] + by_dir = run_vtk(tmp_path / "dir.vtk", "--dem", str(mosaic_dir)) + by_files = run_vtk(tmp_path / "files.vtk", *[arg for f in files for arg in ("--dem", f)]) + same_mesh(by_files, by_dir) + assert field(by_files, "dem_tiles") == field(by_dir, "dem_tiles") + + +class TestC4Refusals: + """C4: mosaic refusals are usage errors naming the files; nothing is written.""" + + def refused(self, tmp_path: Path, *args: str, says: tuple[str, ...]) -> None: + out = tmp_path / "x.vtk" + code, output = invoke(*args, "--out", str(out)) + assert code == USAGE, output + assert "No such option" not in output, output + assert "Traceback" not in output + for word in says: + assert word in output, f"{word!r} not in {output!r}" + assert not out.exists() + + def test_mixed_crs(self, tmp_path: Path, source: DemTile) -> None: + tiles = quadrants(source, row_cut=4, col_cut=6, overlap=1) + directory = tmp_path / "mixed" + write_tiles(directory, tiles) + ne = tiles["ne.tif"] + m = ne.meta + stream = micro_tiff( + np.asarray(ne.array), + tiepoint=(0.0, 0.0, 0.0, m.x_min, m.y_max, 0.0), + scale=(m.delta_x, m.delta_y, 0.0), + geokeys=with_keys({PROJECTED_CS_TYPE: 25832}), + ) + (directory / "ne.tif").write_bytes(stream.getvalue()) + self.refused(tmp_path, "--dem", str(directory), says=("ne.tif", "25832", str(EPSG_UTM33))) + + def test_half_a_cell_off(self, tmp_path: Path, source: DemTile) -> None: + tiles = quadrants(source, row_cut=4, col_cut=6, overlap=1) + tiles["se.tif"] = piece(source, 4, 9, 6, 13, x_min=tiles["se.tif"].meta.x_min + 5.0) + write_tiles(tmp_path / "shifted", tiles) + self.refused(tmp_path, "--dem", str(tmp_path / "shifted"), says=("se.tif", "0.5 cell")) + + def test_a_hole_in_the_tile_set(self, tmp_path: Path) -> None: + from mosaic_fixtures import blocks + + tiles = blocks(whole(6, 8, array=terrain(6, 8), area=True), 3, 4, skip={(1, 1)}) + write_tiles(tmp_path / "holed", tiles) + self.refused(tmp_path, "--dem", str(tmp_path / "holed"), says=("500040",)) + + def test_an_empty_directory_names_it(self, tmp_path: Path) -> None: + (tmp_path / "empty_dir").mkdir() + self.refused(tmp_path, "--dem", str(tmp_path / "empty_dir"), says=("empty_dir",)) + + +class TestQ1Seams: + """Ola's Q1 revised (2026-09-28), through the CLI: a disagreeing overlap + meshes, and each disagreeing pair is recorded in the `dem_seams` field + and, with `--stats`, in a "DEM seams" section. + + `dem_seams` is recorded whenever `dem_tiles` is: `none` when every + overlapping node agrees, else one entry per disagreeing pair, sorted, + `; `-joined, escaped like `dem_tiles`: + ` | : nodes , max , median `. + """ + + @staticmethod + def disagreeing(tmp_path: Path, source: DemTile, name: str = "nw.tif", by: float = 4.0) -> Path: + """The quadrants, `ne.tif` `by` higher at one node of the shared column + (global (2, 6), in `ne.tif` and the tile named `name` only). Was + `TestC4Refusals.test_an_overlap_disagreement`'s directory.""" + tiles = quadrants(source, row_cut=4, col_cut=6, overlap=1) + tiles[name] = tiles.pop("nw.tif") + changed = np.array(tiles["ne.tif"].array) + changed[2, 0] += np.float32(by) + tiles["ne.tif"] = piece(source, 0, 5, 6, 13, array=changed) + write_tiles(tmp_path / "disagree", tiles) + return tmp_path / "disagree" + + def test_an_overlap_disagreement_meshes_and_is_recorded( + self, tmp_path: Path, source: DemTile + ) -> None: + """Was `TestC4Refusals.test_an_overlap_disagreement`, which expected a + refusal naming both tiles and the 4. The same names and number are now + in the record, and the mesh is written.""" + vtk = run_vtk(tmp_path / "m.vtk", "--dem", str(self.disagreeing(tmp_path, source))) + assert field(vtk, "dem_seams") == "ne.tif | nw.tif: nodes 1, max 4, median 4" + assert field(vtk, "dem_tiles") == "ne.tif; nw.tif; se.tif; sw.tif" + + def test_agreeing_tiles_record_none(self, tmp_path: Path, mosaic_dir: Path) -> None: + vtk = run_vtk(tmp_path / "m.vtk", "--dem", str(mosaic_dir)) + assert field(vtk, "dem_seams") == "none" + + def test_one_file_records_no_seams(self, tmp_path: Path, source: DemTile) -> None: + vtk = run_vtk(tmp_path / "one.vtk", "--dem", str(write_one(tmp_path / "one.tif", source))) + assert "dem_seams" not in vtk.field_data + + def test_several_pairs_are_sorted_and_joined(self, tmp_path: Path) -> None: + """Each quadrant 1000 times its rank higher than the grid: every + overlap disagrees, by a constant. Six pairs, sorted by name; counts are + the overlaps' node counts (overlap 1 on a 9 x 13 grid: 5, 5, 7, 7, and + the corner node in the two diagonal pairs).""" + grid = whole(9, 13, array=terrain(9, 13)) + tiles = quadrants(grid, row_cut=4, col_cut=6, overlap=1) + for rank, name in enumerate(sorted(tiles), start=1): + t = tiles[name] + tiles[name] = DemTile(meta=t.meta, array=np.asarray(t.array) + 1000 * rank) + write_tiles(tmp_path / "all", tiles) + vtk = run_vtk(tmp_path / "m.vtk", "--dem", str(tmp_path / "all")) + assert field(vtk, "dem_seams") == "; ".join( + [ + "ne.tif | nw.tif: nodes 5, max 1000, median 1000", + "ne.tif | se.tif: nodes 7, max 2000, median 2000", + "ne.tif | sw.tif: nodes 1, max 3000, median 3000", + "nw.tif | se.tif: nodes 1, max 1000, median 1000", + "nw.tif | sw.tif: nodes 7, max 2000, median 2000", + "se.tif | sw.tif: nodes 5, max 1000, median 1000", + ] + ) + + def test_a_non_ascii_name_is_escaped(self, tmp_path: Path, source: DemTile) -> None: + directory = self.disagreeing(tmp_path, source, name="Ålesund.tif") + listed = next(p.name for p in directory.iterdir() if p.name.endswith("lesund.tif")) + escaped = listed.encode("ascii", "backslashreplace").decode("ascii") + vtk = run_vtk(tmp_path / "m.vtk", "--dem", str(directory)) + recorded = field(vtk, "dem_seams") + assert recorded.isascii() + first, second = ( + n.encode("ascii", "backslashreplace").decode("ascii") + for n in sorted([listed, "ne.tif"]) + ) # the file system may store the name decomposed, which sorts first + assert recorded == f"{first} | {second}: nodes 1, max 4, median 4" + assert escaped in recorded + + def test_stats_has_a_seams_section_listing_only_disagreeing_pairs( + self, tmp_path: Path, source: DemTile + ) -> None: + """After "Sizes", before "Quality": one table row per disagreeing pair.""" + lines = report(tmp_path, "--dem", str(self.disagreeing(tmp_path, source))).splitlines() + at = lines.index("## DEM seams") + assert lines.index("## Sizes") < at < lines.index("## Quality (plan view, x/y)") + rows = [] + for line in lines[at + 1 :]: + if line.startswith("## "): + break + if line.startswith("|"): + rows.append([cell.strip() for cell in line.strip("|").split("|")]) + assert rows == [ + ["tile", "tile", "nodes", "max", "median"], + ["---", "---", "---", "---", "---"], + ["ne.tif", "nw.tif", "1", "4", "4"], + ] + + def test_stats_has_no_seams_section_when_every_overlap_agrees( + self, tmp_path: Path, mosaic_dir: Path + ) -> None: + lines = report(tmp_path, "--dem", str(mosaic_dir)).splitlines() + assert "## Sizes" in lines + assert "## DEM seams" not in lines + + def test_a_sub_millimetre_disagreement_records_none_and_no_section( + self, tmp_path: Path, source: DemTile + ) -> None: + """Ola, 2026-09-28: "Ignore below 1mm". The one differing node is + 0.5 mm off, so no pair qualifies: `none`, and `--stats` has no + "DEM seams" section.""" + directory = self.disagreeing(tmp_path, source, by=0.0005) + vtk = run_vtk(tmp_path / "m.vtk", "--dem", str(directory)) + assert field(vtk, "dem_seams") == "none" + lines = report(tmp_path, "--dem", str(directory)).splitlines() + assert "## Sizes" in lines + assert "## DEM seams" not in lines + + def test_a_pipe_in_a_tile_name_is_escaped_in_the_stats_table( + self, tmp_path: Path, source: DemTile + ) -> None: + """Review suggestion on 15a: a `|` in a tile name would end its + Markdown table cell early. In the "DEM seams" table it is written + `\\|`, so the row still has five cells (split on unescaped pipes). + The `dem_seams` field is not a table and records the name as listed + (`ne.tif` sorts first: `e` < `|`).""" + directory = self.disagreeing(tmp_path, source, name="n|w.tif") + lines = report(tmp_path, "--dem", str(directory)).splitlines() + at = lines.index("## DEM seams") + row = next(line for line in lines[at + 1 :] if line.startswith("| ne.tif")) + cells = [cell.strip() for cell in re.split(r"(? None: + self.refused( + tmp_path, "--dem", str(mosaic_dir), "--dem", str(mosaic_dir / "nw.tif"), says=("--dem",) + ) + + def test_two_directories(self, tmp_path: Path, mosaic_dir: Path) -> None: + other = tmp_path / "other" + write_tiles(other, {"a.tif": whole(3, 4, array=terrain(3, 4))}) + self.refused(tmp_path, "--dem", str(mosaic_dir), "--dem", str(other), says=("--dem",)) + + def test_bbox_without_dem(self, tmp_path: Path) -> None: + self.refused(tmp_path, "catchment", "--bbox", "0", "0", "1", "1", says=("--bbox",)) + + @pytest.mark.parametrize( + "box", [("1", "0", "0", "1"), ("0", "0", "nan", "1")], ids=["inverted", "nan"] + ) # fmt: skip + def test_an_invalid_box(self, tmp_path: Path, mosaic_dir: Path, box: tuple[str, ...]) -> None: + self.refused(tmp_path, "--dem", str(mosaic_dir), "--bbox", *box, says=("--bbox",)) + + def test_bbox_restricts_the_mesh_to_the_snapped_window( + self, tmp_path: Path, mosaic_dir: Path + ) -> None: + # cols floor(1.2)=1 .. ceil(8.3)=9 (x 500010..500090); rows floor(0.8)=0 .. ceil(6.6)=7 + vtk = run_vtk( + tmp_path / "box.vtk", + "--dem", + str(mosaic_dir), + "--bbox", + "500012", + "6599967", + "500083", + "6599996", + ) + xs, ys = vtk.points[:, 0], vtk.points[:, 1] + assert (xs.min(), xs.max()) == (500010.0, 500090.0) + assert (ys.min(), ys.max()) == (6599965.0, 6600000.0) + assert field(vtk, "elevation_source").startswith("mosaic of 4 tiles, 8 x 9 nodes; ") + + def test_a_box_reaching_into_another_lattices_strip_meshes_on_the_covering_one( + self, tmp_path: Path, mosaic_dir: Path, source: DemTile + ) -> None: + """Ola's Q5 reading, end to end. `odd.tif` is half a cell east-west off + the quadrants and overlaps `se.tif`; the box selects both, but only the + main lattice covers it, so it meshes there and records `se.tif` alone.""" + odd = whole(5, 7, array=terrain(5, 7), x_min=500105.0, y_max=6599970.0) + write_tiles(mosaic_dir, {"odd.tif": odd}) + # cols floor(7.2)=7 .. ceil(11.2)=12; rows floor(5.2)=5 .. ceil(7.6)=8 + vtk = run_vtk( + tmp_path / "box.vtk", + "--dem", str(mosaic_dir), + "--bbox", "500072", "6599962", "500112", "6599974", + ) # fmt: skip + assert field(vtk, "dem_tiles") == "se.tif" + xs, ys = vtk.points[:, 0], vtk.points[:, 1] + assert (xs.min(), xs.max()) == (500070.0, 500120.0) + assert (ys.min(), ys.max()) == (6599960.0, 6599975.0) + + def test_bbox_on_one_file(self, tmp_path: Path, source: DemTile) -> None: + """Q2: `--bbox` applies to a single file too; it is then a window of it.""" + single = write_one(tmp_path / "one.tif", source) + vtk = run_vtk( + tmp_path / "box.vtk", + "--dem", + str(single), + "--bbox", + "500012", + "6599967", + "500083", + "6599996", + ) + assert (vtk.points[:, 0].min(), vtk.points[:, 0].max()) == (500010.0, 500090.0) + + +class TestC6NonAscii: + """C6: a non-ASCII tile name is recorded escaped, and the file is written.""" + + def test_the_name_is_backslash_escaped(self, tmp_path: Path, source: DemTile) -> None: + tiles = quadrants(source, row_cut=4, col_cut=6, overlap=1) + tiles["Ålesund.tif"] = tiles.pop("nw.tif") + write_tiles(tmp_path / "names", tiles) + listed = next( + p.name for p in (tmp_path / "names").iterdir() if p.name.endswith("lesund.tif") + ) + vtk = run_vtk(tmp_path / "n.vtk", "--dem", str(tmp_path / "names")) + recorded = field(vtk, "dem_tiles") + assert recorded.isascii() + assert listed.encode("ascii", "backslashreplace").decode("ascii") in recorded + + +class TestRealDtm10: + """The 15a acceptance on real extracts of one release (N5): Q1 and Q5.""" + + def test_the_real_seam_meshes(self, tmp_path: Path) -> None: + vtk = run_vtk(tmp_path / "seam.vtk", "--dem", str(DTM10 / "seam"), "--tolerance", "1") + assert field(vtk, "dem_tiles") == "6400_1_10m_z33.tif; 6400_4_10m_z33.tif" + assert field(vtk, "elevation_source").startswith("mosaic of 2 tiles, 256 x 563 nodes; ") + assert field(vtk, "dem_seams") == "none" # Ola's Q1 revised: it agrees + assert np.isfinite(vtk.points).all() + + def test_the_agreeing_real_seam_meshes_like_its_stitched_tile(self, tmp_path: Path) -> None: + """Ola's Q1 revised changes nothing where the overlap agrees: the seam + meshes exactly like one file holding west's 307 columns and east's + last 256, with west's georeferencing (the mesh before the change).""" + west_path = DTM10 / "seam" / "6400_4_10m_z33.tif" + east_path = DTM10 / "seam" / "6400_1_10m_z33.tif" + with tifffile.TiffFile(west_path) as tif: + page = tif.pages.first + tie, scale, west = page.tags[33922].value, page.tags[33550].value, page.asarray() + east = tifffile.imread(east_path) + assert np.array_equal(west[:, -51:], east[:, :51]) + stitched = micro_tiff( + np.ascontiguousarray(np.hstack([west, east[:, 51:]])), + tiepoint=(0.0, 0.0, 0.0, tie[3], tie[4], 0.0), + scale=tuple(scale), + geokeys=with_keys({GT_RASTER_TYPE: PIXEL_IS_AREA}), + nodata="-32767", + compression="deflate", + ) + single = tmp_path / "stitched.tif" + single.write_bytes(stitched.getvalue()) + expected = run_vtk(tmp_path / "one.vtk", "--dem", str(single), "--tolerance", "1") + got = run_vtk(tmp_path / "seam.vtk", "--dem", str(DTM10 / "seam"), "--tolerance", "1") + same_mesh(got, expected) + + def test_the_two_lattices_are_refused_naming_both(self, tmp_path: Path) -> None: + out = tmp_path / "x.vtk" + code, output = invoke("--dem", str(DTM10 / "lattices"), "--out", str(out)) + assert code == USAGE, output + for token in ("7707_1_10m_z33.tif", "7707_2_10m_z33.tif", "0.5 cell"): + assert token in output, f"{token!r} not in {output!r}" + assert not out.exists() + + def test_a_box_inside_one_lattice_meshes(self, tmp_path: Path) -> None: + vtk = run_vtk( + tmp_path / "box.vtk", + "--dem", str(DTM10 / "lattices"), + "--bbox", "769800", "7749350", "770000", "7749700", + "--tolerance", "1", + ) # fmt: skip + # One tile selected out of a directory: the file used is on record + # (the label is the directory's), but it is not called a mosaic. + assert field(vtk, "dem_tiles") == "7707_2_10m_z33.tif" + assert not field(vtk, "elevation_source").startswith("mosaic") + + +@needs_codecs +def test_t_real_the_benchmark_tile_in_quadrants_meshes_like_the_tile(tmp_path: Path) -> None: + """T-real: the parked design's, stated honestly. It proves the stitch on + real data, real NoData padding and real size. It does not prove that two + real neighbours agree; `TestRealDtm10` and `test_dem_input.py` do that.""" + with tifffile.TiffFile(KARTVERKET) as tif: + page = tif.pages.first + tie, scale, array = page.tags[33922].value, page.tags[33550].value, page.asarray() + rows, cols = array.shape + row_cut, col_cut = rows // 2, cols // 2 + directory = tmp_path / "quadrants" + directory.mkdir() + for name, (r0, r1, c0, c1) in { + "nw.tif": (0, row_cut + 1, 0, col_cut + 1), + "ne.tif": (0, row_cut + 1, col_cut, cols), + "sw.tif": (row_cut, rows, 0, col_cut + 1), + "se.tif": (row_cut, rows, col_cut, cols), + }.items(): + stream = micro_tiff( + np.ascontiguousarray(array[r0:r1, c0:c1]), + tiepoint=(0.0, 0.0, 0.0, tie[3] + c0 * scale[0], tie[4] - r0 * scale[1], 0.0), + scale=tuple(scale), + geokeys=with_keys({GT_RASTER_TYPE: PIXEL_IS_AREA, VERTICAL_UNITS: METRE}), + nodata="-32767", + compression="deflate", + ) + (directory / name).write_bytes(stream.getvalue()) + expected = run_vtk(tmp_path / "tile.vtk", "--dem", str(KARTVERKET), "--tolerance", "1") + got = run_vtk(tmp_path / "split.vtk", "--dem", str(directory), "--tolerance", "1") + same_mesh(got, expected) diff --git a/tests/python/test_dem_input.py b/tests/python/test_dem_input.py new file mode 100644 index 00000000..295ae607 --- /dev/null +++ b/tests/python/test_dem_input.py @@ -0,0 +1,307 @@ +"""`tin_engine.dem_input`: `DemRequest` in, `open_dem` out (increment 15a, R1, R11). + +The declarative front of `docs/increments/15-dem-mosaic.md`: the CLI parses +flags into a `DemRequest` and calls `open_dem`, and a GUI or API worker builds +the same request without Typer. This file tests it through files, synthetic +and real: + +- micro-TIFF mosaics written to `tmp_path`; +- two real extracts of Ola's DTM10 archive (2022-09-24 release), cut by + `tests/fixtures/dtm10/extract.py`: `seam/` (6400_4 | 6400_1, a 51-column + overlap that agrees bit for bit, N3) and `lattices/` (7707_1, half a cell + east-west off the main lattice, over its aligned neighbour 7707_2, N2). + +HOW THIS FILE GOES RED: `tin_engine.dem_input` and `tin_engine.mosaic` are +imported in fixtures; each test fails on its own with `ModuleNotFoundError`. +""" + +from __future__ import annotations + +import importlib +import io +from pathlib import Path +from types import ModuleType +from typing import Any + +import numpy as np +import pytest +import tifffile + +from geotiff_fixtures import ( + GT_RASTER_TYPE, + PIXEL_IS_AREA, + micro_tiff, + truncated_strip, + with_keys, +) +from mosaic_fixtures import ( + DX, + X0, + Y0, + deepest_interior, + quadrants, + same_array, + seams, + seams_of, + whole, +) +from tin_engine.io.geotiff import decode_dem +from tin_engine.io.models import DemTile + +DTM10 = Path(__file__).resolve().parents[1] / "fixtures" / "dtm10" +SEAM = DTM10 / "seam" +LATTICES = DTM10 / "lattices" +WEST, EAST = SEAM / "6400_4_10m_z33.tif", SEAM / "6400_1_10m_z33.tif" +ODD, MAIN = LATTICES / "7707_1_10m_z33.tif", LATTICES / "7707_2_10m_z33.tif" + + +@pytest.fixture(scope="module") +def di() -> ModuleType: + return importlib.import_module("tin_engine.dem_input") + + +@pytest.fixture(scope="module") +def mz() -> ModuleType: + return importlib.import_module("tin_engine.mosaic") + + +def tiff_of(tile: DemTile, **kwargs: Any) -> io.BytesIO: + """A point-registered micro-TIFF of `tile`: tie point at its first node.""" + m = tile.meta + return micro_tiff( + np.asarray(tile.array), + tiepoint=(0.0, 0.0, 0.0, m.x_min, m.y_max, 0.0), + scale=(m.delta_x, m.delta_y, 0.0), + **kwargs, + ) + + +def write_tiles(directory: Path, tiles: dict[str, DemTile]) -> list[Path]: + directory.mkdir(parents=True, exist_ok=True) + paths = [] + for name, tile in tiles.items(): + path = directory / name + path.write_bytes(tiff_of(tile).getvalue()) + paths.append(path) + return paths + + +def decoded(path: Path) -> DemTile: + with path.open("rb") as stream: + return decode_dem(stream) + + +def request( + di: ModuleType, + *sources: Path, + box: tuple[float, ...] | None = None, + nodata: float | None = None, +) -> Any: + bounds = None + if box is not None: + mz = importlib.import_module("tin_engine.mosaic") + bounds = mz.Bounds(x_min=box[0], y_min=box[1], x_max=box[2], y_max=box[3]) + return di.DemRequest(sources=tuple(sources), bounds=bounds, nodata=nodata) + + +@pytest.fixture +def source() -> DemTile: + return whole() # 9 x 13, point-registered + + +@pytest.fixture +def mosaic_dir(tmp_path: Path, source: DemTile) -> Path: + write_tiles(tmp_path / "tiles", quadrants(source, row_cut=4, col_cut=6, overlap=1)) + return tmp_path / "tiles" + + +class TestOpenDem: + def test_one_file_is_the_file(self, di: ModuleType, tmp_path: Path) -> None: + path = tmp_path / "one.tif" + path.write_bytes(micro_tiff().getvalue()) + opened = di.open_dem(request(di, path)) + expected = decoded(path) + assert opened.tile.meta == expected.meta + assert same_array(opened.tile.array, expected.array) + assert [p.name for p in opened.plan.tiles] == ["one.tif"] + assert opened.label == "one" + + def test_a_directory_is_stitched( + self, di: ModuleType, mosaic_dir: Path, source: DemTile + ) -> None: + opened = di.open_dem(request(di, mosaic_dir)) + assert opened.tile.meta == source.meta + assert same_array(opened.tile.array, source.array) + assert [p.name for p in opened.plan.tiles] == ["ne.tif", "nw.tif", "se.tif", "sw.tif"] + assert opened.label == "tiles" + + def test_explicit_files_equal_the_directory(self, di: ModuleType, mosaic_dir: Path) -> None: + files = sorted(mosaic_dir.glob("*.tif"), reverse=True) + by_files = di.open_dem(request(di, *files)) + by_dir = di.open_dem(request(di, mosaic_dir)) + assert by_files.tile.meta == by_dir.tile.meta + assert same_array(by_files.tile.array, by_dir.tile.array) + assert by_files.label == files[0].stem + + def test_bounds_cut_the_window(self, di: ModuleType, mosaic_dir: Path, source: DemTile) -> None: + opened = di.open_dem( + request(di, mosaic_dir, box=(500012.0, 6599987.0, 500033.0, 6599996.0)) + ) + assert (opened.tile.meta.x_min, opened.tile.meta.y_max) == (X0 + DX, Y0) + assert same_array(opened.tile.array, source.array[0:4, 1:5]) + + def test_a_caller_nodata_applies_to_every_tile(self, di: ModuleType, mosaic_dir: Path) -> None: + opened = di.open_dem(request(di, mosaic_dir, nodata=-9999.0)) + assert (opened.tile.meta.nodata, opened.tile.meta.nodata_source) == (-9999.0, "caller") + + def test_the_cap_refuses_before_any_pixel_is_read( + self, di: ModuleType, mz: ModuleType, tmp_path: Path, monkeypatch: pytest.MonkeyPatch + ) -> None: + """M5 through a file: its strip is cut, so a load would raise + `GeoTiffError`; the refusal must be the cap's `MosaicError` instead.""" + path = tmp_path / "cut.tif" + path.write_bytes(truncated_strip().getvalue()) + monkeypatch.setattr(mz, "physical_memory", lambda: 8) # cap 4 bytes: 3 x 4 nodes exceed it + with pytest.raises(mz.MosaicError, match="--bbox"): + di.open_dem(request(di, path)) + + +class TestDemRequest: + """R11: exactly one directory, or one or more files.""" + + def test_a_directory_with_files_is_refused(self, di: ModuleType, mosaic_dir: Path) -> None: + with pytest.raises(ValueError, match="director"): + di.open_dem(request(di, mosaic_dir, mosaic_dir / "nw.tif")) + + def test_two_directories_are_refused( + self, di: ModuleType, mosaic_dir: Path, tmp_path: Path + ) -> None: + other = tmp_path / "other" + other.mkdir() + with pytest.raises(ValueError, match="director"): + di.open_dem(request(di, mosaic_dir, other)) + + def test_no_source_is_refused(self, di: ModuleType) -> None: + with pytest.raises(ValueError): + di.open_dem(request(di)) + + def test_it_is_frozen(self, di: ModuleType, mosaic_dir: Path) -> None: + req = request(di, mosaic_dir) + with pytest.raises(ValueError): + req.nodata = 1.0 + + +def _shifted_copy(path: Path, target: Path, metres: float) -> Path: + """`path` rewritten with its tie point `metres` further east; same values.""" + with tifffile.TiffFile(path) as tif: + page = tif.pages.first + tie, scale, array = page.tags[33922].value, page.tags[33550].value, page.asarray() + target.parent.mkdir(parents=True, exist_ok=True) + stream = micro_tiff( + array, + tiepoint=(0.0, 0.0, 0.0, tie[3] + metres, tie[4], 0.0), + scale=tuple(scale), + geokeys=with_keys({GT_RASTER_TYPE: PIXEL_IS_AREA}), + nodata="-32767", + compression="deflate", + ) + target.write_bytes(stream.getvalue()) + return target + + +class TestRealDtm10: + """Q1 and Q5 on real data, one release (N5).""" + + def test_the_extracts_are_where_the_design_says(self) -> None: + west, east = decoded(WEST), decoded(EAST) + assert (west.meta.epsg, west.meta.pixel_is_area, west.meta.nodata) == ( + 25833, + True, + -32767.0, + ) + assert east.meta.x_min == west.meta.x_min + (307 - 51) * 10.0 # a 51-column overlap + assert np.array_equal(west.array[:, -51:], east.array[:, :51]) + odd, main = decoded(ODD), decoded(MAIN) + assert (odd.meta.x_min - main.meta.x_min) / 10.0 == -0.5 # N2: 5 m east-west + + def test_q1_the_agreeing_seam_is_accepted(self, di: ModuleType) -> None: + west, east = decoded(WEST), decoded(EAST) + opened = di.open_dem(request(di, SEAM)) + assert (opened.tile.meta.rows, opened.tile.meta.cols) == (256, 307 + 307 - 51) + assert opened.tile.meta.x_min == west.meta.x_min + assert same_array(opened.tile.array[:, :307], west.array) + assert same_array(opened.tile.array[:, 256:], east.array) + assert opened.seams == () # Ola's Q1 revised: an agreeing seam is not reported + + def test_q1_the_seam_shifted_by_one_cell_is_reported_and_split( + self, di: ModuleType, tmp_path: Path + ) -> None: + """The control (N3), under Ola's Q1 revised: was + `test_q1_the_seam_shifted_by_one_cell_is_refused`. One column off, the + real overlap (now 50 columns) disagrees. It is no longer refused: the + seam is reported, naming the two tiles, and each node takes the value + of the tile it lies deepest in. Both are checked against the + node-by-node oracle in `mosaic_fixtures`, and the split is checked by + hand on the middle row, where column depth decides.""" + (tmp_path / "shifted").mkdir() + (tmp_path / "shifted" / WEST.name).write_bytes(WEST.read_bytes()) + _shifted_copy(EAST, tmp_path / "shifted" / EAST.name, 10.0) + opened = di.open_dem(request(di, tmp_path / "shifted")) + grid = opened.tile.meta + assert (grid.rows, grid.cols) == (256, 307 + 307 - 50) + tiles = {WEST.name: decoded(WEST), EAST.name: decoded(tmp_path / "shifted" / EAST.name)} + expected = seams_of(tiles, grid) + assert [s[:2] for s in expected] == [(EAST.name, WEST.name)] # 6400_1 sorts first + assert expected[0][2] > 0 + assert seams(opened) == expected + assert same_array( + opened.tile.array, deepest_interior(tiles, grid).astype(opened.tile.array.dtype) + ) + # Row 128: west's columns 257-306 are 49..0 deep, east's 0..49. The 25 + # nearer the west tile's interior are west's, the other 25 east's. + west, east = tiles[WEST.name].array, tiles[EAST.name].array + row = opened.tile.array[128] + assert same_array(row[257:282], west[128, 257:282]) + assert same_array(row[282:307], east[128, 25:50]) + + def test_q5_the_two_lattices_together_are_refused_naming_both( + self, di: ModuleType, mz: ModuleType + ) -> None: + with pytest.raises(mz.MosaicError) as info: + di.open_dem(request(di, LATTICES)) + message = str(info.value) + for token in (ODD.name, MAIN.name, "0.5 cell", "east-west"): + assert token in message, f"{token!r} not in {message!r}" + + @pytest.mark.parametrize("path", [ODD, MAIN], ids=["7707_1", "7707_2"]) + def test_q5_each_lattice_alone_opens(self, di: ModuleType, path: Path) -> None: + opened = di.open_dem(request(di, path)) + assert opened.tile.meta == decoded(path).meta + + def test_q5_reading_a_box_on_7707_2_reaching_into_7707_1_opens_on_7707_2( + self, di: ModuleType + ) -> None: + """Ola's Q5 reading on real extracts. 7707_1 (y 7750690..7749740) lies + over 7707_2 (y 7750250..7749300); this box reaches 60 m into their + overlap strip, so it selects both, and only 7707_2 covers every node.""" + box = (769800.0, 7749350.0, 770000.0, 7749800.0) + opened = di.open_dem(request(di, LATTICES, box=box)) + assert [p.name for p in opened.plan.tiles] == [MAIN.name] + main = decoded(MAIN) + assert opened.tile.meta.x_min == 769800.0 + assert opened.tile.meta.y_max == 7749800.0 + # rows (7750250 - 7749800) / 10 = 45 .. 90, columns 5 .. 25 of 7707_2 + assert same_array(opened.tile.array, main.array[45:91, 5:26]) + + def test_q5_reading_a_box_inside_the_strip_takes_the_tie_by_name(self, di: ModuleType) -> None: + """Both lattices cover a box wholly inside the strip and hold one tile + each here, so the tie goes to the first tile's name: 7707_1.""" + box = (769800.0, 7749800.0, 770000.0, 7750200.0) + opened = di.open_dem(request(di, LATTICES, box=box)) + assert [p.name for p in opened.plan.tiles] == [ODD.name] + assert opened.tile.meta.x_min % 10.0 == 5.0 # on 7707_1's lattice + + def test_q5_a_request_inside_one_lattice_opens_with_both_listed(self, di: ModuleType) -> None: + """South of 7707_1's last row (y 7749740), only 7707_2 is selected.""" + opened = di.open_dem(request(di, LATTICES, box=(769800.0, 7749350.0, 770000.0, 7749700.0))) + assert [p.name for p in opened.plan.tiles] == [MAIN.name] diff --git a/tests/python/test_io_read_meta.py b/tests/python/test_io_read_meta.py new file mode 100644 index 00000000..d3042ab4 --- /dev/null +++ b/tests/python/test_io_read_meta.py @@ -0,0 +1,120 @@ +"""`tin_engine.io.geotiff.read_meta`: the header phase alone (increment 15a, R3). + +G1 and G2 of the parked design, carried by `docs/increments/15-dem-mosaic.md`: +`read_meta` is `decode_dem` stopped before the pixels. It must agree with +`decode_dem(...).meta` on every valid file, refuse every header defect with +the same message, and **not** read the pixels, so a file whose strip is +damaged still gives its header. + +HOW THIS FILE GOES RED: `read_meta` is looked up in a fixture, so each test +fails on its own with `AttributeError` while it is missing. +""" + +from __future__ import annotations + +import io +from collections.abc import Callable +from typing import Any + +import numpy as np +import pytest + +from geotiff_fixtures import ( + GT_RASTER_TYPE, + HAS_CODECS, + KARTVERKET, + METRE, + PIXEL_IS_AREA, + REFUSALS, + VERTICAL_UNITS, + Refusal, + corrupt_deflate, + elevations, + micro_tiff, + needs_codecs, + truncated_strip, + with_keys, +) +from tin_engine.io import geotiff +from tin_engine.io.models import GeoTiffError + +VALID: dict[str, Callable[[], io.BytesIO]] = { + "baseline": micro_tiff, + "area-registered": lambda: micro_tiff(geokeys=with_keys({GT_RASTER_TYPE: PIXEL_IS_AREA})), + "nodata-tag": lambda: micro_tiff(nodata="-32767"), + "vertical-metres": lambda: micro_tiff(geokeys=with_keys({VERTICAL_UNITS: METRE})), + "int16-promoted": lambda: micro_tiff(elevations(np.int16)), + "float64": lambda: micro_tiff(elevations(np.float64)), + "deflate": lambda: micro_tiff(compression="deflate"), + "larger": lambda: micro_tiff(elevations(rows=7, cols=9)), +} + + +@pytest.fixture +def read_meta() -> Callable[..., Any]: + return geotiff.read_meta # type: ignore[attr-defined, no-any-return] + + +@pytest.mark.parametrize("build", VALID.values(), ids=VALID.keys()) +def test_g1_read_meta_equals_the_decoded_meta( + read_meta: Callable[..., Any], build: Callable[[], io.BytesIO] +) -> None: + assert read_meta(build()) == geotiff.decode_dem(build()).meta + + +def test_g1_with_a_caller_nodata(read_meta: Callable[..., Any]) -> None: + assert ( + read_meta(micro_tiff(), nodata=-9999.0) + == geotiff.decode_dem(micro_tiff(), nodata=-9999.0).meta + ) + + +@needs_codecs +def test_g1_on_the_real_tile(read_meta: Callable[..., Any]) -> None: + with KARTVERKET.open("rb") as stream: + meta = read_meta(stream) + with KARTVERKET.open("rb") as stream: + assert meta == geotiff.decode_dem(stream).meta + + +def _header_refusals() -> list[Any]: + # With the codecs extra present, the LZW-relabelled file is not refused by + # its header: it fails later, decoding pixels, which read_meta never does. + return [ + pytest.param( + r, id=r.name, marks=pytest.mark.skipif(HAS_CODECS, reason="LZW decodes with the extra") + ) + if r.name == "refuses_missing_codec" + else pytest.param(r, id=r.name) + for r in REFUSALS + ] + + +@pytest.mark.parametrize("refusal", _header_refusals()) +def test_g2_every_header_refusal_fires_with_the_same_message( + read_meta: Callable[..., Any], refusal: Refusal +) -> None: + with pytest.raises(GeoTiffError) as from_decode: + geotiff.decode_dem(refusal.build(), **refusal.decode_kwargs) + with pytest.raises(GeoTiffError) as from_header: + read_meta(refusal.build(), **refusal.decode_kwargs) + assert str(from_header.value) == str(from_decode.value) + + +def test_a_bool_nodata_is_a_type_error(read_meta: Callable[..., Any]) -> None: + with pytest.raises(TypeError): + read_meta(micro_tiff(), nodata=True) + + +@pytest.mark.parametrize( + "build", [truncated_strip, corrupt_deflate], ids=["truncated-strip", "corrupt-deflate"] +) +def test_the_pixels_are_not_read( + read_meta: Callable[..., Any], build: Callable[[], io.BytesIO] +) -> None: + """The fixture's header is the baseline's; only its strip is damaged.""" + with pytest.raises(GeoTiffError, match="pixel data"): + geotiff.decode_dem(build()) + meta = read_meta(build()) + assert (meta.rows, meta.cols) == (3, 4) + assert meta == geotiff.decode_dem(micro_tiff(compression="deflate")).meta diff --git a/tests/python/test_io_repository.py b/tests/python/test_io_repository.py new file mode 100644 index 00000000..fde14aa5 --- /dev/null +++ b/tests/python/test_io_repository.py @@ -0,0 +1,357 @@ +"""`tin_engine.io.repository`: the directory-backed DEM repository (increment 15a). + +`docs/increments/15-dem-mosaic.md` R1-R3 and Ola's Q3 ruling: the repository +lives in `io/repository.py`, **the one module in `io/` that opens files**, +read-only. Tests F1-F4 are the parked design's, carried; the Q3 guard and +the laziness tests are new. Files are micro-TIFFs under `tmp_path`. + +HOW THIS FILE GOES RED: the module is imported in a fixture, so each test +fails on its own with `ModuleNotFoundError` and collection is unaffected. The +Q3 guard's own scanner is tested on planted source, so it is shown able to +fail before it is trusted on the package. +""" + +from __future__ import annotations + +import ast +import importlib +import io +from pathlib import Path +from types import ModuleType +from typing import Any + +import numpy as np +import pytest + +from geotiff_fixtures import ( + DELTA_X, + TIE_X, + TIE_Y, + elevations, + micro_tiff, + truncated_strip, +) +from tin_engine.io.geotiff import decode_dem +from tin_engine.io.models import GeoTiffError + +IO = Path(__file__).resolve().parents[2] / "src_python" / "tin_engine" / "io" + + +@pytest.fixture(scope="module") +def repo() -> ModuleType: + return importlib.import_module("tin_engine.io.repository") + + +def write(path: Path, stream: io.BytesIO | None = None) -> Path: + path.parent.mkdir(parents=True, exist_ok=True) + path.write_bytes((stream or micro_tiff()).getvalue()) + return path + + +def shifted(columns: int) -> io.BytesIO: + """The baseline micro-TIFF moved `columns` nodes east, so names map to places.""" + return micro_tiff(tiepoint=(0.0, 0.0, 0.0, TIE_X + columns * DELTA_X, TIE_Y, 0.0)) + + +def decoded(path: Path) -> object: + with path.open("rb") as stream: + return decode_dem(stream) + + +class TestF1Listing: + """F1: `from_directory` lists `*.tif` and `*.tiff`, any case, not recursively; sorted.""" + + def test_lists_tif_and_tiff_only_sorted(self, repo: ModuleType, tmp_path: Path) -> None: + write(tmp_path / "b.TIFF", shifted(3)) + write(tmp_path / "a.tif", shifted(0)) + write(tmp_path / "c.Tif", shifted(6)) + (tmp_path / "a.tfw").write_text("10\n0\n0\n-10\n0\n0\n") + (tmp_path / "a.tif.aux.xml").write_text("") + (tmp_path / "notes.txt").write_text("not a tile") + write(tmp_path / "sub" / "d.tif", shifted(9)) + names = [f.name for f in repo.TiffDemRepository.from_directory(tmp_path).footprints()] + assert names == ["a.tif", "b.TIFF", "c.Tif"] + + def test_a_footprint_is_the_files_name_and_header( + self, repo: ModuleType, tmp_path: Path + ) -> None: + path = write(tmp_path / "7908_3_10m_z33.tif", shifted(0)) + (footprint,) = repo.TiffDemRepository.from_directory(tmp_path).footprints() + assert footprint.name == "7908_3_10m_z33.tif" + assert footprint.meta == decoded(path).meta # type: ignore[attr-defined] + + def test_an_empty_directory_is_refused_naming_it( + self, repo: ModuleType, tmp_path: Path + ) -> None: + empty = tmp_path / "no_tiles_here" + empty.mkdir() + (empty / "readme.txt").write_text("x") + with pytest.raises(ValueError, match="no_tiles_here"): + repo.TiffDemRepository.from_directory(empty).footprints() + + def test_a_subdirectory_named_like_a_tile_is_not_a_tile( + self, repo: ModuleType, tmp_path: Path + ) -> None: + write(tmp_path / "a.tif") + (tmp_path / "trap.tif").mkdir() + names = [f.name for f in repo.TiffDemRepository.from_directory(tmp_path).footprints()] + assert names == ["a.tif"] + + +class TestF2HeaderOnly: + """F2 and I6: footprints read headers only, and read them once, lazily.""" + + def test_a_file_with_truncated_pixels_lists_and_refuses_on_load( + self, repo: ModuleType, tmp_path: Path + ) -> None: + """With the pixel read in the footprint path, listing would raise here.""" + write(tmp_path / "cut.tif", truncated_strip()) + repository = repo.TiffDemRepository.from_directory(tmp_path) + (footprint,) = repository.footprints() + assert (footprint.meta.rows, footprint.meta.cols) == (3, 4) + with pytest.raises(GeoTiffError, match="pixel data"): + repository.load("cut.tif") + + def test_construction_reads_nothing(self, repo: ModuleType, tmp_path: Path) -> None: + write(tmp_path / "junk.tif", io.BytesIO(b"not a tiff")) + repository = repo.TiffDemRepository.from_directory(tmp_path) # no raise yet + with pytest.raises(GeoTiffError): + repository.footprints() + + def test_footprints_are_cached(self, repo: ModuleType, tmp_path: Path) -> None: + path = write(tmp_path / "a.tif") + repository = repo.TiffDemRepository.from_directory(tmp_path) + first = repository.footprints() + path.unlink() # a second read of the header would now fail + assert repository.footprints() == first + + def test_load_decodes_the_whole_tile(self, repo: ModuleType, tmp_path: Path) -> None: + path = write(tmp_path / "a.tif", micro_tiff(elevations(rows=5, cols=7))) + tile = repo.TiffDemRepository.from_directory(tmp_path).load("a.tif") + expected = decoded(path) + assert tile.meta == expected.meta # type: ignore[attr-defined] + assert np.array_equal(tile.array, expected.array) # type: ignore[attr-defined] + + def test_load_of_an_unknown_name(self, repo: ModuleType, tmp_path: Path) -> None: + write(tmp_path / "a.tif") + with pytest.raises(KeyError): + repo.TiffDemRepository.from_directory(tmp_path).load("b.tif") + + def test_a_caller_nodata_reaches_both_phases(self, repo: ModuleType, tmp_path: Path) -> None: + """`DemRequest.nodata` (R1) is the caller asserting a sentinel for every tile.""" + write(tmp_path / "a.tif") + repository = repo.TiffDemRepository.from_directory(tmp_path, nodata=-9999.0) + (footprint,) = repository.footprints() + assert (footprint.meta.nodata, footprint.meta.nodata_source) == (-9999.0, "caller") + assert repository.load("a.tif").meta == footprint.meta + + +class TestS2DecodedDtype: + """S2: a footprint carries the dtype its tile decodes to, from the header + through `PROMOTION`, so the cap can count the canvas the plan will build.""" + + @pytest.mark.parametrize( + ("stored", "decoded_as"), + [ + (np.int16, np.float32), + (np.uint8, np.float32), + (np.float32, np.float32), + (np.int32, np.float64), + (np.uint32, np.float64), + (np.float64, np.float64), + ], + ids=["int16", "uint8", "float32", "int32", "uint32", "float64"], + ) + def test_the_footprint_dtype_is_the_decoded_one( + self, repo: ModuleType, tmp_path: Path, stored: Any, decoded_as: Any + ) -> None: + path = write(tmp_path / "a.tif", micro_tiff(elevations().astype(stored))) + repository = repo.TiffDemRepository.from_directory(tmp_path) + (footprint,) = repository.footprints() + assert np.dtype(footprint.dtype) == decoded_as + assert np.dtype(footprint.dtype) == decoded(path).array.dtype # type: ignore[attr-defined] + + +ARCHIVE = Path(__file__).resolve().parents[3] / "rasputin_data" / "DTM10_UTM33_20220924" +needs_archive = pytest.mark.skipif( + not ARCHIVE.is_dir(), reason=f"Ola's DTM10 archive is not at {ARCHIVE}" +) + + +@needs_archive +class TestB1RealArchive: + """Ola's Q5 reading on the design's own 15a acceptance box, headers only. + + The 2 x 2 block 7908_3, 7908_2, 7808_4, 7808_1 is on the main lattice, and + the half-cell tile 7807_2's 51-node overlap reaches into its south-west + corner. Until the reading, the box was refused naming 7807_2. CI has no + archive; the synthetic equivalent is `test_mosaic.py::TestB1LatticeByCoverage` + and the real-extract one `test_dem_input.py::TestRealDtm10`. + """ + + BOX = (799750.0, 7849750.0, 900250.0, 7950250.0) + BLOCK = ("7908_3", "7908_2", "7808_4", "7808_1") + + def test_the_acceptance_box_plans_on_the_main_lattice(self, repo: ModuleType) -> None: + mz = importlib.import_module("tin_engine.mosaic") + prints = repo.TiffDemRepository.from_directory(ARCHIVE).footprints() + x_min, y_min, x_max, y_max = self.BOX + box = mz.Bounds(x_min=x_min, y_min=y_min, x_max=x_max, y_max=y_max) + result = mz.plan_mosaic(prints, box, None) + assert result.reference == (-100250.0, 7950250.0) # the main lattice's (N2) + assert (result.meta.rows, result.meta.cols) == (10051, 10051) + assert (result.meta.x_min, result.meta.y_max) == (799750.0, 7950250.0) + names = {p.name for p in result.tiles} + assert {f"{b}_10m_z33.tif" for b in self.BLOCK} <= names + assert "7807_2_10m_z33.tif" not in names + # Dropping the other lattice's tiles is the same as their absence. + odd = { # N2: the half-cell tiles are 5 m east-west off + f.name for f in prints if (f.meta.x_min - result.reference[0]) / f.meta.delta_x % 1 != 0 + } + assert "7807_2_10m_z33.tif" in odd + main = [f for f in prints if f.name not in odd] + assert result == mz.plan_mosaic(main, box, None) + + +class TestF3Refusals: + """F3: a listed file that fails `read_meta` refuses the listing, naming the file.""" + + def test_a_tif_that_is_not_a_geotiff(self, repo: ModuleType, tmp_path: Path) -> None: + write(tmp_path / "a.tif") + write(tmp_path / "stray.tif", micro_tiff(geokeys={})) + with pytest.raises(GeoTiffError, match=r"stray\.tif"): + repo.TiffDemRepository.from_directory(tmp_path).footprints() + + +class TestF4ExplicitPaths: + """F4: explicit paths, resolved and deduplicated, list like the directory.""" + + def test_duplicates_collapse_and_the_listing_matches( + self, repo: ModuleType, tmp_path: Path, monkeypatch: pytest.MonkeyPatch + ) -> None: + a = write(tmp_path / "a.tif", shifted(0)) + b = write(tmp_path / "b.tif", shifted(3)) + monkeypatch.chdir(tmp_path) + explicit = repo.TiffDemRepository([b, a, Path("a.tif"), tmp_path / "." / "b.tif"]) + listed = repo.TiffDemRepository.from_directory(tmp_path) + assert explicit.footprints() == listed.footprints() + assert [f.name for f in explicit.footprints()] == ["a.tif", "b.tif"] + + def test_two_files_with_one_name_are_refused(self, repo: ModuleType, tmp_path: Path) -> None: + """A footprint's name is the file name, so it must be unique.""" + one = write(tmp_path / "x" / "t.tif") + two = write(tmp_path / "y" / "t.tif", shifted(3)) + with pytest.raises(ValueError, match=r"\bt\.tif"): + repo.TiffDemRepository([one, two]).footprints() + + def test_it_satisfies_the_protocol(self, repo: ModuleType, tmp_path: Path) -> None: + write(tmp_path / "a.tif") + repository: object = repo.TiffDemRepository.from_directory(tmp_path) + assert callable(getattr(repository, "footprints", None)) + assert callable(getattr(repository, "load", None)) + assert set(repo.DemRepository.__dict__) >= {"footprints", "load"} + + +# --------------------------------------------------------------------------- +# Q3: io/repository.py is the one module in io/ that opens files +# --------------------------------------------------------------------------- + +#: Calls that reach the filesystem. A call by attribute is matched by name +#: alone, so `path.open()` and `tifffile.imread()` are both found. +OPENERS = frozenset( + { + "open", + "read_bytes", + "read_text", + "write_bytes", + "write_text", + "imread", + "imwrite", + "memmap", + "fromfile", + } +) + + +def file_openers(source: str) -> list[str]: + """Every call in `source` that opens a file, as `name:line`.""" + found = [] + for node in ast.walk(ast.parse(source)): + if isinstance(node, ast.Call): + func = node.func + name = ( + func.id + if isinstance(func, ast.Name) + else func.attr + if isinstance(func, ast.Attribute) + else "" + ) + if name in OPENERS: + found.append(f"{name}:{node.lineno}") + return found + + +def open_modes(source: str) -> list[str]: + """The mode of every `open` call in `source` ("r" when none is given).""" + modes = [] + for node in ast.walk(ast.parse(source)): + if isinstance(node, ast.Call): + func = node.func + name = ( + func.id + if isinstance(func, ast.Name) + else func.attr + if isinstance(func, ast.Attribute) + else "" + ) + if name == "open": + positional = node.args[1:] if isinstance(func, ast.Name) else node.args + mode = ( + positional[0] + if positional + else next((k.value for k in node.keywords if k.arg == "mode"), None) + ) + modes.append( + mode.value if isinstance(mode, ast.Constant) else "r" if mode is None else "?" + ) + return modes + + +class TestQ3OneModuleOpensFiles: + @pytest.mark.parametrize( + "planted", + [ + "open(p)", + "with path.open('rb') as f: pass", + "Path(p).read_bytes()", + "tifffile.imread(p)", + "np.memmap(p)", + ], + ) + def test_the_scanner_finds_a_planted_opener(self, planted: str) -> None: + assert file_openers(planted), planted + + def test_the_scanner_ignores_streams_and_prose(self) -> None: + clean = '"""Nothing here opens a file."""\ntif = tifffile.TiffFile(stream)\nstream.read()\n' + assert file_openers(clean) == [] + + def test_no_other_io_module_opens_a_file(self) -> None: + offenders = { + path.name: found + for path in sorted(IO.glob("*.py")) + if path.name != "repository.py" + and (found := file_openers(path.read_text(encoding="utf-8"))) + } + assert offenders == {} + + def test_the_repository_opens_files_read_only(self) -> None: + source = (IO / "repository.py").read_text(encoding="utf-8") + assert file_openers(source), "the repository must be the module that opens the tiles" + modes = open_modes(source) + assert modes and all(mode == "rb" for mode in modes), modes + + def test_the_package_docstring_says_so(self) -> None: + """Q3 (a): `io/`'s rule narrowed, in the package's own docstring.""" + doc = ast.get_docstring(ast.parse((IO / "__init__.py").read_text(encoding="utf-8"))) or "" + assert "repository.py" in doc + assert "Nothing here opens a file." not in doc diff --git a/tests/python/test_mosaic.py b/tests/python/test_mosaic.py new file mode 100644 index 00000000..0597f9ef --- /dev/null +++ b/tests/python/test_mosaic.py @@ -0,0 +1,1634 @@ +"""`tin_engine.mosaic`: planning and assembly of a multi-tile DEM (increment 15a). + +The design is `docs/increments/15-dem-mosaic.md` (R1, R4, R5, R7; invariants +I1-I7); names not fixed there are pinned in its "Pinned by the red suite (15a)" +subsection. Test ids follow its "Tests for @tester" list, which carries the +parked design's M1-M12 (`git show f8cedbe:docs/increments/15-dem-repository.md`) +with 15a's changes: M2 is lattice grouping, M5 the cap from physical memory, +and M13-M15 are new. + +This is 15a's **invariant-critical** suite. Its tests were run against a +scratch implementation and against deliberate mutants of it (a wrong lattice +alignment check, an off-by-one in stitching offsets, an overlap check that +accepts a disagreement, a coverage check that misses a gap); the handback of +the red commit lists which test killed which. + +AMENDED FOR OLA'S Q1 REVISED (2026-09-28). Overlaps that disagree are no +longer refused: each node takes the value of the tile it lies deepest in, ties +by name, and each disagreeing pair is reported in `Mosaic.seams` +(`TestQ1DeepestInterior`, `TestQ1SeamReport`, and four tests of +`TestM8Overlaps` that pinned the refusal and now pin the report). + +AMENDED FOR OLA'S 1 MM THRESHOLD (2026-09-28, "Ignore below 1mm"). A seam +counts only nodes where |a - b| >= 0.001; the midline rule does not look at +the threshold (`TestSeamThreshold`; `TestM8Overlaps`' one-ulp test inverted). + +HOW THIS FILE GOES RED. `tin_engine.mosaic` and `tin_engine.io.repository` +are imported in module-scoped fixtures, as `test_bench.py` loads its tool, so +while they are missing each test fails on its own with `ModuleNotFoundError` +and the rest of `tests/python` still collects and runs. + +Tiles are built from `RasterMeta` and `DemTile` directly (`mosaic_fixtures`), +and the repository is a dict: nothing here touches a file. +""" + +from __future__ import annotations + +import importlib +import itertools +import math +import re +from pathlib import Path +from types import ModuleType +from typing import Any, ClassVar + +import numpy as np +import pytest +import shapely + +from mosaic_fixtures import ( + DX, + DY, + SENTINEL, + X0, + Y0, + Loads, + blocks, + deepest_interior, + footprints, + meta, + piece, + quadrants, + same_array, + seams, + seams_of, + shifted_by, + values, + whole, +) +from tin_engine.io.models import DemTile + +SRC = Path(__file__).resolve().parents[2] / "src_python" / "tin_engine" + + +@pytest.fixture(scope="module") +def mz() -> ModuleType: + return importlib.import_module("tin_engine.mosaic") + + +@pytest.fixture(scope="module") +def footprint() -> Any: + return importlib.import_module("tin_engine.io.repository").TileFootprint + + +@pytest.fixture +def plan(mz: ModuleType, footprint: Any) -> Any: + """`plan(tiles, box=None, needed=None)`: plan a dict of tiles; `box` is a 4-tuple.""" + + def run( + tiles: dict[str, DemTile], box: tuple[float, ...] | None = None, needed: Any = None + ) -> Any: + bounds = None if box is None else bounds_of(mz, box) + return mz.plan_mosaic(footprints(footprint, tiles), bounds, needed) + + return run + + +@pytest.fixture +def build(mz: ModuleType, plan: Any) -> Any: + """`build(tiles, box=None, needed=None) -> (Mosaic, Loads)`: plan, then assemble.""" + + def run( + tiles: dict[str, DemTile], box: tuple[float, ...] | None = None, needed: Any = None + ) -> Any: + loads = Loads(tiles) + return mz.assemble(plan(tiles, box, needed), loads), loads + + return run + + +@pytest.fixture +def refused(mz: ModuleType) -> Any: + """`refused(call, *tokens)`: `call()` raises `MosaicError` naming each token.""" + + def check(call: Any, *tokens: str) -> str: + with pytest.raises(mz.MosaicError) as info: + call() + message = str(info.value) + for token in tokens: + assert token.lower() in message.lower(), f"{token!r} not named in {message!r}" + return message + + return check + + +def bounds_of(mz: ModuleType, box: tuple[float, ...]) -> Any: + x_min, y_min, x_max, y_max = box + return mz.Bounds(x_min=x_min, y_min=y_min, x_max=x_max, y_max=y_max) + + +def window(mz: ModuleType, row0: int, col0: int, rows: int, cols: int) -> Any: + return mz.IndexWindow(row0=row0, col0=col0, rows=rows, cols=cols) + + +def names_number(message: str, number: int) -> None: + """`number` appears as a whole number, not inside another (`3` is not in `317`).""" + assert re.search(rf"(? tuple[int, int]: + """The global index of tile `name`'s first node, as the plan places it.""" + (p,) = [p for p in plan.tiles if p.name == name] + return ( + plan.window.row0 + p.canvas.row0 - p.source.row0, + plan.window.col0 + p.canvas.col0 - p.source.col0, + ) + + +# --------------------------------------------------------------------------- +# The module's surface +# --------------------------------------------------------------------------- + + +def test_mosaic_error_is_a_value_error(mz: ModuleType) -> None: + """R11: refusals become `typer.BadParameter` like `GeoTiffError`, one type.""" + assert issubclass(mz.MosaicError, ValueError) + + +def test_align_tolerance_is_a_millionth_of_a_cell(mz: ModuleType) -> None: + assert mz.ALIGN_TOLERANCE == 1e-6 + + +def test_an_empty_repository_is_refused(mz: ModuleType, refused: Any) -> None: + refused(lambda: mz.plan_mosaic([], None, None)) + + +class TestBounds: + """R6: `Bounds` is four finite floats with `x_min < x_max`, `y_min < y_max`.""" + + def test_a_valid_box(self, mz: ModuleType) -> None: + box = bounds_of(mz, (1.0, 2.0, 3.0, 4.0)) + assert (box.x_min, box.y_min, box.x_max, box.y_max) == (1.0, 2.0, 3.0, 4.0) + + @pytest.mark.parametrize( + "box", + [ + (math.nan, 0.0, 1.0, 1.0), + (0.0, 0.0, math.inf, 1.0), + (0.0, -math.inf, 1.0, 1.0), + (1.0, 0.0, 0.0, 1.0), # inverted in x + (0.0, 1.0, 1.0, 0.0), # inverted in y + (0.0, 0.0, 0.0, 1.0), # zero width + (0.0, 1.0, 1.0, 1.0), # zero height + ], + ids=["nan", "inf", "-inf", "inverted-x", "inverted-y", "zero-width", "zero-height"], + ) + def test_refused(self, mz: ModuleType, box: tuple[float, ...]) -> None: + with pytest.raises(ValueError): + bounds_of(mz, box) + + +# --------------------------------------------------------------------------- +# Planning (no pixels) +# --------------------------------------------------------------------------- + + +class TestM1Layout: + """M1: canvas shape, origin and each tile's two windows.""" + + def test_point_registered_2x2_sharing_a_row_and_a_column( + self, mz: ModuleType, plan: Any + ) -> None: + source = whole() # 9 x 13 + tiles = quadrants(source, row_cut=4, col_cut=6, overlap=1) + result = plan(tiles) + assert result.meta == source.meta + assert result.window == window(mz, 0, 0, 9, 13) + assert [p.name for p in result.tiles] == ["ne.tif", "nw.tif", "se.tif", "sw.tif"] + placed = {p.name: p for p in result.tiles} + expected = {"nw.tif": (0, 0), "ne.tif": (0, 6), "sw.tif": (4, 0), "se.tif": (4, 6)} + for name, (row0, col0) in expected.items(): + assert placed[name].canvas == window(mz, row0, col0, 5, 7), name + assert placed[name].source == window(mz, 0, 0, 5, 7), name + assert placed[name].meta == tiles[name].meta, name + + def test_area_registered_2x2_abutting(self, mz: ModuleType, plan: Any) -> None: + """Disjoint node sets one spacing apart: the canvas is contiguous and + nothing between them is a gap (the coverage check must not see one).""" + source = whole(rows=8, cols=12, area=True) + result = plan(quadrants(source, row_cut=4, col_cut=6, overlap=0)) + assert result.meta == source.meta + placed = {p.name: p.canvas for p in result.tiles} + assert placed == { + "nw.tif": window(mz, 0, 0, 4, 6), + "ne.tif": window(mz, 0, 6, 4, 6), + "sw.tif": window(mz, 4, 0, 4, 6), + "se.tif": window(mz, 4, 6, 4, 6), + } + + def test_one_tile_plans_as_itself(self, mz: ModuleType, plan: Any) -> None: + source = whole() + result = plan({"only.tif": source}) + assert result.meta == source.meta + assert result.reference == (X0, Y0) + assert result.window == window(mz, 0, 0, 9, 13) + + +class TestM2LatticeGrouping: + """M2, changed for 15a (R4 points 1 and 3, Q5): lattices, not one lattice. + + The main lattice is the 9 x 13 point grid cut in four. `odd.tif` lies + south-east of it, half a cell east-west off (N2's case), overlapping `se`. + """ + + @staticmethod + def repository(**odd_changes: Any) -> dict[str, DemTile]: + tiles = quadrants(whole(), row_cut=4, col_cut=6, overlap=1) + fields: dict[str, Any] = {"x_min": X0 + 10.5 * DX, "y_max": Y0 - 6 * DY} + fields.update(odd_changes) + tiles["odd.tif"] = whole(rows=5, cols=7, **fields) + return tiles + + def test_a_request_inside_the_main_lattice_plans(self, plan: Any) -> None: + result = plan(self.repository(), (500012.0, 6599987.0, 500033.0, 6599996.0)) + assert [p.name for p in result.tiles] == ["nw.tif"] + assert result.reference == (X0, Y0) + + def test_a_request_inside_the_odd_lattice_plans_on_that_lattice( + self, mz: ModuleType, plan: Any + ) -> None: + result = plan(self.repository(), (500130.0, 6599952.0, 500160.0, 6599958.0)) + assert [p.name for p in result.tiles] == ["odd.tif"] + assert result.reference == (X0 + 10.5 * DX, Y0 - 6 * DY) + assert result.meta.x_min == 500125.0 # odd lattice: 500105 + 2 * 10 + assert result.window == window(mz, 2, 2, 3, 5) + + def test_a_request_straddling_both_is_refused_naming_one_of_each( + self, plan: Any, refused: Any + ) -> None: + tiles = self.repository() + refused( + lambda: plan(tiles, (500100.0, 6599955.0, 500130.0, 6599965.0)), + "se.tif", "odd.tif", "0.5 cell", "east-west", + ) # fmt: skip + + def test_no_bounds_selects_both_lattices_and_is_refused(self, plan: Any, refused: Any) -> None: + """Unchanged by Ola's Q5 reading: without a box the request is every + tile, and neither lattice covers the other's nodes (odd.tif reaches + x 500165, past the main lattice's 500120), so none covers it (R6).""" + tiles = self.repository() + refused(lambda: plan(tiles), "odd.tif", "0.5 cell") + + def test_a_half_cell_north_south_is_named_so(self, plan: Any, refused: Any) -> None: + tiles = self.repository(x_min=X0 + 11 * DX, y_max=Y0 - 6.5 * DY) + refused( + lambda: plan(tiles, (500100.0, 6599955.0, 500130.0, 6599965.0)), + "odd.tif", + "0.5 cell", + "north-south", + ) + + def test_a_thousandth_of_a_cell_is_another_lattice(self, plan: Any, refused: Any) -> None: + tiles = self.repository(x_min=X0 + (11 + 1e-3) * DX) + refused( + lambda: plan(tiles, (500100.0, 6599955.0, 500130.0, 6599965.0)), + "se.tif", + "odd.tif", + "cell", + ) + + @pytest.mark.parametrize( + ("change", "tokens"), + [ + ({"epsg": 25832}, ("25833", "25832")), + ({"epsg": 32633}, ("25833", "32633")), + ({"dx": 12.5}, ("spacing", "12.5")), + ({"dy": 7.5}, ("spacing", "7.5")), + ({"area": True}, ("registration",)), + ({"nodata": -9999.0, "nodata_source": "tag"}, ("nodata", "-9999", "none")), + ], + ids=["crs-25832", "crs-32633", "spacing-dx", "spacing-dy-alone", "registration", "nodata"], + ) + def test_other_differences_are_named_as_such( + self, plan: Any, refused: Any, change: dict[str, Any], tokens: tuple[str, ...] + ) -> None: + """The parked `refuses_mixed_crs_mosaic`, `_spacing`, `_registration`, `_nodata`. + + `ne.tif` differs from `nw.tif` in exactly one field and is otherwise + where `quadrants` puts it, so only that field can be the reason. + """ + source = whole() + tiles = { + "nw.tif": piece(source, 0, 5, 0, 7), + "ne.tif": piece(source, 0, 5, 6, 13, **_meta_change(change)), + } + refused(lambda: plan(tiles), "nw.tif", "ne.tif", *tokens) + + def test_mixed_nodata_sentinels_are_refused_q4(self, plan: Any, refused: Any) -> None: + """Q4 (a): two sentinels, both named.""" + source = whole(nodata=SENTINEL) + tiles = { + "nw.tif": piece(source, 0, 5, 0, 7), + "ne.tif": piece(source, 0, 5, 6, 13, nodata=-9999.0), + } + refused(lambda: plan(tiles), "nw.tif", "ne.tif", "-32767", "-9999") + + def test_a_different_crs_outside_the_request_is_not_a_refusal(self, plan: Any) -> None: + """R4 changed the parked rule: only the *selected* tiles share a lattice.""" + tiles = quadrants(whole(), row_cut=4, col_cut=6, overlap=1) + tiles["far.tif"] = whole(rows=3, cols=3, x_min=X0 + 1000 * DX, epsg=25832) + result = plan(tiles, (500012.0, 6599987.0, 500033.0, 6599996.0)) + assert [p.name for p in result.tiles] == ["nw.tif"] + + +class TestB1LatticeByCoverage: + """Ola's reading of Q5 after review ("Ruled by Ola", 2026-09-27): per + lattice, whether **its own tiles cover every node the request needs**. + + Exactly one covers: plan on it, and the other lattices' tiles are not in + the plan. None covers: the Q5 refusal (the straddle tests in M2). Several + cover (a box wholly inside an overlap strip): the lattice with the most + tiles in the repository, ties by the name of its first tile. + + The repository is M2's: the 9 x 13 main lattice in four, and `odd.tif` + (x 500105..500165, y 6599970..6599950) half a cell east-west off it. Their + overlap strip is x 500105..500120, y 6599970..6599960. + + The odd tile is also named `a_odd.tif`, which sorts before the main + lattice's `ne.tif`: an implementation that takes whichever lattice comes + first, whatever it covers, fails one of the two names in each case. + """ + + MAIN_REACHING_INTO_THE_STRIP = (500072.0, 6599962.0, 500112.0, 6599974.0) + ODD_REACHING_INTO_THE_STRIP = (500106.0, 6599952.0, 500150.0, 6599969.0) + INSIDE_THE_STRIP = (500106.0, 6599961.0, 500114.0, 6599969.0) + ODD_ORIGIN = (X0 + 10.5 * DX, Y0 - 6 * DY) + + @staticmethod + def main() -> dict[str, DemTile]: + return quadrants(whole(), row_cut=4, col_cut=6, overlap=1) + + @staticmethod + def odd(name: str = "odd.tif", pieces: int = 1) -> dict[str, DemTile]: + """The odd lattice's 5 x 7 tile, whole, or in four named `-`.""" + x_min, y_max = TestB1LatticeByCoverage.ODD_ORIGIN + tile = whole(rows=5, cols=7, x_min=x_min, y_max=y_max) + if pieces == 1: + return {name: tile} + return { + f"{name}-{q}": t for q, t in quadrants(tile, row_cut=2, col_cut=3, overlap=1).items() + } + + @pytest.mark.parametrize("odd_name", ["odd.tif", "a_odd.tif"]) + def test_a_main_box_reaching_into_the_strip_plans_on_the_main_lattice( + self, mz: ModuleType, plan: Any, build: Any, odd_name: str + ) -> None: + """The reviewer's case: the box selects the odd tile, and today is + refused as mixed-lattice, but the main lattice covers every node.""" + main = self.main() + tiles = {**main, **self.odd(odd_name)} + result = plan(tiles, self.MAIN_REACHING_INTO_THE_STRIP) + assert [p.name for p in result.tiles] == ["se.tif"] + assert result.reference == (X0, Y0) + assert result.window == window(mz, 5, 7, 4, 6) + assert result == plan(main, self.MAIN_REACHING_INTO_THE_STRIP) + mosaic, loads = build(tiles, self.MAIN_REACHING_INTO_THE_STRIP) + assert loads.calls == ["se.tif"] + assert same_array(mosaic.tile.array, whole().array[5:9, 7:13]) + + @pytest.mark.parametrize("odd_name", ["odd.tif", "a_odd.tif"]) + def test_an_odd_box_reaching_into_the_strip_plans_on_the_odd_lattice( + self, mz: ModuleType, plan: Any, odd_name: str + ) -> None: + """The mirror case. The main lattice's tiles cover the part of the box + over them, but not x 500130..500150 or y 6599955..6599950, which the + odd tile holds: the box is not clamped to the main lattice's union.""" + odd = self.odd(odd_name) + result = plan({**self.main(), **odd}, self.ODD_REACHING_INTO_THE_STRIP) + assert [p.name for p in result.tiles] == [odd_name] + assert result.reference == self.ODD_ORIGIN + assert result.window == window(mz, 0, 0, 5, 6) + assert result == plan(odd, self.ODD_REACHING_INTO_THE_STRIP) + + def test_several_cover_the_most_tiles_in_the_repository_wins( + self, mz: ModuleType, plan: Any + ) -> None: + """Main 4 tiles, odd 1; each selects exactly one tile for this box, so + a count of *selected* tiles would tie.""" + main = self.main() + result = plan({**main, **self.odd()}, self.INSIDE_THE_STRIP) + assert [p.name for p in result.tiles] == ["se.tif"] + assert result.reference == (X0, Y0) + assert result == plan(main, self.INSIDE_THE_STRIP) + + def test_the_count_is_of_the_repository_not_of_the_selection( + self, footprint: Any, mz: ModuleType + ) -> None: + """Main 1 tile (se alone), odd 2 (one far away, never selected); both + select one. The odd names sort after `se.tif`, so a count of the + selection, tied and broken by name, would choose the main lattice.""" + x_min, y_max = self.ODD_ORIGIN + tiles = { + "se.tif": self.main()["se.tif"], + "x_odd.tif": self.odd()["odd.tif"], + "x_far.tif": whole(rows=3, cols=3, x_min=x_min + 100 * DX, y_max=y_max), + } + result = mz.plan_mosaic( + footprints(footprint, tiles), bounds_of(mz, self.INSIDE_THE_STRIP), None + ) + assert [p.name for p in result.tiles] == ["x_odd.tif"] + assert result.reference == (x_min, y_max) + + @pytest.mark.parametrize( + ("prefix", "winner"), + [("a", "odd"), ("o", "main")], + ids=["odd-first-name-sorts-first", "main-first-name-sorts-first"], + ) + def test_a_tie_goes_to_the_lattice_whose_first_tile_sorts_first( + self, plan: Any, prefix: str, winner: str + ) -> None: + """Four tiles each. Main's first tile is `ne.tif` (last `sw.tif`); the + odd lattice's first is `-ne.tif` (last `-sw.tif`). + With prefix `o`, main's first name sorts first but the odd lattice's + last name does: a tie broken by the last tile, or by the larger first + name, picks the wrong one in one of the two cases.""" + odd = self.odd(prefix, pieces=4) + result = plan({**self.main(), **odd}, self.INSIDE_THE_STRIP) + names = {p.name for p in result.tiles} + if winner == "odd": + assert names <= set(odd), names + assert result.reference == self.ODD_ORIGIN + else: + assert names == {"se.tif"} + assert result.reference == (X0, Y0) + + @pytest.mark.parametrize( + "odd_pieces", [1, 4], ids=["most-tiles", "tie-by-name"] + ) # fmt: skip + def test_the_choice_does_not_depend_on_footprint_order( + self, mz: ModuleType, footprint: Any, odd_pieces: int + ) -> None: + tiles = {**self.main(), **self.odd("a", pieces=odd_pieces)} + prints = footprints(footprint, tiles) + box = bounds_of(mz, self.INSIDE_THE_STRIP) + first = mz.plan_mosaic(prints, box, None) + for k in range(len(prints)): # every rotation, forwards and backwards + rotated = prints[k:] + prints[:k] + for order in (rotated, rotated[::-1]): + assert mz.plan_mosaic(order, box, None) == first + + +def _meta_change(change: dict[str, Any]) -> dict[str, Any]: + """`piece` takes `RasterMeta` field names.""" + names = {"dx": "delta_x", "dy": "delta_y", "area": "pixel_is_area"} + return {names.get(key, key): value for key, value in change.items()} + + +class TestM3AlignmentTolerance: + """M3 and R4 point 1: offsets within `ALIGN_TOLERANCE` cells of an integer.""" + + def test_a_decimetre_grid_with_float_noise_plans_on_integer_windows( + self, mz: ModuleType, plan: Any + ) -> None: + source = whole(x_min=600_000.0, y_max=7_000_000.0, dx=0.1, dy=0.1) + tiles = quadrants(source, row_cut=4, col_cut=6, overlap=1) + # A producer that accumulates its tie point: 6 steps of 0.1 is not 0.6. + accumulated = 600_000.0 + for _ in range(6): + accumulated += 0.1 + tiles["ne.tif"] = piece(source, 0, 5, 6, 13, x_min=accumulated) + assert (accumulated - 600_000.0) / 0.1 != 6 # the noise is there (1.4e-9 cells) + result = plan(tiles) + placed = {p.name: p.canvas for p in result.tiles} + assert placed["ne.tif"] == window(mz, 0, 6, 5, 7) + assert placed["se.tif"] == window(mz, 4, 6, 5, 7) + assert result.meta.x_min == 600_000.0 + + @pytest.mark.parametrize(("noise", "accepted"), [(5e-7, True), (2e-6, False)]) + def test_the_tolerance_boundary( + self, plan: Any, noise: float, accepted: bool, refused: Any + ) -> None: + source = whole() + tiles = { + "nw.tif": piece(source, 0, 5, 0, 7), + "ne.tif": piece(source, 0, 5, 6, 13, x_min=X0 + (6 + noise) * DX), + } + if accepted: + assert len(plan(tiles).tiles) == 2 + else: + refused(lambda: plan(tiles), "nw.tif", "ne.tif") + + +class TestM4Bounds: + """M4 and R4 point 4: the box snapped outward to the lattice, clamped.""" + + @pytest.fixture + def tiles(self) -> dict[str, DemTile]: + return quadrants(whole(), row_cut=4, col_cut=6, overlap=1) + + @pytest.mark.parametrize( + ("box", "expected"), + [ + # cols floor(1.2)=1 .. ceil(3.3)=4; rows floor(0.8)=0 .. ceil(2.6)=3 + ((500012.0, 6599987.0, 500033.0, 6599996.0), (0, 1, 4, 4)), + # inside one cell: 2 x 2 nodes + ((500012.0, 6599991.0, 500018.0, 6599994.0), (1, 1, 2, 2)), + # on nodes exactly: floor and ceil of integers add nothing + ((500010.0, 6599985.0, 500030.0, 6599995.0), (1, 1, 3, 3)), + # past the west and north edges: clamped + ((499000.0, 6599991.0, 500025.0, 6600500.0), (0, 0, 3, 4)), + # at the east edge, two columns remain + ((500115.0, 6599981.0, 500200.0, 6599984.0), (3, 11, 2, 2)), + ], + ids=["outward", "inside-one-cell", "on-nodes", "clamped", "east-edge"], + ) + def test_the_window( + self, + mz: ModuleType, + build: Any, + tiles: dict[str, DemTile], + box: tuple[float, ...], + expected: tuple[int, ...], + ) -> None: + row0, col0, rows, cols = expected + result, _ = build(tiles, box) + assert result.plan.window == window(mz, *expected) + assert result.plan.meta.x_min == X0 + col0 * DX + assert result.plan.meta.y_max == Y0 - row0 * DY + assert (result.plan.meta.rows, result.plan.meta.cols) == (rows, cols) + assert same_array(result.tile.array, whole().array[row0 : row0 + rows, col0 : col0 + cols]) + + def test_a_tile_touching_the_box_on_one_node_line_contributes_it( + self, mz: ModuleType, build: Any + ) -> None: + source = whole(rows=8, cols=12, area=True) + tiles = quadrants(source, row_cut=4, col_cut=6, overlap=0) + # x_max lands exactly on ne's first column (500060); y on the top rows. + result, loads = build(tiles, (500012.0, 6599991.0, 500060.0, 6599999.0)) + placed = {p.name: p for p in result.plan.tiles} + assert set(placed) == {"nw.tif", "ne.tif"} + assert placed["ne.tif"].source == window(mz, 0, 0, 3, 1) + assert placed["ne.tif"].canvas == window(mz, 0, 5, 3, 1) + assert same_array(result.tile.array, source.array[0:3, 1:7]) + assert sorted(loads.calls) == ["ne.tif", "nw.tif"] + + def test_float_noise_in_a_decimetre_spacing_does_not_add_a_node_line( + self, mz: ModuleType, plan: Any + ) -> None: + """S3, R4 point 4's snap. At spacing 0.1 from a reference at (0, 1): + 0.3 / 0.1 is 2.9999999999999996 (floor 2), (1 - 0.9) / 0.1 is + 0.9999999999999998 (floor 0) and (1 - 0.7) / 0.1 is 3.0000000000000004 + (ceil 4). Each edge is a node, so each would add a line without the snap.""" + assert (0.3 / 0.1, (1.0 - 0.9) / 0.1, (1.0 - 0.7) / 0.1) == ( + 2.9999999999999996, + 0.9999999999999998, + 3.0000000000000004, + ) # the noise is there + tiles = {"d.tif": whole(x_min=0.0, y_max=1.0, dx=0.1, dy=0.1)} + result = plan(tiles, (0.3, 0.7, 0.6, 0.9)) + assert result.window == window(mz, 1, 3, 3, 4) + + def test_a_box_meeting_no_tile_is_refused( + self, plan: Any, refused: Any, tiles: dict[str, DemTile] + ) -> None: + refused(lambda: plan(tiles, (400000.0, 6000000.0, 400100.0, 6000100.0))) + + def test_a_box_leaving_one_column_at_the_edge_is_refused( + self, plan: Any, refused: Any, tiles: dict[str, DemTile] + ) -> None: + refused(lambda: plan(tiles, (500120.0, 6599980.0, 500200.0, 6599990.0))) + + +class TestM5MemoryCap: + """M5, changed for 15a (R7): the cap is half of physical memory, from a function.""" + + NODES = 9 * 13 + + def test_physical_memory_is_the_machines(self, mz: ModuleType) -> None: + import os + + pages = os.sysconf("SC_PHYS_PAGES") * os.sysconf("SC_PAGE_SIZE") + assert mz.physical_memory() == pages + + @pytest.mark.parametrize( + ("memory", "accepted"), [(2 * NODES * 4, True), (2 * NODES * 4 - 1, False)] + ) + def test_the_boundary_at_four_bytes_a_node( + self, + mz: ModuleType, + plan: Any, + refused: Any, + monkeypatch: pytest.MonkeyPatch, + memory: int, + accepted: bool, + ) -> None: + monkeypatch.setattr(mz, "physical_memory", lambda: memory) + tiles = quadrants(whole(), row_cut=4, col_cut=6, overlap=1) + if accepted: + assert plan(tiles).meta.rows == 9 + else: + refused(lambda: plan(tiles), "--bbox") + + @pytest.mark.parametrize( + ("memory", "accepted"), [(2 * NODES * 8, True), (2 * NODES * 8 - 1, False)] + ) + def test_the_boundary_at_eight_bytes_a_node_when_a_selected_tile_decodes_to_float64( + self, + mz: ModuleType, + footprint: Any, + refused: Any, + monkeypatch: pytest.MonkeyPatch, + memory: int, + accepted: bool, + ) -> None: + """S2: the cap counts the planned decoded dtype, `np.result_type` of the + selected footprints' `dtype`. One float64 tile makes the canvas float64 + (R5), so the window costs 8 bytes a node, not 4.""" + monkeypatch.setattr(mz, "physical_memory", lambda: memory) + tiles = quadrants(whole(), row_cut=4, col_cut=6, overlap=1) + prints = [ + footprint(name=name, meta=tile.meta, dtype=np.dtype(np.float64)) + if name == "se.tif" + else footprint(name=name, meta=tile.meta) + for name, tile in tiles.items() + ] + call = lambda: mz.plan_mosaic(prints, None, None) # noqa: E731 + if accepted: + assert call().meta.rows == 9 + else: + refused(call, "--bbox", "float64") + + def test_a_float64_tile_outside_the_window_does_not_count( + self, mz: ModuleType, footprint: Any, monkeypatch: pytest.MonkeyPatch + ) -> None: + """Only the selected tiles decode, so only their dtypes size the canvas.""" + monkeypatch.setattr(mz, "physical_memory", lambda: 2 * 16 * 4) + tiles = quadrants(whole(), row_cut=4, col_cut=6, overlap=1) + prints = [ + footprint(name=name, meta=tile.meta, dtype=np.dtype(np.float64)) + if name == "se.tif" + else footprint(name=name, meta=tile.meta) + for name, tile in tiles.items() + ] + box = bounds_of(mz, (500012.0, 6599987.0, 500033.0, 6599996.0)) # 4 x 4, nw alone + assert [p.name for p in mz.plan_mosaic(prints, box, None).tiles] == ["nw.tif"] + + def test_a_footprints_dtype_defaults_to_float32(self, footprint: Any) -> None: + """S2: `TileFootprint` gains an optional decoded `dtype`, float32 unless given.""" + tile = whole() + assert np.dtype(footprint(name="a.tif", meta=tile.meta).dtype) == np.float32 + given = footprint(name="a.tif", meta=tile.meta, dtype=np.dtype(np.float64)) + assert np.dtype(given.dtype) == np.float64 + + def test_the_cap_is_on_the_window_not_the_union( + self, mz: ModuleType, plan: Any, monkeypatch: pytest.MonkeyPatch + ) -> None: + monkeypatch.setattr(mz, "physical_memory", lambda: 2 * 16 * 4) + tiles = quadrants(whole(), row_cut=4, col_cut=6, overlap=1) + assert plan(tiles, (500012.0, 6599987.0, 500033.0, 6599996.0)).meta.cols == 4 + + +class TestM6PlanOrderIndependence: + """M6 and I2: every permutation of the footprints gives the same plan.""" + + def test_every_permutation(self, mz: ModuleType, footprint: Any) -> None: + tiles = quadrants(whole(), row_cut=4, col_cut=6, overlap=1) + prints = footprints(footprint, tiles) + first = mz.plan_mosaic(prints, None, None) + for order in itertools.permutations(prints): + assert mz.plan_mosaic(list(order), None, None) == first + + +class TestM13ReferenceNode: + """M13 and R4 point 2: global indices count from the lattice's reference node. + + Spacing 0.3 x 0.7 and an origin picked (by search) so that stepping from + `se`'s own origin, `(XR + 6 dx) + dx`, differs in the last bit from + `XR + 7 dx`: a formula other than `X_ref + c0 * dx` fails bit-for-bit. + """ + + XR, YR, SX, SY = 3448.3132, 515.2302, 0.3, 0.7 # searched: se's own origin + 1 cell differs + + @pytest.fixture + def tiles(self) -> dict[str, DemTile]: + source = whole(x_min=self.XR, y_max=self.YR, dx=self.SX, dy=self.SY) + return quadrants(source, row_cut=4, col_cut=6, overlap=1) + + def box(self, c0: float, r0: float, c1: float, r1: float) -> tuple[float, ...]: + return ( + self.XR + c0 * self.SX, + self.YR - r1 * self.SY, + self.XR + c1 * self.SX, + self.YR - r0 * self.SY, + ) + + def test_the_reference_is_the_north_west_node_of_the_whole_lattice( + self, plan: Any, tiles: dict[str, DemTile] + ) -> None: + only_se = plan(tiles, self.box(7.5, 5.5, 10.2, 7.3)) + assert [p.name for p in only_se.tiles] == ["se.tif"] + assert only_se.reference == (self.XR, self.YR) + + @pytest.mark.parametrize( + "box", [None, (7.5, 5.5, 10.2, 7.3), (0.5, 0.5, 12.0, 8.0), (5.5, 3.5, 6.5, 4.5)], + ids=["no-bounds", "se-alone", "most", "the-shared-corner"], + ) # fmt: skip + def test_the_origin_is_reference_plus_index_times_spacing_bit_for_bit( + self, plan: Any, tiles: dict[str, DemTile], box: tuple[float, ...] | None + ) -> None: + result = plan(tiles, None if box is None else self.box(*box)) + assert result.meta.x_min == result.reference[0] + result.window.col0 * self.SX + assert result.meta.y_max == result.reference[1] - result.window.row0 * self.SY + + def test_a_tiles_global_index_does_not_depend_on_the_request( + self, plan: Any, tiles: dict[str, DemTile] + ) -> None: + requests = [ + None, + self.box(7.5, 5.5, 10.2, 7.3), + self.box(0.5, 0.5, 12.0, 8.0), + self.box(5.5, 3.5, 6.5, 4.5), + ] + for box in requests: + assert global_origin(plan(tiles, box), "se.tif") == (4, 6), box + + def test_the_mosaic_meta_takes_the_combined_provenance(self, plan: Any) -> None: + """R4: `nodata_source` from the first tile by name; `vertical_unit_assumed` if any.""" + source = whole(nodata=SENTINEL, vertical_unit_assumed=False) + tiles = quadrants(source, row_cut=4, col_cut=6, overlap=1) + tiles["ne.tif"] = piece(source, 0, 5, 6, 13, nodata_source="caller") + tiles["sw.tif"] = piece(source, 4, 9, 0, 7, vertical_unit_assumed=True) + result = plan(tiles) + assert result.meta.nodata == SENTINEL + assert result.meta.nodata_source == "caller" + assert result.meta.vertical_unit_assumed is True + + +class TestM14Coverage: + """M14 and R4 point 5 (Ola's ruling in 18 R6): a needed node no tile covers. + + Area-registered 3 x 4-node blocks of a 6 x 8 grid, with block (1, 1) + missing: its nodes are x 500040..500070, y 6599975..6599985. + """ + + @pytest.fixture + def source(self) -> DemTile: + return whole(rows=6, cols=8, area=True) + + @pytest.fixture + def holed(self, source: DemTile) -> dict[str, DemTile]: + return blocks(source, 3, 4, skip={(1, 1)}) + + def test_a_hole_inside_the_window_is_refused_naming_it( + self, plan: Any, refused: Any, holed: dict[str, DemTile] + ) -> None: + refused(lambda: plan(holed), "500040", "500070", "6599975", "6599985") + + def test_a_hole_in_the_middle_is_refused(self, plan: Any, refused: Any) -> None: + """No corner of the window is uncovered: a check of the corners, or of + the union's bounding box, passes this; only a check of every node fails it.""" + tiles = blocks(whole(rows=9, cols=12, area=True), 3, 4, skip={(1, 1)}) + refused(lambda: plan(tiles), "500040", "500070") + + def test_refusing_a_mostly_uncovered_window_costs_less_than_its_canvas( + self, plan: Any, refused: Any + ) -> None: + """S1: the check runs on windows the cap allows, up to half of physical + memory at 4 bytes a node, so what it allocates per node bounds what the + cap can promise. Two 2 x 2 tiles at opposite corners of a 2000 x 2000 + window: nearly every node is uncovered. Index and coordinate arrays of + the uncovered nodes cost 32 bytes a node, 8x the float32 canvas; the + refusal must peak below the canvas itself (16 MB here). tracemalloc + sees numpy's buffers, so this is a count of bytes, not a timing.""" + import tracemalloc + + n = 2000 + corners = { + "nw.tif": whole(rows=2, cols=2), + "se.tif": whole(rows=2, cols=2, x_min=X0 + (n - 2) * DX, y_max=Y0 - (n - 2) * DY), + } + canvas_bytes = n * n * 4 + tracemalloc.start() + try: + refused(lambda: plan(corners), f"{X0 + (n - 1) * DX:.0f}", f"{Y0 - (n - 1) * DY:.0f}") + _, peak = tracemalloc.get_traced_memory() + finally: + tracemalloc.stop() + assert peak < canvas_bytes, f"peak {peak} bytes for a {canvas_bytes}-byte canvas" + + def test_a_request_away_from_the_hole_plans(self, plan: Any, holed: dict[str, DemTile]) -> None: + result = plan(holed, (500001.0, 6599991.0, 500069.0, 6599999.0)) + assert {p.name for p in result.tiles} == {"b00.tif", "b01.tif"} + + def test_a_request_reaching_one_hole_node_is_refused( + self, plan: Any, refused: Any, holed: dict[str, DemTile] + ) -> None: + # rows 0..3 (y 6600000..6599985), cols 0..4 (x 500000..500040): node + # (500040, 6599985) is the hole's north-west node. + refused( + lambda: plan(holed, (500000.0, 6599985.0, 500040.0, 6600000.0)), "500040", "6599985" + ) + + def test_the_hole_outside_the_needed_region_is_nan_filler( + self, build: Any, source: DemTile, holed: dict[str, DemTile] + ) -> None: + needed = shapely.box(500000.0, 6599990.0, 500070.0, 6600000.0).union( + shapely.box(500000.0, 6599975.0, 500030.0, 6599990.0) + ) + result, _ = build(holed, None, needed) + array = result.tile.array + assert np.isnan(array[3:, 4:]).all() + mask = np.ones(array.shape, dtype=bool) + mask[3:, 4:] = False + assert np.array_equal(array[mask], source.array[mask]) + + def test_a_needed_region_touching_one_hole_node_is_refused( + self, plan: Any, refused: Any, holed: dict[str, DemTile] + ) -> None: + needed = shapely.box(500000.0, 6599990.0, 500070.0, 6600000.0).union( + shapely.box(500000.0, 6599975.0, 500040.0, 6599990.0) + ) # reaches x 500040: the hole's west column, on the boundary + refused(lambda: plan(holed, None, needed), "500040") + + +# --------------------------------------------------------------------------- +# Assembly (pixels, no files) +# --------------------------------------------------------------------------- + + +class TestM7Values: + """M7: values at the right nodes; the result's array contract.""" + + def test_values_land_at_their_nodes(self, build: Any) -> None: + source = whole() + result, _ = build(quadrants(source, row_cut=4, col_cut=6, overlap=1)) + assert result.tile.meta == source.meta + assert same_array(result.tile.array, source.array) + + def test_the_array_is_read_only_and_c_contiguous(self, build: Any) -> None: + result, _ = build(quadrants(whole(), row_cut=4, col_cut=6, overlap=1)) + array = result.tile.array + assert not array.flags.writeable + assert array.flags.c_contiguous + with pytest.raises(ValueError): + array[0, 0] = 1.0 + + def test_float64_when_any_tile_is(self, build: Any) -> None: + source = whole() + tiles = quadrants(source, row_cut=4, col_cut=6, overlap=1) + tiles["se.tif"] = piece( + source, 4, 9, 6, 13, array=source.array[4:9, 6:13].astype(np.float64) + ) + result, _ = build(tiles) + assert result.tile.array.dtype == np.float64 + assert same_array(result.tile.array, source.array.astype(np.float64)) + + def test_float32_stays_float32(self, build: Any) -> None: + result, _ = build(quadrants(whole(), row_cut=4, col_cut=6, overlap=1)) + assert result.tile.array.dtype == np.float32 + + +class TestM8Overlaps: + """M8, R5 and I3: valid beats NoData; a disagreement is not refused but + decided by depth and reported (Ola's Q1 revised, 2026-09-28). + + `w.tif` and `e.tif` cut from a 4 x 9 grid, sharing `overlap` columns. + With overlap 3 the shared columns are 4, 5 and 6, and each node's own + depth (`mosaic_fixtures.own_depth`) is, in `w.tif` then `e.tif`: + + rows 0 and 3: every node is on both tiles' top or bottom border, + 0 against 0, a tie, so `e.tif` (it sorts first); + rows 1 and 2: column 4 is 1 against 0, `w.tif`; column 5 is 1 + against 1, `e.tif`; column 6 is 0 against 1, `e.tif`. + + The four tests below that pinned the old refusal (Q1 as first ruled) now + pin the report; each says what it kept. + """ + + ROWS, COLS = 4, 9 + + @staticmethod + def pair(overlap: int, nodata: float | None = None) -> tuple[DemTile, dict[str, np.ndarray]]: + source = whole(rows=TestM8Overlaps.ROWS, cols=TestM8Overlaps.COLS, nodata=nodata) + cut = 4 + return source, { + "w.tif": source.array[:, : cut + overlap].copy(), + "e.tif": source.array[:, cut:].copy(), + } + + @staticmethod + def tiles(source: DemTile, arrays: dict[str, np.ndarray]) -> dict[str, DemTile]: + w, e = arrays["w.tif"], arrays["e.tif"] + return { + "w.tif": piece(source, 0, w.shape[0], 0, w.shape[1], array=w), + "e.tif": piece(source, 0, e.shape[0], 4, 4 + e.shape[1], array=e), + } + + @pytest.mark.parametrize("overlap", [1, 3]) + @pytest.mark.parametrize("holder", ["w.tif", "e.tif"]) + def test_valid_beats_nan_in_either_tile(self, build: Any, overlap: int, holder: str) -> None: + source, arrays = self.pair(overlap) + column = 4 if holder == "e.tif" else 4 + overlap - 1 + local = column - (4 if holder == "e.tif" else 0) + arrays[holder][1, local] = np.nan + result, _ = build(self.tiles(source, arrays)) + assert same_array(result.tile.array, source.array) + + @pytest.mark.parametrize("holder", ["w.tif", "e.tif"]) + def test_valid_beats_the_sentinel_in_either_tile(self, build: Any, holder: str) -> None: + source, arrays = self.pair(3, nodata=SENTINEL) + arrays[holder][2, 5 if holder == "w.tif" else 1] = SENTINEL # canvas column 5 + result, _ = build(self.tiles(source, arrays)) + assert same_array(result.tile.array, source.array) + + def test_both_nan_stays_nan_and_both_sentinel_stays_sentinel(self, build: Any) -> None: + source, arrays = self.pair(3, nodata=SENTINEL) + arrays["w.tif"][0, 5] = arrays["e.tif"][0, 1] = np.nan + arrays["w.tif"][3, 6] = arrays["e.tif"][3, 2] = SENTINEL + result, _ = build(self.tiles(source, arrays)) + array = result.tile.array + assert np.isnan(array[0, 5]) + assert array[3, 6] == SENTINEL + expected = source.array.copy() + expected[0, 5], expected[3, 6] = np.nan, SENTINEL + assert same_array(array, expected) + + def test_nan_against_the_sentinel_is_nodata_in_either_order( + self, mz: ModuleType, plan: Any + ) -> None: + """Which NoData wins is not ruled; that it is NoData, and the same in + every order, is (I2, I3).""" + source, arrays = self.pair(3, nodata=SENTINEL) + arrays["w.tif"][1, 4] = np.nan + arrays["e.tif"][1, 0] = SENTINEL + tiles = self.tiles(source, arrays) + planned = plan(tiles) + results = [ + mz.assemble(planned.model_copy(update={"tiles": order}), Loads(tiles)).tile.array + for order in itertools.permutations(planned.tiles) + ] + value = results[0][1, 4] + assert np.isnan(value) or value == SENTINEL + assert all(same_array(r, results[0]) for r in results) + + def test_equal_overlaps_are_accepted(self, build: Any) -> None: + source, arrays = self.pair(3) + result, _ = build(self.tiles(source, arrays)) + assert same_array(result.tile.array, source.array) + + @pytest.mark.parametrize( + ("west_wins", "expected"), + [(False, (3, 2.5, 1.0)), (True, (4, 4.0, 1.75))], + ids=["odd-count", "even-count"], + ) + @pytest.mark.parametrize("order", ["as-named", "reversed"]) + def test_a_disagreement_is_reported_with_the_count_the_largest_and_the_median( + self, + mz: ModuleType, + plan: Any, + order: str, + west_wins: bool, + expected: tuple[int, float, float], + ) -> None: + """Was `..._is_refused_naming_both_the_count_and_the_largest`. Same + planted differences; the run is no longer refused. It keeps: both names, + the count of differing nodes only (a NoData node in the overlap is not + one, nor are the agreeing ones), the largest |difference|, in either + order. New: the median, of the differing nodes only, and the value each + node takes. The median of an even count is the mean of the middle two. + """ + source, arrays = self.pair(3) + arrays["e.tif"][0, 1] += 0.5 # row 0, column 5: a tie, e.tif's value + arrays["e.tif"][2, 1] += 2.5 # row 2, column 5, mid-overlap: 1 against 1, e.tif + arrays["e.tif"][3, 2] -= 1.0 # row 3, column 6: |difference| is 1 + arrays["w.tif"][1, 4] = np.nan # a NoData node in the overlap is not counted + if west_wins: + arrays["e.tif"][2, 0] += 4.0 # row 2, column 4: w.tif is deeper, its value + tiles = self.tiles(source, arrays) + planned = plan(tiles) + if order == "reversed": + planned = planned.model_copy(update={"tiles": tuple(reversed(planned.tiles))}) + result = mz.assemble(planned, Loads(tiles)) + assert seams(result) == [("e.tif", "w.tif", *expected)] + taken = source.array.copy() + taken[0, 5] += 0.5 + taken[2, 5] += 2.5 + taken[3, 6] -= 1.0 + assert same_array(result.tile.array, taken) + + def test_the_report_names_the_tiles_whose_values_disagree( + self, mz: ModuleType, plan: Any + ) -> None: + """B2, kept in the report (was `test_the_refusal_names_...`). Three + tiles on one 3 x 3 grid, equal everywhere but node (0, 0), where `a` is + NoData, `b` 1.0 and `c` 5.0. The seam is b | c; `a` covers the node + too, but its value is not in the disagreement, so no seam names it. + Node (0, 0) is on all three borders: a tie, and of the two valid + values `b.tif`'s, whose name sorts first. In every assembly order.""" + source = whole(rows=3, cols=3) + tiles = {} + for name, first in (("a.tif", np.nan), ("b.tif", 1.0), ("c.tif", 5.0)): + array = source.array.copy() + array[0, 0] = first + tiles[name] = piece(source, 0, 3, 0, 3, array=array) + planned = plan(tiles) + for order in itertools.permutations(planned.tiles): + result = mz.assemble(planned.model_copy(update={"tiles": order}), Loads(tiles)) + names = [p.name for p in order] + assert seams(result) == [("b.tif", "c.tif", 1, 4.0, 4.0)], names + assert result.tile.array[0, 0] == 1.0, names + + def test_one_ulp_is_below_the_threshold_and_still_decided_by_depth(self, build: Any) -> None: + """Was `test_one_ulp_is_a_disagreement`, which pinned `!=` (one ulp + reported). Ola, 2026-09-28: "Ignore below 1mm", so one ulp is not + reported. What it kept: the node's value is still decided by the + midline rule. Overlap 1: the shared column is on both tiles' borders, + a tie, `e.tif`'s value.""" + source, arrays = self.pair(1) + value = arrays["e.tif"][2, 0] + bumped = np.nextafter(value, np.float32(np.inf), dtype=np.float32) + arrays["e.tif"][2, 0] = bumped + result, _ = build(self.tiles(source, arrays)) + assert result.seams == () + assert result.tile.array[2, 4] == bumped + + def test_a_disagreement_against_a_sentinel_tile_is_still_one(self, build: Any) -> None: + """The sentinel is NoData only where a node holds it; elsewhere a value + is a value, reported and decided by depth like any other.""" + source, arrays = self.pair(3, nodata=SENTINEL) + arrays["w.tif"][0, 4] = SENTINEL # NoData: e.tif's value, not counted + arrays["w.tif"][0, 5] += 7.0 # a tie on row 0: e.tif's value + arrays["w.tif"][1, 4] += 3.0 # w.tif is deeper: its value + result, _ = build(self.tiles(source, arrays)) + assert seams(result) == [("e.tif", "w.tif", 2, 7.0, 5.0)] + taken = source.array.copy() + taken[1, 4] += 3.0 + assert same_array(result.tile.array, taken) + + +def taken_from(result: Any, source: np.ndarray, offsets: dict[str, float]) -> list[str]: + """Row strings naming, node by node, the tile each value came from. + + Every tile is `source` plus its own offset, so the offset identifies it; + a tile is written as its name's first letter. + """ + letter = {offset: name[0] for name, offset in offsets.items()} + taken = np.asarray(result.tile.array, dtype=np.float64) - source + return ["".join(letter[float(v)] for v in row) for row in taken] + + +def offset_tiles(tiles: dict[str, DemTile]) -> tuple[dict[str, DemTile], dict[str, float]]: + """Each tile plus 1000 times its rank by name: every overlap disagrees.""" + offsets = {name: 1000.0 * (i + 1) for i, name in enumerate(sorted(tiles))} + return {name: shifted_by(t, offsets[name]) for name, t in tiles.items()}, offsets + + +class TestQ1DeepestInterior: + """Ola's Q1 revised (2026-09-28): each overlap node takes the value of the + tile whose interior it lies deepest in (the largest distance, in nodes, to + that tile's *own* nearest border); ties go to the name that sorts first; + NoData still loses to a valid value; the result is independent of tile + order. + + Every tile is the source plus its own offset, so every overlap disagrees + and the offset names the tile a value came from. The expected maps are + written out by hand, and the whole array is also checked against + `mosaic_fixtures.deepest_interior`, a node-by-node oracle that places + tiles by coordinates and shares no code with `tin_engine.mosaic`. + """ + + ROWS, WIDTH = 7, 8 + + def two(self, overlap: int, west: str, east: str) -> tuple[DemTile, dict[str, DemTile]]: + """Two 7 x 8 tiles side by side sharing `overlap` columns; west + 1000, + east + 2000.""" + cols = 2 * self.WIDTH - overlap + source = whole(self.ROWS, cols) + return source, { + west: shifted_by(piece(source, 0, self.ROWS, 0, self.WIDTH), 1000.0), + east: shifted_by(piece(source, 0, self.ROWS, self.WIDTH - overlap, cols), 2000.0), + } + + #: The overlap columns only, row by row. W: the west tile is deeper; E: the + #: east; T: equally deep, so the name that sorts first. Rows 0 and 6 lie on + #: both tiles' borders. An odd overlap has a tie column down the middle; an + #: even one splits cleanly where the rows are deep enough (rows 2-4), and + #: rows 1 and 5, 1 deep, tie the middle two columns. + MAPS: ClassVar[dict[int, list[str]]] = { + 3: ["TTT", "WTE", "WTE", "WTE", "WTE", "WTE", "TTT"], + 4: ["TTTT", "WTTE", "WWEE", "WWEE", "WWEE", "WTTE", "TTTT"], + } + + @pytest.mark.parametrize("overlap", [3, 4], ids=["odd-tie-column", "even-midline"]) + @pytest.mark.parametrize( + ("west", "east"), [("w.tif", "e.tif"), ("a.tif", "b.tif")], ids=["east-first", "west-first"] + ) + def test_two_tiles_split_down_the_middle_ties_by_name( + self, build: Any, overlap: int, west: str, east: str + ) -> None: + source, tiles = self.two(overlap, west, east) + result, _ = build(tiles) + letter = {"W": west[0], "E": east[0], "T": min(west, east)[0]} + own = self.WIDTH - overlap + expected = [ + west[0] * own + "".join(letter[k] for k in row) + east[0] * own + for row in self.MAPS[overlap] + ] + assert taken_from(result, source.array, {west: 1000.0, east: 2000.0}) == expected + oracle = deepest_interior(tiles, result.tile.meta).astype(np.float32) + assert same_array(result.tile.array, oracle) + + def test_one_shared_line_is_all_ties(self, build: Any) -> None: + """Point-registered neighbours share one line, 0 deep in both: every + node is a tie, and the name that sorts first takes the whole line.""" + source, tiles = self.two(1, "w.tif", "e.tif") + result, _ = build(tiles) + taken = taken_from(result, source.array, {"w.tif": 1000.0, "e.tif": 2000.0}) + assert [row[self.WIDTH - 1] for row in taken] == ["e"] * self.ROWS + + def test_depth_is_to_the_tiles_own_border_not_the_requests_window(self, build: Any) -> None: + """A box of rows 2-4 and columns 4-8 cuts both tiles of the odd pair. + The window's first and last rows are neither tile's border, so the + answer is rows 2-4 of the full tiles' map (W T E in columns 5-7), not + the window's (all ties, `e.tif`, on its first and last row).""" + source, tiles = self.two(3, "w.tif", "e.tif") + box = (X0 + 4 * DX, Y0 - 4 * DY, X0 + 8 * DX, Y0 - 2 * DY) + result, _ = build(tiles, box) + assert (result.tile.meta.rows, result.tile.meta.cols) == (3, 5) + taken = taken_from(result, source.array[2:5, 4:9], {"w.tif": 1000.0, "e.tif": 2000.0}) + assert taken == ["wweee"] * 3 + + #: Rows 2-6, columns 7-11 of a 9 x 12 tile holding a 5 x 5 one flush with + #: its east border. The small tile's own depth never exceeds the big one's, + #: so it takes only the ties, and only when its name sorts first. + INSIDE: ClassVar[dict[str, list[str]]] = { + "a.tif": ["bbbba", "bbbaa", "bbaaa", "bbbaa", "bbbba"], + "s.tif": ["bbbbb"] * 5, + } + + @pytest.mark.parametrize("small", ["a.tif", "s.tif"], ids=["small-first", "small-last"]) + def test_unequal_sizes_measure_depth_to_each_tiles_own_border( + self, build: Any, small: str + ) -> None: + """Depth is to each tile's own border: measured to the mosaic's border + (here the big tile's), both would tie at every node.""" + source = whole(9, 12) + tiles = { + "b.tif": shifted_by(piece(source, 0, 9, 0, 12), 1000.0), + small: shifted_by(piece(source, 2, 7, 7, 12), 2000.0), + } + result, _ = build(tiles) + taken = taken_from(result, source.array, {"b.tif": 1000.0, small: 2000.0}) + assert [row[7:] for row in taken[2:7]] == self.INSIDE[small] + assert all(set(row) == {"b"} for row in taken[:2] + taken[7:]) + oracle = deepest_interior(tiles, result.tile.meta).astype(np.float32) + assert same_array(result.tile.array, oracle) + + def test_a_corner_of_three_tiles(self, build: Any) -> None: + """`a` 10 x 7 down the west, `b` 7 x 8 and `c` 6 x 8 stacked on the + east; rows 4-6, columns 4-6 are in all three. At (6, 6) `c` is 2 deep + and the others 0; at (5, 5) all three are 1 deep, so `a`.""" + source = whole(10, 12) + tiles, offsets = offset_tiles( + { + "a.tif": piece(source, 0, 10, 0, 7), + "b.tif": piece(source, 0, 7, 4, 12), + "c.tif": piece(source, 4, 10, 4, 12), + } + ) + result, _ = build(tiles) + assert taken_from(result, source.array, offsets) == [ + "aaaaaaabbbbb", + "aaaaaabbbbbb", + "aaaaaabbbbbb", + "aaaaaabbbbbb", + "aaaaaabbbbbb", + "aaaaaabbbbbb", + "aaaaaacccccb", + "aaaaaacccccc", + "aaaaaacccccc", + "aaaaaaaccccc", + ] + oracle = deepest_interior(tiles, result.tile.meta).astype(np.float32) + assert same_array(result.tile.array, oracle) + + #: Quadrants of a 10 x 12 grid cut at row 5 and column 6 with overlap 3: + #: 8 x 9, 8 x 6, 5 x 9 and 5 x 6 tiles, all four sharing rows 5-7 and + #: columns 6-8. Named so the name order is nw < ne < sw < se, and then its + #: reverse; the ties move with it, the deeper tile does not. + CORNERS: ClassVar[dict[str, list[str]]] = { + "abcd": [ + "aaaaaaaaabbb", + "aaaaaaaabbbb", + "aaaaaaaabbbb", + "aaaaaaaabbbb", + "aaaaaaaabbbb", + "aaaaaaaabbbb", + "aaaaaaaabbbb", + "acccccccdddb", + "ccccccccdddd", + "cccccccccddd", + ], + "dcba": [ + "ddddddcccccc", + "dddddddccccc", + "dddddddccccc", + "dddddddccccc", + "dddddddccccc", + "bddddddcccca", + "bbbbbbbaaaaa", + "bbbbbbbaaaaa", + "bbbbbbbaaaaa", + "bbbbbbaaaaaa", + ], + } + + @staticmethod + def four( + letters: str, nodata: float | None = None + ) -> tuple[DemTile, dict[str, DemTile], dict[str, float]]: + source = whole(10, 12, nodata=nodata) + cut = quadrants(source, row_cut=5, col_cut=6, overlap=3) + named = { + f"{letter}.tif": cut[q] + for letter, q in zip(letters, ("nw.tif", "ne.tif", "sw.tif", "se.tif"), strict=True) + } + tiles, offsets = offset_tiles(named) + return source, tiles, offsets + + @pytest.mark.parametrize("letters", ["abcd", "dcba"]) + def test_a_corner_of_four_tiles(self, build: Any, letters: str) -> None: + source, tiles, offsets = self.four(letters) + result, _ = build(tiles) + assert taken_from(result, source.array, offsets) == self.CORNERS[letters] + oracle = deepest_interior(tiles, result.tile.meta).astype(np.float32) + assert same_array(result.tile.array, oracle) + + @pytest.mark.parametrize("nodata", [np.nan, SENTINEL], ids=["nan", "sentinel"]) + def test_nodata_in_the_deepest_tile_loses_to_a_shallower_valid_value( + self, build: Any, nodata: float + ) -> None: + """At (5, 6) `a.tif` (nw) is 2 deep and the other three 0: with NoData + there, the three valid values tie, and `b.tif` sorts first. At (4, 7), + in `a.tif` and `b.tif` only, `a.tif` is 1 deep and `b.tif` 1: a tie + `a.tif` would take, but its NoData leaves `b.tif`.""" + sentinel = None if np.isnan(nodata) else SENTINEL + source, tiles, offsets = self.four("abcd", sentinel) + array = np.asarray(tiles["a.tif"].array).copy() + array[5, 6] = array[4, 7] = nodata + tiles["a.tif"] = DemTile(meta=tiles["a.tif"].meta, array=array) + result, _ = build(tiles) + expected = [list(row) for row in self.CORNERS["abcd"]] + assert (expected[5][6], expected[4][7]) == ("a", "a") # with no NoData + expected[5][6] = expected[4][7] = "b" + assert taken_from(result, source.array, offsets) == ["".join(r) for r in expected] + + def test_every_order_gives_the_same_mosaic_and_report(self, mz: ModuleType, plan: Any) -> None: + """I2 under the new rule: all 24 assembly orders of the four-tile + corner, with NoData in the deepest tile at one node, give one array bit + for bit and one seam report.""" + _, tiles, _ = self.four("dcba") + array = np.asarray(tiles["d.tif"].array).copy() + array[5, 6] = np.nan + tiles["d.tif"] = DemTile(meta=tiles["d.tif"].meta, array=array) + planned = plan(tiles) + first = mz.assemble(planned, Loads(tiles)) + for order in itertools.permutations(planned.tiles): + result = mz.assemble(planned.model_copy(update={"tiles": order}), Loads(tiles)) + names = [p.name for p in order] + assert same_array(result.tile.array, first.tile.array), names + assert seams(result) == seams(first), names + oracle = deepest_interior(tiles, first.tile.meta).astype(np.float32) + assert same_array(first.tile.array, oracle) + + +class TestQ1SeamReport: + """Ola's Q1 revised: each disagreeing seam is reported, the two tiles (by + name, the one that sorts first first), the number of nodes where both hold + a valid value and the values differ, and the largest and the median + |difference| over those nodes. Pairs that agree are not listed. A + `Mosaic`'s `seams` is a tuple sorted by the pair's names. + """ + + def test_every_pair_of_the_four_corner(self, build: Any) -> None: + """Every overlap of the offset quadrants disagrees by a constant, so + each pair's count is its overlap's node count: 8 x 3 for a | b (nw, + ne), 3 x 9 for a | c (nw, sw), the 3 x 3 corner for a | d and b | c, + 3 x 6 for b | d and 5 x 3 for c | d.""" + _, tiles, _ = TestQ1DeepestInterior.four("abcd") + result, _ = build(tiles) + assert seams(result) == [ + ("a.tif", "b.tif", 24, 1000.0, 1000.0), + ("a.tif", "c.tif", 27, 2000.0, 2000.0), + ("a.tif", "d.tif", 9, 3000.0, 3000.0), + ("b.tif", "c.tif", 9, 1000.0, 1000.0), + ("b.tif", "d.tif", 18, 2000.0, 2000.0), + ("c.tif", "d.tif", 15, 1000.0, 1000.0), + ] + assert seams(result) == seams_of(tiles, result.tile.meta) + + @pytest.mark.parametrize("overlap", [0, 1, 3], ids=["abutting", "shared-line", "overlap-3"]) + def test_agreeing_seams_are_not_listed(self, build: Any, overlap: int) -> None: + area = overlap == 0 + source = whole(8, 12, area=area) if area else whole(9, 13) + result, _ = build(quadrants(source, row_cut=4, col_cut=6, overlap=overlap)) + assert result.seams == () + + def test_one_tile_has_no_seams(self, build: Any) -> None: + result, _ = build({"only.tif": whole()}) + assert result.seams == () + + def test_only_the_disagreeing_pair_is_listed(self, build: Any) -> None: + """Four quadrants that agree but for one node on the nw | ne seam + (outside the other two tiles): one entry, not six.""" + source = whole(9, 13) + tiles = quadrants(source, row_cut=4, col_cut=6, overlap=3) + array = np.asarray(tiles["ne.tif"].array).copy() + array[1, 1] -= 6.0 + tiles["ne.tif"] = DemTile(meta=tiles["ne.tif"].meta, array=array) + result, _ = build(tiles) + assert seams(result) == [("ne.tif", "nw.tif", 1, 6.0, 6.0)] + + def test_a_three_way_node_is_counted_in_each_disagreeing_pair(self, build: Any) -> None: + """At node (0, 0) `a` and `b` hold 1.0 and `c` 5.0: a | c and b | c + each count it; a | b agree and are not listed.""" + source = whole(rows=3, cols=3) + tiles = {} + for name, first in (("a.tif", 1.0), ("b.tif", 1.0), ("c.tif", 5.0)): + array = source.array.copy() + array[0, 0] = first + tiles[name] = piece(source, 0, 3, 0, 3, array=array) + result, _ = build(tiles) + assert seams(result) == [ + ("a.tif", "c.tif", 1, 4.0, 4.0), + ("b.tif", "c.tif", 1, 4.0, 4.0), + ] + + def test_the_report_covers_the_mosaics_nodes_only(self, build: Any) -> None: + """A difference outside `--bbox` is in no node the run reads, so it is + not reported; one inside is.""" + source = whole(9, 13) + tiles = quadrants(source, row_cut=4, col_cut=6, overlap=3) + array = np.asarray(tiles["ne.tif"].array).copy() + array[0, 0] += 2.0 # global (0, 6): outside the box + array[3, 1] += 5.0 # global (3, 7): inside it + tiles["ne.tif"] = DemTile(meta=tiles["ne.tif"].meta, array=array) + box = (X0 + 5 * DX, Y0 - 6 * DY, X0 + 8 * DX, Y0 - 2 * DY) # rows 2-6, cols 5-8 + result, _ = build(tiles, box) + assert seams(result) == [("ne.tif", "nw.tif", 1, 5.0, 5.0)] + + def test_differences_are_taken_in_float64(self, build: Any) -> None: + """0.5 against 2**24 + 2, both exact in float32: the difference is + 16777217.5 in float64 and would round to 16777218 in float32.""" + source = whole(rows=3, cols=3) + tiles = {} + for name, first in (("a.tif", 0.5), ("b.tif", 2.0**24 + 2)): + array = source.array.copy() + array[1, 1] = first + tiles[name] = piece(source, 0, 3, 0, 3, array=array) + result, _ = build(tiles) + assert seams(result) == [("a.tif", "b.tif", 1, 16777217.5, 16777217.5)] + + +class TestSeamThreshold: + """Ola, 2026-09-28: "Ignore below 1mm". A seam counts, and the report + lists, only nodes where both tiles hold a valid value and + |a - b| >= 1 mm, 0.001 in the DEM's units, in float64; `nodes`, `largest` + and `median` are over those nodes only, and a pair with none is not + listed. Which tile's value a node takes does not depend on the threshold. + """ + + #: Exactly 1 mm (the float64 0.001 the rule compares with) and the float64 + #: just below it. Planted as `gap` against 0.0, so `|a - b|` is the float64 + #: itself: no rounding stands between the planted value and the compare. + AT = 0.001 + BELOW = float(np.nextafter(0.001, 0.0)) + + @staticmethod + def same_grid(first: float, second: float) -> dict[str, DemTile]: + """`a.tif` and `b.tif` on one float64 3 x 3 grid, equal but at the + centre, where they hold `first` and `second`. Both are 1 deep there: + a tie, `a.tif`'s value.""" + source = whole(rows=3, cols=3, dtype=np.float64) + tiles = {} + for name, centre in (("a.tif", first), ("b.tif", second)): + array = source.array.copy() + array[1, 1] = centre + tiles[name] = piece(source, 0, 3, 0, 3, array=array) + return tiles + + @pytest.mark.parametrize("larger", ["a.tif", "b.tif"]) + def test_exactly_one_millimetre_counts(self, build: Any, larger: str) -> None: + assert self.AT == 1e-3 # the literal the ruling names, not a neighbour + first, second = (self.AT, 0.0) if larger == "a.tif" else (0.0, self.AT) + result, _ = build(self.same_grid(first, second)) + assert seams(result) == [("a.tif", "b.tif", 1, self.AT, self.AT)] + assert result.tile.array[1, 1] == first + + @pytest.mark.parametrize("larger", ["a.tif", "b.tif"]) + def test_just_below_one_millimetre_does_not(self, build: Any, larger: str) -> None: + assert 0.0 < self.BELOW < self.AT + first, second = (self.BELOW, 0.0) if larger == "a.tif" else (0.0, self.BELOW) + result, _ = build(self.same_grid(first, second)) + assert result.seams == () + assert result.tile.array[1, 1] == first + + def test_nodes_largest_and_median_are_over_the_qualifying_nodes_only(self, build: Any) -> None: + """`TestM8Overlaps`' pair with overlap 3; `e.tif` differs at four + nodes, by 0.0005, 0.0009, 2 and 4. Two qualify: nodes 2, largest 4, + median 3. Over all four the count would be 4 and the median ~1.""" + source, arrays = TestM8Overlaps.pair(3) + arrays["e.tif"][1, 0] += np.float32(0.0005) # global (1, 4) + arrays["e.tif"][1, 1] += np.float32(0.0009) # global (1, 5) + arrays["e.tif"][2, 1] += np.float32(2.0) # global (2, 5) + arrays["e.tif"][2, 2] += np.float32(4.0) # global (2, 6) + tiles = TestM8Overlaps.tiles(source, arrays) + result, _ = build(tiles) + assert seams(result) == [("e.tif", "w.tif", 2, 4.0, 3.0)] + assert seams(result) == seams_of(tiles, result.tile.meta) + + def test_a_pair_below_the_threshold_everywhere_is_not_listed(self, build: Any) -> None: + """Quadrants with overlap 3: `ne.tif` 0.5 mm off `nw.tif` at global + (1, 7), in those two only; `sw.tif` 3 off `nw.tif` at global (5, 1), + in those two only. One entry, nw | sw.""" + source = whole(9, 13) + tiles = quadrants(source, row_cut=4, col_cut=6, overlap=3) + for name, (r, c), delta in (("ne.tif", (1, 1), 0.0005), ("sw.tif", (1, 1), 3.0)): + array = np.asarray(tiles[name].array).copy() + array[r, c] += np.float32(delta) + tiles[name] = DemTile(meta=tiles[name].meta, array=array) + result, _ = build(tiles) + assert seams(result) == [("nw.tif", "sw.tif", 1, 3.0, 3.0)] + + def test_a_sub_millimetre_disagreement_is_still_decided_by_depth_then_name( + self, mz: ModuleType, plan: Any + ) -> None: + """The midline rule does not look at the threshold. `TestM8Overlaps`' + pair with overlap 3, 0.5 mm apart at three nodes of row 1: at column 4 + `w.tif` is deeper (1 against 0), at 5 a tie (`e.tif`, by name), at 6 + `e.tif` is deeper. So the name that sorts first does not take every + node, and in every assembly order; nothing is reported.""" + source, arrays = TestM8Overlaps.pair(3) + arrays["w.tif"][1, 4] += np.float32(0.0005) # global (1, 4): w.tif deeper + arrays["e.tif"][1, 1] += np.float32(0.0005) # global (1, 5): a tie + arrays["e.tif"][1, 2] -= np.float32(0.0005) # global (1, 6): e.tif deeper + tiles = TestM8Overlaps.tiles(source, arrays) + expected = source.array.copy() + expected[1, 4] = arrays["w.tif"][1, 4] + expected[1, 5] = arrays["e.tif"][1, 1] + expected[1, 6] = arrays["e.tif"][1, 2] + assert (expected[1, 4:7] != source.array[1, 4:7]).all() # all three planted + planned = plan(tiles) + for order in itertools.permutations(planned.tiles): + result = mz.assemble(planned.model_copy(update={"tiles": order}), Loads(tiles)) + names = [p.name for p in order] + assert same_array(result.tile.array, expected), names + assert result.seams == (), names + + +class TestM9SplitAndRestitch: + """M9 and I5: a grid cut into tiles reassembles to itself, meta and array.""" + + @pytest.mark.parametrize( + ("area", "overlap", "shape"), + [(False, 1, (9, 13)), (True, 0, (8, 12)), (True, 3, (8, 12)), (False, 3, (9, 13))], + ids=["point-shared-line", "area-abutting", "area-overlap-3", "point-overlap-3"], + ) + def test_quadrants(self, build: Any, area: bool, overlap: int, shape: tuple[int, int]) -> None: + source = whole(*shape, area=area) + result, _ = build(quadrants(source, row_cut=4, col_cut=6, overlap=overlap)) + assert result.tile.meta == source.meta + assert same_array(result.tile.array, source.array) + + def test_with_nodata_inside_and_across_the_seams(self, build: Any) -> None: + grid = values(9, 13) + grid[0, 0] = grid[4, 6] = grid[5, 7] = grid[8, 12] = SENTINEL # (4, 6) is in all four + grid[4, 2] = grid[6, 6] = grid[2, 9] = np.nan # on the seams + source = whole(9, 13, array=grid, nodata=SENTINEL) + result, _ = build(quadrants(source, row_cut=4, col_cut=6, overlap=3)) + assert result.tile.meta == source.meta + assert same_array(result.tile.array, source.array) + + def test_a_three_by_three_block_split(self, build: Any) -> None: + source = whole(9, 12, area=True) + result, _ = build(blocks(source, 3, 4)) + assert result.tile.meta == source.meta + assert same_array(result.tile.array, source.array) + + +class TestM10AssemblyOrderIndependence: + """M10 and I2: the assembled array does not depend on tile order.""" + + def test_every_order_of_the_plans_tiles(self, mz: ModuleType, plan: Any) -> None: + grid = values(9, 13) + grid[5, 7] = np.nan + grid[4, 6] = SENTINEL + source = whole(9, 13, array=grid, nodata=SENTINEL) + tiles = quadrants(source, row_cut=4, col_cut=6, overlap=3) + planned = plan(tiles) + for order in itertools.permutations(planned.tiles): + result = mz.assemble(planned.model_copy(update={"tiles": order}), Loads(tiles)) + assert same_array(result.tile.array, source.array), [p.name for p in order] + + def test_tiles_are_loaded_once_each_in_plan_order(self, build: Any) -> None: + result, loads = build(quadrants(whole(), row_cut=4, col_cut=6, overlap=1)) + assert loads.calls == [p.name for p in result.plan.tiles] == sorted(loads.calls) + + +class TestM11SingleTile: + """M11 and I7: one tile, no bounds: the loaded tile itself, no canvas.""" + + def test_the_loaded_tile_is_returned_as_is(self, build: Any) -> None: + source = whole() + result, loads = build({"only.tif": source}) + assert result.tile is loads.tiles["only.tif"] + + def test_with_bounds_it_is_a_window(self, build: Any) -> None: + source = whole() + result, _ = build({"only.tif": source}, (500012.0, 6599987.0, 500033.0, 6599996.0)) + assert same_array(result.tile.array, source.array[0:4, 1:5]) + + +class TestM12Loads: + """M12: only the selected tiles are loaded; a changed tile is refused.""" + + def test_only_selected_tiles_are_loaded(self, build: Any) -> None: + _, loads = build( + quadrants(whole(), row_cut=4, col_cut=6, overlap=1), + (500012.0, 6599987.0, 500033.0, 6599996.0), + ) + assert loads.calls == ["nw.tif"] + + def test_a_tile_changed_since_it_was_listed( + self, mz: ModuleType, plan: Any, refused: Any + ) -> None: + source = whole() + tiles = quadrants(source, row_cut=4, col_cut=6, overlap=1) + planned = plan(tiles) + moved = dict(tiles) + moved["nw.tif"] = piece(source, 0, 5, 0, 7, y_max=Y0 + DY) + refused(lambda: mz.assemble(planned, Loads(moved)), "nw.tif", "changed since it was listed") + + +class TestM15Adopt: + """M15 and R7: one canvas, handed to `DemTile` without a copy.""" + + def test_the_canvas_is_adopted_not_copied( + self, build: Any, monkeypatch: pytest.MonkeyPatch + ) -> None: + adopted: list[np.ndarray] = [] + original = DemTile._adopt + + def spy(meta: Any, array: np.ndarray) -> DemTile: + adopted.append(array) + return original(meta, array) + + monkeypatch.setattr(DemTile, "_adopt", staticmethod(spy)) + result, _ = build(quadrants(whole(), row_cut=4, col_cut=6, overlap=1)) + assert len(adopted) == 1 + assert np.shares_memory(adopted[0], result.tile.array) + assert not result.tile.array.flags.writeable + + def test_adopt_keeps_the_buffer_and_makes_it_read_only(self) -> None: + buffer = values(3, 4) + tile = DemTile._adopt(meta(rows=3, cols=4), buffer) + assert np.shares_memory(tile.array, buffer) + assert not tile.array.flags.writeable + assert not buffer.flags.writeable + assert tile.meta == meta(rows=3, cols=4) + + @pytest.mark.parametrize( + "array", + [ + values(4, 3), + values(3, 4, np.int32), + values(3, 4).reshape(1, 3, 4), + np.asfortranarray(values(3, 4)), + ], + ids=["shape", "dtype", "ndim", "fortran-order"], + ) + def test_adopt_runs_the_same_checks(self, array: np.ndarray) -> None: + with pytest.raises(ValueError): + DemTile._adopt(meta(rows=3, cols=4), array) + + def test_the_public_constructor_still_copies(self) -> None: + buffer = values(3, 4) + tile = DemTile(meta=meta(rows=3, cols=4), array=buffer) + assert not np.shares_memory(tile.array, buffer) + assert buffer.flags.writeable + + def test_adopt_is_called_only_from_the_mosaic(self) -> None: + """R7: "Its one caller is `assemble`". Grep the package for the name.""" + users = sorted( + str(path.relative_to(SRC)) + for path in SRC.rglob("*.py") + if "_adopt" in path.read_text(encoding="utf-8") + ) + assert users == ["io/models.py", "mosaic.py"] + + +def test_the_plan_never_loads(mz: ModuleType, footprint: Any) -> None: + """I6 as a type fact: `plan_mosaic` takes footprints, and no `load`.""" + import inspect + + assert list(inspect.signature(mz.plan_mosaic).parameters) == ["footprints", "bounds", "needed"]