Increment 15a: a DEM in several tiles (Norway, one projected CRS) - #105
Merged
Merged
Conversation
… size, CRS, roadmap) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…on CRS (the basin) Redesigns the parked increment 15 (f8cedbe) for Ola's two archives, with Norway first per Ola's priority ruling of 2026-09-27: 15a (many tiles, one projected CRS) and 15b (the domain in its own CRS) ship on their own; 15c (geographic DEMs, the lattice frame, --out-crs) and 15d (window decoding, basin memory) follow. Measured on the DTM10 archive (8 of 254 tiles half a cell off the main lattice; 51-node overlaps equal) and on ANADEM by range reads (EPSG:4326, 0.970 arc-second lattice shared by all tiles, seams equal, bilinear resampling up to 13 m off the source). Probe scripts under docs/increments/15-probes/. Ten questions for Ola. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nt 15 row and Ola's order (Norway first) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The failing suites for 15a (docs/increments/15-dem-mosaic.md, "Tests for @tester"), with Ola's rulings Q1-Q5: - test_mosaic.py (invariant-critical): planning and assembly from RasterMeta and DemTile, a dict as the repository. M1-M15: lattice grouping and the half-cell refusal (Q5), alignment tolerance, bounds snapping, the cap from physical memory, order independence, the reference node, coverage, the overlap rule (Q1: agree accepted, disagree refused with names, count, largest difference; one ulp is a disagreement), mixed NoData (Q4), split and re-stitch, and _adopt without a copy. - test_io_repository.py: F1-F4, header-only and cached footprints, and Q3's guard that no io/ module but repository.py opens a file. - test_io_read_meta.py: G1-G2. - test_dem_input.py: DemRequest/open_dem, and real DTM10 extracts. - test_cli_mesh_mosaic.py: C1-C6, --bbox (Q2), T-real on the benchmark tile. Fixtures: tests/fixtures/dtm10/{seam,lattices}, cut from Ola's DTM10 archive (one release, N5) by tests/fixtures/dtm10/extract.py: 6400_4 | 6400_1 (51 agreeing columns) and 7707_1 over 7707_2 (5 m apart), 0.6 MB. Every module is imported in a fixture, so each test fails on its own and collection is unaffected. Run green against a scratch implementation that is not committed; test_mosaic.py killed 39 of 39 of its mutants. Two corrections to R4 (selection by index window, coverage per node) and the open names are in the new "Pinned by the red suite (15a)" subsection. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The production code for the red commit 2696bc2, per docs/increments/15-dem-mosaic.md (R1-R7, R11, Q1-Q5, and "Pinned by the red suite (15a)"). No test or fixture is touched; no C++. - tin_engine/mosaic.py: Bounds, IndexWindow, TilePlacement, MosaicPlan, Mosaic, MosaicError. plan_mosaic groups footprints into lattices (same EPSG, spacing, registration, NoData, integer offsets within 1e-6 cells), counts global indices from each lattice's north-west node, selects by index window, refuses a request on two lattices naming a tile of each and the offset (Q5), then the 2 x 2 minimum, the cap (4 bytes a node against half of physical_memory()), and per-node coverage. assemble merges tiles one at a time: valid beats NoData, sentinel beats NaN, two different valid values refuse naming both tiles, the count and the largest difference (Q1); one tile whose grid is the mosaic's is returned as loaded. - io/repository.py: TileFootprint, DemRepository, TiffDemRepository (the one io/ module that opens files, "rb" only; Q3). Headers read once, lazily. - io/geotiff.py: read_meta and decode_dem share _header; no duplication. - io/models.py: DemTile._adopt, the no-copy constructor for the canvas. - dem_input.py: DemRequest, DemInput, open_dem. - cli.py: --dem repeatable (one directory or files), --bbox with metavar BOX, refusals as usage errors, elevation_source prefix and stderr line for N >= 2, dem_tiles for a directory or several files. - Docstrings of io/__init__.py (Q3) and cli.py (no longer the only module with a path) corrected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
check_citations.py --base origin/master listed four citations into files ff7cc8d edits. Re-read as quotations: 12-dem-to-mesh.md's geotiff.py:169 (the codecs-extra advice) is now :198 and :307 (_representable) is now :336; 15-dem-mosaic.md R3's geotiff.py:70-87 (decode_dem before page.asarray()) is now _header itself, :96-131. io/models.py:82 still quotes the copy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # ROADMAP.md
…cap message, edge snap, project_structure; the basin note's spacing and scale errors corrected Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…le named, float64 cap, edge snap A test amendment after green (docs/increments/README.md, "Steps 2 and 3 are not strictly once each"): @Reviewer's CHANGES REQUESTED on 15a, and Ola's reading of Q5 after review, land after ff7cc8d, so the tests that encode them come after it, in their own commit. No production code. - B1: per lattice, whether its own tiles cover every node the request needs; exactly one -> plan on it, the other lattices' tiles dropped; none -> the Q5 refusal; several -> most tiles in the repository, ties by first tile name, order-independent. Synthetic (test_mosaic.py), real extracts (test_dem_input.py, CI runs them), CLI, and the design's acceptance box on Ola's archive (headers only, skipped when the archive is absent). The bare two-lattice request is still refused: the "Pinned" section says why. - B2: the overlap refusal names the two tiles whose values disagree. - S2: TileFootprint.dtype from the header via PROMOTION; the cap counts np.result_type of the selected tiles; float64 boundary beside the float32 one. - S3: 0.1 m spacing, edges 0.3/0.7/0.9 add no node line (passes at HEAD; fails with _snap as identity). - S1: refusing a mostly uncovered window peaks below its float32 canvas (tracemalloc; 164 MB against 16 MB at HEAD). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… Q5 reading Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ile named, float64-aware cap, cheap coverage refusal Makes the 24 tests of 1e45970 pass; no test or fixture touched. - B1: when the request selects tiles on several lattices, each is asked whether its own tiles cover every node it needs: the box snapped outward on it, clamped to the bounding box of every selected tile on any lattice. One covers: plan on it alone (a re-plan of its tiles, so the plan equals theirs). None: the Q5 refusal. Several: most tiles in the repository, ties by the lattice's first tile name. Header-only, order-independent. - B2: the overlap refusal names the merged tile whose value the incoming one disagrees with, re-loading candidates on the error path. - S2: TileFootprint.dtype (default float32), filled by the repository via the new geotiff.read_header (read_meta plus PROMOTION of the file dtype). The cap counts np.result_type of the selected footprints; the plan's placements carry the dtype, so assemble allocates the canvas once in it, keeping load-time promotion for footprints that under-state it. The false "headers do not give the decoded dtype" comment and BYTES_PER_NODE go. - S1: the coverage refusal counts with count_nonzero and bounds with any(axis=...); per-node index arrays only on the `needed` path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
geotiff.py:198 -> 207 and :336 -> 345 (12-dem-to-mesh.md), :96-131 -> 105-140 (15-dem-mosaic.md), each re-read as a quotation; and read_meta now reaches _header through read_header. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ured LOC; a paragraph re-wrapped Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… reported; N3 and R5 corrected Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…sed) Test amendment after green ff7cc8d, per docs/increments/README.md: Ola revised Q1 on 2026-09-28. Real DTM10 overlaps exported on different dates disagree (18 of 39 sampled pairs), so refusing them blocked much of the archive. The new rule: each overlap node takes the value of the tile it lies deepest in (distance in nodes to that tile's own nearest border), ties to the name that sorts first, NoData still loses to a valid value; nothing is refused; each disagreeing pair is reported in Mosaic.seams / DemInput.seams, the dem_seams file field and a "DEM seams" --stats section. New: TestQ1DeepestInterior and TestQ1SeamReport (test_mosaic.py), TestQ1Seams (test_cli_mesh_mosaic.py), the real seam meshing like its stitched single file, and an oracle in mosaic_fixtures.py that places tiles by coordinates. Changed, because they pinned the old refusal: - TestM8Overlaps: the refusal test with count and largest now asserts the report (plus the median and the value each node takes); B2's "names the disagreeing tile" now asserts the report names b | c and not a; one ulp and the sentinel case are reported instead of refused. - TestC4Refusals.test_an_overlap_disagreement is now TestQ1Seams.test_an_overlap_disagreement_meshes_and_is_recorded. - test_dem_input: the real seam shifted by one cell is reported and split, not refused; the agreeing real seam has no seams. - test_cli_mesh_mosaic TestRealDtm10: the agreeing seam records "none". Names and formats pinned in 15-dem-mosaic.md, "Pinned by the red suite (15a)"; I3 and the disagreement-message bullet corrected. Red at this commit: 38 tests, all in the three amended files. Green against an uncommitted scratch implementation; test_mosaic.py killed 16 of 16 mutants of it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Green for 0c97ac9. An overlap node takes the valid value of the tile it lies deepest in (min(r, c, rows-1-r, cols-1-c) in the whole tile's own indices), ties to the name that sorts first; NoData still loses to a valid value, and with none valid the sentinel wins over NaN in every order. The overlap-disagreement refusal and its message are gone. assemble writes each tile whole, keeps its values where it meets another placement as a strip, and drops the tile; once all are in, each pairwise overlap is decided from the strips (_decide) and each pair whose valid values differ is reported as a Seam (nodes, largest and median |a - b| in float64). Peak beyond the canvas and one loaded tile: the strips plus one overlap's temporaries (an int64 depth, a value and a mask of the overlap's size), never a canvas-sized array. One tile without bounds is returned as loaded, as before. Mosaic.seams and DemInput.seams carry the report; cli.py records dem_seams whenever it records dem_tiles ("none" when all agree), and --stats gets a "DEM seams" table between Sizes and Quality when some pair disagrees. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d probe (120 pairs; a same-date pair can disagree); R5 row, prior art, assembly prose; ROADMAP 15 row Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s escaped Ola, 2026-09-28: "Ignore below 1mm". A seam counts, and the report lists, only nodes with |a - b| >= 0.001 (float64, DEM units); nodes, largest and median are over those nodes only, and a pair with none is not listed. The midline decision does not look at the threshold. Separately (review suggestion), a `|` in a tile name is escaped `\|` in the --stats DEM seams table so the row keeps five cells. - New TestSeamThreshold (test_mosaic.py): exactly 1 mm counts and nextafter(0.001, 0) does not (float64 tiles, either sign); count/max/median over qualifying nodes only; a sub-mm-only pair is not listed; a sub-mm disagreement still resolves by depth then name, in every order. - Changed: TestM8Overlaps.test_one_ulp_is_a_disagreement is now test_one_ulp_is_below_the_threshold_and_still_decided_by_depth, since the ruling makes one ulp no seam; it keeps the value check. - The oracle seams_of (mosaic_fixtures.py) applies SEAM_THRESHOLD, so test_dem_input's shifted real seam now expects 12,787 nodes, not 12,800. - TestQ1Seams (CLI): a 0.5 mm disagreement records "none" with no stats section; a pipe in a tile name is escaped in the table, raw in dem_seams. Red at this commit: the 9 tests above that the ruling changes; the rest of tests/python is green. Test-only; no production code. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ped; assemble frees each tile before the next load - mosaic.py: SEAM_THRESHOLD = 0.001 (Ola, 2026-09-28: "Ignore below 1mm"). _seam counts only nodes where both tiles are valid and |a - b| >= 1 mm in float64; nodes, max and median are over those; a pair with none is not listed. _decide does not consult the threshold. - stats.py: _table escapes a `|` in a cell as `\|`; the dem_seams field (Seam.entry) keeps the name as listed. - assemble (review finding B4): the previous tile's `array`/`incoming` are dropped before the next load. tracemalloc peak on the acceptance box (--bbox 799750 7849750 900250 7950250, nine DTM10 tiles, 404 MB canvas, 102 MB tiles): 822 MB (canvas + 4.1 tiles) before, 720 MB (canvas + 3.1 tiles) after. One DTM10 load alone peaks at 2.8 tiles (decoder buffers plus DemTile's copy), so the docstring now states that, not "one tile". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…the per-load memory range Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ked superseded Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…the tip Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Increment 15a (
docs/increments/15-dem-mosaic.md). Python only, no C++; 597 production lines (CLAUDE.md§2, blank lines excluded per Ola's ruling in this PR).rasputin mesh --dem DIR(or several files) and--bbox: tiles selected by index window from their headers (io/repository.py, the oneio/module that opens files), grouped by lattice, stitched onto one node grid (mosaic.py,dem_input.py).read_metasplit out ofdecode_dem.15-probes/dtm10_dates.py; Kartverket: NDH laser data replacing the 2013 contour model). Each node takes the tile it lies deepest in, ties by name; seams differing by ≥ 1 mm are reported in--statsand adem_seamsfield.assemblefrees each tile before the next load (peak 720 MB on the acceptance box = canvas + 3.1 tiles).CLAUDE.md§2: blank lines do not count (Ola, 2026-09-28).docs/research/sao-francisco-basin.md).Acceptance box (9 tiles, 10,051² nodes): 11.05 M triangles in ~30 s, 4 seams reported.
Review
Red → green with test amendments after each ruling; @Reviewer through several rounds, approved at
201a4e7. Local: pytest 1979 passed, mypy, ruff, governance gates.🤖 Generated with Claude Code