Repository navigation
Expand file tree
/
Copy pathflowdivide.py
More file actions
2072 lines (1927 loc) · 152 KB
/
Copy pathflowdivide.py
File metadata and controls
2072 lines (1927 loc) · 152 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
"""flowdivide.py -- the FlowDivide workflow from a flow-direction grid and the HydroBASINS Level-03
vectors to the partition, its views, the figures and the attributes, on one machine, step by step.
python flowdivide.py run south-america --out-root /data/flowdivide
python flowdivide.py run north-america --out-root /data/flowdivide --drop acc,aca,l3
python flowdivide.py run merit-global --out-root /data/flowdivide --steps fd1,fd2 --drop aca,l3
python flowdivide.py run south-america --out-root /data/flowdivide --steps fd3 --attributes shv,ord --capacity 2^31
python flowdivide.py run mygrid --dir /data/dir.tif --convention merit --group-vectors /data/basins.shp --out-root /data/flowdivide
python flowdivide.py list
Three grids are built in (the three of the paper): HydroSHEDS v2.0 at 1 arc-second over South
America and over North America, and MERIT Hydro at 3 arc-seconds over the whole globe (periodic in
longitude). Any other flow-direction grid runs with --dir and --convention, at any resolution and in any
coding: the codings of MERIT Hydro, ESRI, HydroSHEDS, TauDEM, GRASS and the degrees are known by name, and
any other is given as --convention custom --direction-codes E=1,SE=2,... with --sink-code, --nodata-code
and --mouth-code. The basin groups come from --group-vectors, or are made here along a Hilbert curve.
The inputs are the flow directions and, for the given groups, the Level-03 vectors; on a grid of your own the
upstream pixel count and area are computed from the directions (fd1.1) unless --acc and --aca give them. The
built-in grids also read their provider's upstream count and area (ACC and ACA), and MERIT Hydro the table of its
published basin ids. The chain, in the order of the paper:
FD1 fd1.0 recode -> fd1.4 the Level-03 vectors burned -> fd1.1 accumulation -> fd1.2 outlets
-> fd1.3 delineation -> fd2.1 the basin views (the automatic grouping reads one)
-> fd1.4 the basin groups: by Level-03 unit, and automatically along a Hilbert curve
-> fd1.5 the Pfafstetter cut and the regions, once per capacity
FD2 the group rasters on the fine grid; the group and region views at every factor, as rasters and coloured polygons
figures Figures 3, 4 and 5 of the manuscript (South America and North America; not for MERIT)
FD3 the attributes on one capacity: shv ldn hck lup lfp ord, or those asked for
The products are laid out as follows:
<out-root>/<dataset>/global/fineresolution/{dir,upg,upa,str,l3,bsn}/ the continental rasters
<out-root>/<dataset>/global/fineresolution/grp/ the basin groups on the fine grid
<out-root>/<dataset>/global/table/ outlets, basins, groups
<out-root>/<dataset>/global/coarseresolution/{bsn,grp}/, global/vector/ the views (GeoParquet, GeoPackage or both: --vector)
<out-root>/<dataset>/partitions/<capacity>/global/{table,preview,fineresolution,coarseresolution,vector}/
<out-root>/<dataset>/figures/, _logs/
Every step leaves a marker in _logs/ with its arguments, its wall time, the time of its own checks and
its peak memory, so a run that stopped is taken up where it stopped and a step is never repeated with
the same arguments; _logs/timings_<dataset>.csv gathers the times. Every raster written to disk is a
variable with a keep option (--drop): a dropped variable is moved into _tmp_/ beside it as soon as the last
step of the run that reads it is done (nothing is deleted, so dropping frees no disk space until _tmp_/ is
emptied by hand). The variables and the steps that read them:
dir the recoded flow directions read by everything; never dropped
acc upstream pixel count (upg) fd1.2, fd1.5 (the cut)
aca upstream area (upa) fd1.2, fd1.5 (the cut, for the areas of the pieces), fd3 hck
str channel mask (>= 1 km2, 10 on MERIT) fd1.5 (the cut), fd3 shv hck ord
l3 the Level-03 codes burned fd1.4
bsn the basin of every pixel fd1.4, fd1.5, fd2, figures
rgn the region of every pixel fd2, figures
pieces the piece rasters of the cut basins fd1.5 (the regions), fd2 (the piece views), figures
shv ldn hck lup lfp ord products of fd3 (lfp reads the member table of ldn, not its raster)
"""
import argparse
import contextlib
import glob
import hashlib
import math
import os
import platform
import re
import resource
import shutil
import subprocess
import sys
import time
import numpy as np
import pandas as pd
import rasterio
HERE = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, HERE)
import fd1_partition as fd1
import fd_tables
import fd2_views as fd2
import fd3_attributes as fd3
from fd1_partition import FlowDivideError, log
VERSION = fd_tables.VERSION # the package; pyproject.toml says the same (test_native_cache_provenance.py checks it)
RULES_VERSION = 2 # version 2 uses left-closed, right-open rectangles; a marker of another version is stale.
# Step signatures use this value, so a change of wording or timing reruns nothing.
REGIONS_RULES_VERSION = 2 # the rules of fd1.5.regions_final that are not its arguments (the order of the
# joins of merge=level2, partition_config.txt)
FD2_VECTOR_RULES_VERSION = 4 # the rules of the polygons of stage 2: the boundary lines carry their table row as the
# GeoPackage fid; the polygons from 4-connected cells as GDAL gives
# them, OGC-valid, the boundary lines from the 8-connected ones (fd2_views.VECTOR_RULE_TAG);
# a GeoPackage view carries the default style on color_id (fd2_views.embed_color_id_style);
# named in the arguments of the steps that write polygons, so that polygons made under other
# rules are made again. The automatic (Hilbert) grouping reads the basin views and so
# inherits the signature of fd2_basin_views: where it runs, it is made again as well, and a
# partition cut on the Hilbert groups with it, stage 3 included (the results do not change)
CHANNEL_RULES_VERSION = 2 # the test of a channel pixel in fd1.1: the threshold in the raster's unit and in float32
# against the float32 area, as in FD3 (a double threshold decides otherwise at
# 1.00000001 km2). In fd1.1's arguments, so that a mask of another test is made again,
# and asked of fd1.1's marker when fd3 runs alone
LFP_RULES_VERSION = 2 # the longest flow path's rules, in its step's signature only; other attributes keep
# their markers. On a periodic grid, ties between members' heads use the members
# laid out by their links, not their regions' windows.
FD3_RULES_VERSION = 7 # each child's outlet must flow into the table's inlet pixel, belonging to the parent;
# region windows must fit 2^31 - 1 pixels. Products under other rules are made again.
# The channel and area grids must agree over the whole raster within 0.01 pixel.
# the rules of stage 3 alone: the distance and the upstream flow length cover every basin
# however small, their tables still list the basins of min_basin_area_km2 and more; the
# basin tables are printed column by column, with the longitudes and
# latitudes transformed on a projected grid; the lfp outputs carry .done markers;
# the distance raster's nodata is -1, and two donors of equal area are
# separated by their place in the window; a break in the channel
# mask is answered for by following the flow, not by the first break and the members'
# rectangles, so a product made under other rules may have passed a break it should
# have stopped at
# The inputs of the three built-in grids come from environment variables, with no default path: a run of a
# built-in grid whose inputs are not set stops and names the variables (unset_inputs).
HYBAS_ROOT_DEFAULT = os.environ.get("FLOWDIVIDE_HYBAS", "")
# HydroSHEDS v2's own mosaics, <continent>_DIR_1s_v2r0.tif with the _ACC_ and _ACA_ mosaics beside it.
# FLOWDIVIDE_HYDROSHEDS names the directory.
HYDROSHEDS_RAW_DEFAULT = os.environ.get("FLOWDIVIDE_HYDROSHEDS", "")
MERIT_RAW_DEFAULT = os.environ.get("FLOWDIVIDE_MERIT", "")
# The built-in MERIT Hydro grid reads MERIT Hydro's own upstream count and area instead of computing them.
MERIT_ACC_DEFAULT = os.environ.get("FLOWDIVIDE_MERIT_ACC", "")
MERIT_ACA_DEFAULT = os.environ.get("FLOWDIVIDE_MERIT_ACA", "")
# The outlet table of the published MERIT Hydro basins: the built-in MERIT Hydro grid takes its basin ids from it,
# pixel by pixel. No sort of today's rasters gives them back: ordering one band of rows by MERIT's own Float32
# upstream area agrees with the published ids on 85.9% of neighbouring pairs, the others one or two Float32 steps
# apart, since the areas the published run sorted on differ from the distributed raster by about 1e-5.
MERIT_BASIN_IDS_DEFAULT = os.environ.get("FLOWDIVIDE_MERIT_BASIN_IDS", "")
ALL_ATTRIBUTES = ["shv", "ldn", "hck", "lup", "lfp", "ord"]
DROPPABLE = ["acc", "aca", "str", "l3", "bsn", "rgn", "pieces"] + ALL_ATTRIBUTES
RASTER_FILE_NAME = {"acc": "upg", "aca": "upa"} # the variables of the paper and their file names
PREVIEW_STEP = 30 # the piece preview of Figure 4: one cell per 30 x 30 pixels
GROUPINGS = ["l3", "hilbert"]
DISK_RESERVE_GB = 20.0
# what a step adds to the disk, in bytes per pixel of the grid, from the South America products at 1
# arc-second (the DIR 6.3 GB, the upstream count 22.7 GB and area about as much, the members under
# 1 GB, lup 44 GB, ldn 18 GB); a guard, not an accounting
DISK_BYTES_PER_PIXEL = {"fd1.0": 0.12, "fd1.1": 0.85, "fd1.2": 0.01, "fd1.3": 0.02, "fd1.4_level3": 0.01, "fd1.4_groups": 0.01, "fd1.5_cut": 0.02, "fd1.5_regions": 0.03,
"fd2": 0.03, "figures": 0.0, "fd3_shv": 0.02, "fd3_ldn": 0.35, "fd3_hck": 0.02, "fd3_lup": 0.80, "fd3_lfp": 0.01, "fd3_ord": 0.02}
METRES_PER_DEGREE = 40075017.0 / 360.0 # along the equator (WGS84)
def view_name_of_factor(grid, factor):
"""the name of a view whose cell is `factor` pixels wide, for a grid this package has no fixed names for:
the size of the cell, without a space and without a fraction, as a file name can carry it -- 100m, 1km,
30s (arc-seconds on a geographic grid). The name is only a name; the factor is what the view is made
with, so any grid and any resolution gets a name a reader understands."""
width = abs(grid.pixel_width) * factor
if grid.geographic:
seconds = width * 3600.0
if abs(seconds - round(seconds)) < 1e-6:
return "%ds" % int(round(seconds))
per_degree = 1.0 / width
if abs(per_degree - round(per_degree)) < 1e-6 * per_degree:
return "%dth" % int(round(per_degree))
return "%gdeg" % width
if width >= 1000.0:
return "%gkm" % (width / 1000.0)
return "%gm" % width
VIEW_FACTORS_DEFAULT = (10, 50, 250)
def footprint_pixels(dir_argument, probe):
"""the pixels of the rectangle the flow directions span: one file's, or that of all the tiles of a directory"""
if not os.path.isdir(dir_argument):
return probe.nrow * probe.ncol
left = bottom = float("inf")
right = top = float("-inf")
for path in sorted(glob.glob(os.path.join(dir_argument, "*.tif"))):
with fd1.open_raster(path) as tile:
left, bottom = min(left, tile.bounds.left), min(bottom, tile.bounds.bottom)
right, top = max(right, tile.bounds.right), max(top, tile.bounds.top)
return int(round((right - left) / abs(probe.pixel_width))) * int(round((top - bottom) / abs(probe.pixel_height)))
def view_min_basin_area_km2(grid, factor):
"""FD2's smallest basin of a view (Chain.min_basin_area_km2_of_view, before a chain is made): a quarter of a cell,
at the equator on a geographic grid, rounded as fd2.min_basin_area_km2_of rounds it"""
scale = METRES_PER_DEGREE if grid.geographic else 1.0
return fd2.min_basin_area_km2_of(abs(grid.pixel_width) * factor * scale / 1000.0, abs(grid.pixel_height) * factor * scale / 1000.0)
def largest_basin_area_km2(grid, pixels, slack=False):
"""all the grid's pixels at the area of its largest pixel (on a geographic grid the pixel at the equator, whether
the grid reaches it or not): no basin covers more. With slack, a bound no stored basin area passes either (FD1
keeps areas in whole square metres): a square metre a pixel more, and a thousandth"""
if grid.geographic:
pixel_m2 = fd1.pixel_area_m2(0.0, abs(grid.pixel_width), abs(grid.pixel_height), grid.earth_model)
else:
pixel_m2 = abs(grid.pixel_width) * abs(grid.pixel_height)
area_m2 = pixels * pixel_m2
if slack:
area_m2 = (area_m2 + pixels + 1.0) * 1.001
return area_m2 / 1e6
def views_by_cell_size(grid, factors=VIEW_FACTORS_DEFAULT, pixels=None, prune=True):
"""the views of a grid this package has no fixed names for: coarse grids, each a whole number of pixels to a
cell and a divisor of the block, so that a block holds whole cells; named by their cell size. With prune (the
default views), a view whose smallest basin (FD2's threshold) is larger than any basin of the grid could be
(pixels: all its tiles together) is left out: no basin would be in it and the run would stop there, so no grid
that ran before loses a view. The finest is kept, made finer until its threshold is below that area. Views
given with --view-factors (prune=False) are neither left out nor made finer; each is still lowered to a divisor
of the block (250 is 240 on a block of 3600 pixels), and two that come out the same are kept once"""
pixels = grid.nrow * grid.ncol if pixels is None else pixels
largest = largest_basin_area_km2(grid, pixels, slack=True)
covered = largest_basin_area_km2(grid, pixels)
chosen = []
for wanted in factors:
divisors = [f for f in range(1, max(wanted, 1) + 1) if grid.block_pixels % f == 0]
factor = max(divisors, default=1)
if prune and view_min_basin_area_km2(grid, factor) > largest:
if chosen:
continue
factor = max([f for f in divisors if view_min_basin_area_km2(grid, f) < covered], default=1)
name = view_name_of_factor(grid, factor)
if factor not in [f for _, f in chosen] and name not in [n for n, _ in chosen]:
chosen.append((name, factor))
return chosen or [(view_name_of_factor(grid, 1), 1)]
def resolution_text(pixel_width, geographic, factor=1):
"""the size of a pixel as a reader says it: the 1 and 3 arc-second grids by their usual names, 30 m
and 90 m (not 1/3600 degree); any other geographic cell as the
fraction of a degree with the metres along the equator, 1/400° (280 m); a projected grid in metres.
The factor is that of a view, whose cell is factor pixels wide"""
width = abs(pixel_width) * factor
if not geographic:
return "%g m" % width
seconds = width * 3600.0
if abs(seconds - round(seconds)) < 1e-6 and int(round(seconds)) in (1, 3):
return "%d m" % int(round(seconds * METRES_PER_DEGREE / 3600.0 / 10.0) * 10)
per_degree = 1.0 / width
fraction = "1/%d\u00b0" % int(round(per_degree)) if abs(per_degree - round(per_degree)) < 1e-6 * per_degree else "%g\u00b0" % width
metres = width * METRES_PER_DEGREE
if metres >= 10000:
about = "%d km" % int(round(metres / 1000.0))
elif metres >= 1000:
about = "%.1f km" % (metres / 1000.0)
elif metres >= 20:
about = "%d m" % int(round(metres / 10.0) * 10)
elif metres >= 1:
about = "%d m" % int(round(metres))
else:
about = "%.1f m" % metres
return "%s (%s)" % (fraction, about)
# =============================================================================
# [1] The datasets
# =============================================================================
class Dataset:
"""everything the chain needs to know about one grid"""
def __init__(self, name, raw_dir, convention, group_layers=(), continent=None, level1=None, periodic=False, block_pixels=None, capacities=(), merge_policy="none",
land_cut_above=None, land_merge_below=0, island_reach_blocks=6, island_max_blocks=36, cut_cap_pixels=2 ** 31 - 1, views=(), hilbert=True,
figures=False, figure_basin_id=1, figure_basin_name="", figure_view=None, figure_ticks=None, figure3_size=(7.48, 4.3), figure3_stacked=False,
figure5_size=(7.48, 4.3), vectorise_basins=True, acc_path=None, aca_path=None, aca_unit="m2",
direction_codes=None, sink_code=None, nodata_code=None, mouth_code=None, level3_layers=None,
earth_model=fd1.EARTH_MODEL_WGS84_ZONE, basin_order="area-then-count", basin_id_table=None,
aca_single_pixel_tolerance=1.0e-3, channel_threshold_km2=1.0):
self.name = name
self.continent = continent or name # the word in every file name
self.level1 = level1 # the level-1 number of the island and seam groups (6 South America, 7 North America, 0 the whole grid); None: from the codes
self.acc_path = acc_path # a provider's upstream count and area given instead of fd1.1's own (None: computed)
self.aca_path = aca_path
self.aca_unit = aca_unit if aca_path else "m2" # fd1.1's own area is in square metres; a given one in its unit (km2 for HydroSHEDS and MERIT)
self.raw_dir = raw_dir # one file, or a directory of tiles
self.convention = convention # a name of fd1.DIRECTION_CONVENTIONS, or "custom" with direction_codes
self.direction_codes = direction_codes # "E=1,SE=2,..." of a coding this package does not know by name
self.sink_code = sink_code # the producer's value of an inland sink, no data and a river mouth,
self.nodata_code = nodata_code # where they differ from the named coding's (None: the coding's own)
self.mouth_code = mouth_code
# the vectors that give the basin groups: HydroBASINS Level-03 for the three grids of the paper, any
# polygons of whole basins for a grid of one's own (level3_layers, another name for it, is accepted too)
self.group_layers = list(group_layers if group_layers else (level3_layers or []))
self.periodic = periodic
# which pixel area the products of this grid are published with: the exact area on the WGS84
# ellipsoid, or MERIT Hydro's own on the MERIT 90 m grid, so that the upstream area is MERIT Hydro's
# pixel for pixel. It goes into every step's arguments, so changing it makes
# the steps that computed an area run again.
if earth_model not in fd1.EARTH_MODELS:
raise ValueError("unknown earth model '%s': it must be one of %s" % (earth_model, ", ".join(fd1.EARTH_MODELS)))
self.earth_model = earth_model
# how basins are numbered (fd1.BASIN_ORDER_RULES): the built-in MERIT Hydro grid keeps the published ids
if basin_order not in fd1.BASIN_ORDER_RULES:
raise ValueError("unknown basin order '%s': it must be one of %s" % (basin_order, ", ".join(fd1.BASIN_ORDER_RULES)))
if basin_order == "from-a-published-table" and basin_id_table is None:
raise ValueError("the basin order 'from-a-published-table' needs basin_id_table")
self.basin_order = basin_order
self.basin_id_table = basin_id_table # the table whose ids are carried over, with that rule
# fd1.2: how far the upstream area of a one-pixel basin may be off the pixel area of its row
# (0.1 %, and 0.5 % on MERIT, whose provider has an area formula of its own)
self.aca_single_pixel_tolerance = aca_single_pixel_tolerance
# the upstream area a channel pixel has at least on this grid, when --channel-threshold-km2 does not
# say otherwise: 1 km2 on the HydroSHEDS 30 m grids, 10 km2 on MERIT 90 m
self.channel_threshold_km2 = channel_threshold_km2
self.block_pixels = block_pixels # None: one degree on a geographic grid
self.capacities = list(capacities) # [(name, blocks, cut levels), ...], the largest first
self.merge_policy = merge_policy
self.land_cut_above = land_cut_above # None: the capacity itself (no cut for the land)
self.land_merge_below = land_merge_below
self.island_reach_blocks = island_reach_blocks
self.island_max_blocks = island_max_blocks
self.cut_cap_pixels = cut_cap_pixels # a basin of more pixels needs a cut (fd1.2 counts them)
self.views = list(views) # [(name, factor), ...]
self.hilbert = hilbert # the automatic grouping as well (Figure 3c)
self.figures = figures
self.figure_basin_id = figure_basin_id
self.figure_basin_name = figure_basin_name
self.figure_view = figure_view # lon min, lon max, lat min, lat max
self.figure_ticks = figure_ticks # (longitude ticks, latitude ticks)
self.figure3_size = figure3_size
self.figure3_stacked = figure3_stacked
self.figure5_size = figure5_size
self.vectorise_basins = vectorise_basins
@property
def capacity_names(self):
return [name for name, _, _ in self.capacities]
@property
def aca_to_km2(self):
return 1.0 if self.aca_unit == "km2" else 1e-6
@property
def has_given_groups(self):
"""are the basin groups given as vectors (the l3 grouping), or made here along a Hilbert curve?"""
return len(self.group_layers) > 0
# other names for the two, still accepted
level3_layers = property(lambda self: self.group_layers)
has_level3 = property(lambda self: self.has_given_groups)
def earth_in_the_arguments(dataset):
"""" earth=<model>" when this grid is not on the default earth model, an empty string when it is: a
step's arguments decide whether its marker still holds, and the grids published under the default must
keep theirs"""
return "" if dataset.earth_model == fd1.EARTH_MODEL_WGS84_ZONE else " earth=%s" % dataset.earth_model
def order_in_the_arguments(dataset):
"""the same for the basin numbering rule, with the table's identity when the ids are carried over"""
if dataset.basin_order == "area-then-count":
return ""
if dataset.basin_order == "from-a-published-table":
return " order=%s ids=%s" % (dataset.basin_order, file_identity(dataset.basin_id_table))
return " order=%s" % dataset.basin_order
def hybas(root, code):
return os.path.join(root, "hybas_%s_lev03_v1c.shp" % code)
def builtin_datasets(hybas_root=HYBAS_ROOT_DEFAULT, hydrosheds_raw=HYDROSHEDS_RAW_DEFAULT, merit_raw=MERIT_RAW_DEFAULT):
"""the three grids of the paper, with the settings of the paper. A capacity is named by a
power of two and realised as a round number of whole blocks below it: at 1 arc-second 2^31 pixels is
165.7 blocks of one degree and the partition uses 160 (2^30: 80, 2^29: 40); on the 3 arc-second grid
the tiers are 1,400, 700 and 350 blocks (round numbers under 2^31, 2^30 and 2^29). The Pfafstetter
cut goes three levels deep at the two larger capacities and four at the smallest. The 30 m grids
keep the land balance of the paper (a group of whole basins is also cut when it holds more than
7.5e8 pixels of land, and a part under 1.5e8 is merged back); the 90 m grid looks at the window only
and takes the Level-02 units as its regions (merge=level2); its basins are not vectorised and the
automatic grouping is not made."""
return {
"south-america": Dataset(
"south-america", os.path.join(hydrosheds_raw, "south-america_DIR_1s_v2r0.tif"), "hydrosheds", [(hybas(hybas_root, "sa"), None)], level1=6,
# the provider's own upstream count and area: what a product uses is
# the number that was downloaded, on both lines, so that the two are one standard. fd1.1 then
# makes the channel mask only.
acc_path=os.path.join(hydrosheds_raw, "south-america_ACC_1s_v2r0.tif"),
aca_path=os.path.join(hydrosheds_raw, "south-america_ACA_1s_v2r0.tif"), aca_unit="km2",
capacities=[("2^31", 160, 3), ("2^30", 80, 3), ("2^29", 40, 4)], merge_policy="none", land_cut_above=750000000, land_merge_below=150000000,
island_reach_blocks=6, island_max_blocks=36, cut_cap_pixels=1000000000, views=[("400th", 9), ("200th", 18), ("120th", 30), ("20th", 180)], hilbert=True,
figures=True, figure_basin_id=1, figure_basin_name="Amazon", figure_view=(-93.0, -32.0, -57.0, 15.0), figure_ticks=([-90, -75, -60, -45], [-45, -30, -15, 0]),
figure3_size=(7.48, 4.24), figure3_stacked=False, figure5_size=(7.48, 4.3), vectorise_basins=True),
"north-america": Dataset(
"north-america", os.path.join(hydrosheds_raw, "north-america_DIR_1s_v2r0.tif"), "hydrosheds",
[(hybas(hybas_root, "na"), None), (hybas(hybas_root, "ar"), "PFAF_ID != 353")], level1=7,
acc_path=os.path.join(hydrosheds_raw, "north-america_ACC_1s_v2r0.tif"),
aca_path=os.path.join(hydrosheds_raw, "north-america_ACA_1s_v2r0.tif"), aca_unit="km2",
capacities=[("2^31", 160, 3), ("2^30", 80, 3), ("2^29", 40, 4)], merge_policy="none", land_cut_above=750000000, land_merge_below=150000000,
island_reach_blocks=6, island_max_blocks=36, cut_cap_pixels=1000000000, views=[("400th", 9), ("200th", 18), ("120th", 30), ("20th", 180)], hilbert=True,
figures=True, figure_basin_id=1, figure_basin_name="Mississippi", figure_view=(-170.0, -52.0, 7.0, 84.0), figure_ticks=([-160, -140, -120, -100, -80, -60], [15, 30, 45, 60, 75]),
figure3_size=(7.48, 13.8), figure3_stacked=True, figure5_size=(7.48, 5.0), vectorise_basins=True),
"merit-global": Dataset(
"merit-global", merit_raw, "merit", [(hybas(hybas_root, code), None) for code in ("af", "ar", "as", "au", "eu", "gr", "na", "sa", "si")], continent="global", level1=0,
periodic=True, capacities=[("1400deg2", 1400, 3), ("700deg2", 700, 3), ("350deg2", 350, 4)], aca_single_pixel_tolerance=5.0e-3, merge_policy="level2", land_cut_above=None, land_merge_below=0,
# the upstream count and the upstream area are MERIT Hydro's own rasters, read as they are, from
# fd1.1 on. So this line's basin areas are the published ones to the bit, and fd1.1 only makes
# the channel mask. MERIT's pixel area is CaMa-Flood's rgetara, 2.3e-3 smaller than the exact
# figure on the WGS84 ellipsoid. The areas of the pieces and the members are differences of
# that same upstream area (the outlet's less the children's), so they are on MERIT's footing
# too and add up to the basin exactly; the exact ellipsoid is what this code integrates pixel
# areas with -- the region windows and the land share.
acc_path=MERIT_ACC_DEFAULT, aca_path=MERIT_ACA_DEFAULT, aca_unit="km2",
# Basin ids come from the published outlet table (see fd1.BASIN_ORDER_RULES).
basin_order="from-a-published-table", basin_id_table=MERIT_BASIN_IDS_DEFAULT,
island_reach_blocks=18, island_max_blocks=1296, cut_cap_pixels=2 ** 31 - 1, views=[("400th", 3), ("200th", 6), ("120th", 10), ("20th", 60)], hilbert=False,
figures=False, vectorise_basins=False,
# a river pixel of this grid has at least 10 km2 upstream, the fd3 default here
channel_threshold_km2=10.0),
}
def unset_inputs(name, steps, hybas_root, acc_given=False, attributes=()):
"""the inputs a run of the built-in grid `name` needs whose environment variable (or option) is not set;
there is no default path. The flow directions are read by every run, the upstream area by FD1 and by FD3
when it computes the Hack order (hck), the other inputs by FD1 only"""
unset = []
if name in ("south-america", "north-america"):
if not HYDROSHEDS_RAW_DEFAULT:
unset.append("FLOWDIVIDE_HYDROSHEDS (the directory of HydroSHEDS v2's DIR, ACC and ACA mosaics)")
else:
if not MERIT_RAW_DEFAULT:
unset.append("FLOWDIVIDE_MERIT (MERIT Hydro's flow directions, one file)")
if "fd1" in steps and not MERIT_ACC_DEFAULT and not acc_given:
unset.append("FLOWDIVIDE_MERIT_ACC (MERIT Hydro's upstream pixel count; or --acc)")
if ("fd1" in steps or ("fd3" in steps and "hck" in attributes)) and not MERIT_ACA_DEFAULT:
unset.append("FLOWDIVIDE_MERIT_ACA (MERIT Hydro's upstream area, read by FD1 and by FD3's hck)")
if "fd1" in steps and not MERIT_BASIN_IDS_DEFAULT:
unset.append("FLOWDIVIDE_MERIT_BASIN_IDS (the outlet table whose basin ids are carried over)")
if "fd1" in steps and not hybas_root:
unset.append("FLOWDIVIDE_HYBAS (the directory of the HydroBASINS Level-03 shapefiles; or --hybas-root)")
return unset
def file_identity(path):
"""what identifies an input file's content well enough for the markers: its size and its modification
time in nanoseconds (a shapefile: those of its .shp and .dbf)"""
paths = [path]
if path.lower().endswith(".shp"):
paths += [path[:-4] + extension for extension in (".dbf", ".shx", ".prj")]
return ";".join("%d:%d" % (os.path.getsize(p), os.stat(p).st_mtime_ns) if os.path.exists(p) else "absent" for p in paths)
def capacity_blocks_of(name, block_pixels):
"""a capacity written as 2^k, or as a whole number of blocks (160, 160deg2, 1400blocks): the number of whole blocks:
the top tier, 2^31, is (2^31 - 1) / block^2 rounded down to two significant
digits (160 blocks of 3600 pixels, 1400 of 1200), and every other tier that halved or doubled"""
if name.startswith("2^"):
power = int(name[2:])
if not 20 <= power <= 40:
raise FlowDivideError("the capacity must be 2^k with k from 20 to 40, got %s" % name)
top = fd1.top_tier_blocks(block_pixels)
blocks = top >> (31 - power) if power <= 31 else top << (power - 31)
if blocks < 1:
raise FlowDivideError("%s is less than one block of %d x %d pixels" % (name, block_pixels, block_pixels))
return blocks
digits = name.replace("deg2", "").replace("blocks", "")
if not digits.isdigit() or int(digits) <= 0:
raise FlowDivideError("the capacity must be 2^k or a whole number of blocks such as 160deg2, got %s" % name)
return int(digits)
# =============================================================================
# [2] The layout of the output directory: the directories and file names
# =============================================================================
class Layout:
"""where every product of one dataset lives under the output root"""
def __init__(self, root, dataset, grid=None):
self.root = os.path.join(root, dataset.name)
self.dataset = dataset
self.name = dataset.continent
self.grid = grid
self.tag = None # the resolution tag of the file names (1s, 3s, 10m), known with the grid
self.external = {} # a variable given from outside (acc, aca): its path, never written or dropped here
for directory in ("global/fineresolution/dir", "global/table", "global/coarseresolution/bsn", "global/coarseresolution/grp", "global/vector", "_logs", "figures"):
fd1.ensure_directory(os.path.join(self.root, directory))
def set_grid(self, grid):
self.grid = grid
if grid.geographic:
seconds = abs(grid.pixel_width) * 3600.0
self.tag = "%ds" % int(round(seconds)) if abs(seconds - round(seconds)) < 1e-6 else "%gs" % seconds
else:
self.tag = "%gm" % abs(grid.pixel_width)
# ---- the continental products ----
def raster(self, variable):
"""the file of a variable of stage 1: dir_<c>_<tag>_merit.tif, upg_, upa_, str_, l3_, bsn_; or the file
given from outside for acc and aca"""
if variable in self.external:
return self.external[variable]
if variable == "dir":
return os.path.join(self.root, "global", "fineresolution", "dir", "dir_%s_%s_merit.tif" % (self.name, self.tag))
file_name = RASTER_FILE_NAME.get(variable, variable)
fd1.ensure_directory(os.path.join(self.root, "global", "fineresolution", file_name))
return os.path.join(self.root, "global", "fineresolution", file_name, "%s_%s_%s.tif" % (file_name, self.name, self.tag))
def dir_path_of(self, grid_of_native):
"""the recoded flow directions are named before the grid is known: from the native file"""
self.set_grid(grid_of_native)
return self.raster("dir")
def table(self, name):
return os.path.join(self.root, "global", "table", "%s_%s.csv" % (name, self.name))
def basin_table(self):
"""the one basin table of the run"""
return fd_tables.basin_table_path(self.root, self.name)
def group_table(self, grouping):
return self.table("group_%s_fine" % grouping)
def basin_group_table(self, grouping):
return self.table("basin_group_%s_fine" % grouping)
def basin_view(self, view_name):
return os.path.join(self.root, "global", "coarseresolution", "bsn", "bsn_%s_%s.tif" % (self.name, view_name))
def group_view(self, grouping, view_name):
return os.path.join(self.root, "global", "coarseresolution", "grp", "grp_%s_%s_%s.tif" % (grouping, self.name, view_name))
def group_raster(self, grouping):
directory = os.path.join(self.root, "global", "fineresolution", "grp")
fd1.ensure_directory(directory)
return os.path.join(directory, "grp_%s_%s_%s.tif" % (grouping, self.name, self.tag))
def vector(self, stem, view_name):
"""bsn_, bnd_, grp_<g>_ polygons of one view: the path without its extension (.parquet, .gpkg: --vector)"""
return os.path.join(self.root, "global", "vector", "%s_%s_%s" % (stem, self.name, view_name))
@staticmethod
def vector_files(stem_path, formats):
"""the files a vector stem is written as, one per format"""
return [stem_path + (".parquet" if name == "geoparquet" else ".gpkg") for name in formats]
# ---- the partitions ----
def capacity_directory_name(self, blocks, grouping="l3"):
"""<N>deg2 on a geographic grid with blocks of one degree, else <N>blocks; the block is part of the name
(_block<B>) unless it is one degree on a geographic grid or 5000 pixels on a projected one, since two block
sizes can give the same number of blocks but not the same windows"""
grid = self.grid
block_is_one_degree = grid is not None and grid.geographic and abs(grid.block_pixels * abs(grid.pixel_width) - 1.0) < 1e-6
text = "%ddeg2" % blocks if block_is_one_degree else "%dblocks" % blocks
if grid is not None and not block_is_one_degree and not (not grid.geographic and grid.block_pixels == 5000):
text += "_block%d" % grid.block_pixels
return text if grouping == "l3" else "hilbert_%s" % text
def partition(self, blocks, grouping="l3"):
directory = os.path.join(self.root, "partitions", self.capacity_directory_name(blocks, grouping))
for sub in ("global/table", "global/preview", "global/fineresolution", "global/coarseresolution/rgn", "global/vector"):
fd1.ensure_directory(os.path.join(directory, sub))
return directory
def partition_table(self, blocks, grouping, name):
return os.path.join(self.partition(blocks, grouping), "global", "table", "%s_%s.csv" % (name, self.name))
def partition_raster(self, blocks, grouping, variable):
fd1.ensure_directory(os.path.join(self.partition(blocks, grouping), "global", "fineresolution", variable))
return os.path.join(self.partition(blocks, grouping), "global", "fineresolution", variable, "%s_%s_%s.tif" % (variable, self.name, self.tag))
def region_view(self, blocks, grouping, view_name, kind="tif"):
"""the region view raster, or (kind "vector") the path of its polygons without the extension"""
if kind == "tif":
return os.path.join(self.partition(blocks, grouping), "global", "coarseresolution", "rgn", "rgn_%s_%s.tif" % (self.name, view_name))
return os.path.join(self.partition(blocks, grouping), "global", "vector", "rgn_%s_%s" % (self.name, view_name))
def cut_listing(self, blocks, grouping):
return os.path.join(self.partition(blocks, grouping), "global", "table", "cut_basins_%s.txt" % self.name)
def piece_table(self, blocks, grouping, basin_id):
return os.path.join(self.partition(blocks, grouping), "global", "table", "pfafstetter_pieces_%s_basin%d.csv" % (self.name, basin_id))
def mainstem_table(self, blocks, grouping, basin_id):
return os.path.join(self.partition(blocks, grouping), "global", "table", "pfafstetter_mainstem_%s_basin%d.csv" % (self.name, basin_id))
def piece_codes(self, blocks, grouping, basin_id):
return os.path.join(self.partition(blocks, grouping), "global", "table", "pfafstetter_codes_%s_basin%d.csv" % (self.name, basin_id))
def piece_raster(self, blocks, grouping, basin_id, view_name=None):
return os.path.join(self.partition(blocks, grouping), "global", "preview", "pfafstetter_pieces_%s_basin%d_%s.tif" % (self.name, basin_id, view_name or self.tag))
def attribute(self, code, blocks, grouping="l3"):
return self.partition_raster(blocks, grouping, code)
def attribute_table(self, code, blocks, grouping="l3"):
return self.partition_table(blocks, grouping, "%s_basin" % fd3.ATTRIBUTES[code]["table"])
def attribute_member_table(self, code, blocks, grouping="l3"):
return self.partition_table(blocks, grouping, "%s_member" % fd3.ATTRIBUTES[code]["table"])
def piece_preview(self, blocks, grouping, basin_id):
"""the preview of the pieces of a cut basin, one cell per 30 x 30 pixels of its rectangle, the pixel at the
corner of each cell"""
return os.path.join(self.partition(blocks, grouping), "global", "preview", "pfafstetter_pieces_%s_basin%d_%s.tif" % (self.name, basin_id, self.preview_tag()))
def preview_tag(self):
grid = self.grid
if grid.geographic:
seconds = abs(grid.pixel_width) * 3600.0 * PREVIEW_STEP
return "%ds" % int(round(seconds)) if abs(seconds - round(seconds)) < 1e-6 else "%gs" % seconds
return "%gm" % (abs(grid.pixel_width) * PREVIEW_STEP)
def cut_basins(self, blocks, grouping):
"""the basins a finished cut step listed"""
listing = self.cut_listing(blocks, grouping)
if not os.path.exists(listing):
return []
with open(listing) as handle:
return [int(line) for line in handle.read().split()]
def cuts(self, blocks, grouping):
"""{basin id: (piece table, piece raster)} of a finished cut step"""
return {basin_id: (self.piece_table(blocks, grouping, basin_id), self.piece_raster(blocks, grouping, basin_id)) for basin_id in self.cut_basins(blocks, grouping)}
def files_of_variable(self, variable, partitions):
"""every file on disk that belongs to a variable (for the keep option); partitions: [(blocks, grouping)]"""
if variable in self.external:
return [] # a file given from outside is never deleted
if variable in ("acc", "aca", "str", "l3", "bsn", "dir"):
return [self.raster(variable)]
if variable == "rgn":
return [self.partition_raster(b, g, variable) for b, g in partitions]
if variable == "pieces":
files = []
for b, g in partitions:
files += [raster for _, raster in self.cuts(b, g).values()]
return files
if variable in ALL_ATTRIBUTES:
return [self.attribute(variable, b, g) for b, g in partitions]
return []
# =============================================================================
# [3] The chain: one step after another, with markers, timing and the keep option
# =============================================================================
def run_in_child_process(function, args):
"""one step run in a forked child process, so that its memory goes back to the system when it ends
and its own peak memory is measured; the result, the peak and the time of the step's own checks
come back through a pipe, an error is raised again in the parent"""
import multiprocessing
import traceback
context = multiprocessing.get_context("fork")
receiver, sender = context.Pipe(duplex=False)
def child():
try:
fd1.reset_timing()
result = function(*args)
peak = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss * (1 if platform.system() == "Darwin" else 1024)
sender.send(("ok", result, peak, fd1.CHECK_SECONDS[0], dict(fd1.IO_SECONDS)))
except BaseException as error:
sender.send(("error", "%s: %s\n%s" % (type(error).__name__, error, traceback.format_exc()), 0, 0.0, {"input": 0.0, "output": 0.0}))
finally:
sender.close()
process = context.Process(target=child)
process.start()
sender.close()
try:
status, payload, peak, check_seconds, io_seconds = receiver.recv()
except EOFError:
process.join()
raise FlowDivideError("the step's process ended without a result (exit code %s); killed for memory?" % process.exitcode)
process.join()
if status != "ok":
raise FlowDivideError("the step failed in its process:\n%s" % payload)
return payload, peak, check_seconds, io_seconds
class Chain:
"""runs the steps of one dataset in order, keeps a marker per step, drops the variables that are
not kept once nothing left in the run reads them"""
def __init__(self, dataset, root, steps, attributes, capacity_name, drop, tile, min_basin_area_km2, channel_threshold_km2, open_figures, grouping="l3",
levels=None, vector_formats=("geoparquet",), colours=fd2.COLOURS_AT_LEAST, separate_processes=True):
self.dataset = dataset
self.separate_processes = separate_processes
self.vector_formats = tuple(vector_formats) # geoparquet, gpkg, or both (--vector)
self.colours = colours # at least this many colour numbers (--colours)
self.steps = steps
self.attributes = attributes
self.drop = set(drop)
self.tile = tile
self.min_basin_area_km2 = min_basin_area_km2
self.channel_threshold_km2 = channel_threshold_km2
self.open_figures = open_figures
self.grouping = grouping # the grouping the partition is built on: l3 or hilbert
self.levels = levels # None: the depth of each capacity from the dataset; "auto" or 2..4 for every capacity
self.layout = Layout(root, dataset)
self.summary = os.path.join(self.layout.root, "_logs", "chain_summary.txt")
self.summary_only = False # --summary-only: the page and the plan, nothing run, nothing logged
self.only = None # --only: the labels of the steps allowed to execute (None: every planned step)
self.held_back = [] # the planned steps --only held back, reported once
self.timing = True # --timing on|off: the split of every step's time recorded, or only its total
self.gdal_cache_mb = None # --gdal-cache-mb: GDAL's block cache in fd3, MiB (None: FLOWDIVIDE_GDAL_CACHE_MB or 512)
self.done = set()
self.steps_run = 0 # the steps this run executed (the timings table is made when it has)
self.grid = None
self.capacity_name = capacity_name
self.blocks_of = {name: blocks for name, blocks, _ in dataset.capacities}
self.levels_of = {name: levels for name, _, levels in dataset.capacities}
if capacity_name not in self.blocks_of:
raise FlowDivideError("the capacity %s is not one of the dataset's: %s" % (capacity_name, ", ".join(self.blocks_of)))
if grouping == "hilbert" and not dataset.hilbert:
raise FlowDivideError("the automatic grouping is not made for this dataset; the partition can only follow the Level-03 groups")
if grouping == "l3" and not dataset.has_given_groups:
raise FlowDivideError("no group vectors were given (--group-vectors): the partition can only follow the automatic groups (--groups hilbert)")
for variable, path in (("acc", dataset.acc_path), ("aca", dataset.aca_path)):
if path is not None:
if not os.path.exists(path):
# the count is read by FD1 only, the area by FD1 and by FD3 when it computes the Hack order;
# any other run works from the products already on disk, and must not be stopped because the
# drive holding the provider's mosaics is not mounted
reads_it = "fd1" in self.steps or (variable == "aca" and "fd3" in self.steps
and "hck" in self.attributes)
if reads_it:
raise FlowDivideError("the %s raster given, %s, is not there" % (variable, path))
log("flowdivide", "the %s raster %s is not there; no step of this run reads it" % (variable, path))
continue
self.layout.external[variable] = path
if (dataset.acc_path is None) != (dataset.aca_path is None):
raise FlowDivideError("the upstream count and the upstream area are given together or not at all")
# ---- markers and the plan ----
def marker(self, label):
return os.path.join(self.layout.root, "_logs", label.replace("/", "_") + ".ok")
def marker_signature(self, label):
"""the signature a finished step recorded in its marker, or None"""
path = self.marker(label)
if not os.path.exists(path):
return None
with open(path) as handle:
for line in handle.read().splitlines():
if line.startswith("signature: "):
return line[len("signature: "):]
return None
def marker_arguments(self, label):
"""the arguments a finished step recorded in its marker ("args: ..."), or None"""
path = self.marker(label)
if not os.path.exists(path):
return None
with open(path) as handle:
for line in handle.read().splitlines():
if line.startswith("args: "):
return line[len("args: "):]
return None
def marker_channel_threshold(self, label):
"""the channel threshold a finished fd1.1 recorded in its arguments ("threshold=..."), or None"""
path = self.marker(label)
if not os.path.exists(path):
return None
with open(path) as handle:
for line in handle.read().splitlines():
if line.startswith("args: "):
found = re.search(r"threshold=([^ ]+)", line)
return float(found.group(1)) if found else None
return None
def signature(self, step, steps_by_label, cache):
"""what identifies a run of the step: its own arguments and the signatures of the steps it depends
on, so that a change of a parameter upstream (the channel threshold, the merge policy) changes the
signature of everything after it. A dependency that is not in this run's list contributes the
signature its marker recorded, or "missing" when it never ran."""
if step.label in cache:
return cache[step.label]
parts = ["flowdivide rules %d" % RULES_VERSION, step.arguments]
# an input from outside the run's own tree (the provider's mosaics, the Level-03 vectors) enters with its
# size and its time to the nanosecond, so that a mosaic replaced in place makes the steps that read it run again
# (without it, an ACA replaced in place leaves the Hack order of the old one standing)
parts += self.external_identities(step)
for label in step.depends_on:
if label in steps_by_label:
parts.append(self.signature(steps_by_label[label], steps_by_label, cache))
else:
# a step not in this run is taken from its marker only while the outside inputs it recorded are as
# they were; an ACA replaced in place since fd1.1 leaves the channel mask of the old one, and fd3
# alone would read it under a new signature
changed = self.marker_external_that_changed(label)
if changed:
raise FlowDivideError("%s was made from %s, which has changed since; run that step again (add its stage "
"to --steps)" % (label, changed))
parts.append(self.marker_signature(label) or "missing")
cache[step.label] = hashlib.sha1("|".join(parts).encode("utf8")).hexdigest()[:16]
return cache[step.label]
def external_identities(self, step):
"""an input from outside the run's own tree (the provider's mosaics, the Level-03 vectors), as "input
<path> <bytes> <time in ns>", so that a mosaic replaced in place makes the steps that read it run again
(without it, an ACA replaced in place leaves the Hack order of the old one standing). The provider's
mosaics the run was given (acc, aca) enter every step, since what a step reads was made from them upstream even
when the step does not read them itself"""
root = os.path.abspath(self.layout.root) + os.sep
external_paths = [path for path in getattr(step, "inputs", None) or [] if path]
external_paths += [path for path in getattr(self.layout, "external", {}).values() if path]
native = getattr(self.dataset, "raw_dir", None)
native_paths = set()
if native:
native_paths.add(os.fspath(native))
if os.path.isdir(native):
native_paths.update(glob.glob(os.path.join(native, "*.tif")))
external_paths.extend(native_paths)
identities = []
for path in sorted(set(external_paths)):
if os.path.exists(path) and (path in native_paths or not os.path.abspath(path).startswith(root)):
status = os.stat(path)
identities.append("input %s %d %d" % (os.path.abspath(path), status.st_size, status.st_mtime_ns))
return identities
def native_input_identity(self):
"""The native DIR identity in recode arguments, including the tile list."""
native = self.dataset.raw_dir
if not os.path.exists(native):
return None
paths = sorted(glob.glob(os.path.join(native, "*.tif"))) if os.path.isdir(native) else [native]
if not paths:
return None
return ";".join("%s:%s" % (os.path.basename(path), file_identity(path)) for path in paths)
def marker_native_that_changed(self):
"""Older recode markers kept the native identity in args, rather than external lines."""
arguments = self.marker_arguments("fd1.0")
if arguments is None:
return None
current = self.native_input_identity()
if current is None or not arguments.endswith(" input=" + current):
return self.dataset.raw_dir
return None
def marker_external_that_changed(self, label):
"""the first outside input a finished step's marker recorded ("external: input <path> <bytes>
<ns>") that is no longer of that size and time, or None; a marker without such lines says nothing"""
path = self.marker(label)
if not os.path.exists(path):
return None
native_changed = self.marker_native_that_changed()
if native_changed:
return native_changed
with open(path) as handle:
for line in handle.read().splitlines():
if not line.startswith("external: input "):
continue
recorded_path, recorded_bytes, recorded_ns = line[len("external: input "):].rsplit(" ", 2)
if not os.path.exists(recorded_path):
return recorded_path
status = os.stat(recorded_path)
if status.st_size != int(recorded_bytes) or status.st_mtime_ns != int(recorded_ns):
return recorded_path
return None
def is_done(self, step):
return self.marker_signature(step.label) == step.signature
@staticmethod
def product_paths(step):
"""the paths of a step's products: its dict maps a variable to one path or to a list of paths"""
paths = []
for value in (getattr(step, "products", None) or {}).values():
paths.extend(value if isinstance(value, (list, tuple)) else [value])
return [path for path in paths if path]
def an_output_lost_its_marker(self, step):
"""an output that is empty, or that carried a .done marker when the step finished and carries none now
(a table left half written or its marker taken away by a failed rerun), is not taken as done. An output the
step names as one that may be empty (the list of the basins cut, where none had to be) is taken as it is"""
for path in list(step.outputs) + self.product_paths(step):
if os.path.exists(path) and os.path.isfile(path) and os.path.getsize(path) == 0 \
and path not in getattr(step, "may_be_empty", ()):
return True
path = self.marker(step.label)
if not os.path.exists(path):
return False
with open(path) as handle:
for line in handle.read().splitlines():
if line.startswith("done_marked: "):
return any(marked and not os.path.exists(marked + ".done")
for marked in line[len("done_marked: "):].split("|"))
# a marker without that line does not say which outputs had .done markers; the step is made again once
# so that its marker says it
return True
def note(self, text):
with open(self.summary, "a") as handle:
handle.write(text + "\n")
print(text, flush=True)
class Step:
"""one step of the chain as it is planned: its label, the arguments that identify a run of it,
the function, the files it must leave (outputs), the variables it makes (products: name ->
file or files), the files it needs (inputs), the steps it depends on, and the kind of step for
the disk estimate"""
def __init__(self, label, arguments, function, args, outputs=(), products=None, inputs=(), depends_on=(), kind=None,
refresh=None, may_be_empty=()):
self.label = label
self.arguments = arguments
self.function = function
self.args = args
self.outputs = list(outputs)
self.products = products or {}
self.inputs = list(inputs)
self.depends_on = list(depends_on) # the labels of the steps whose results this one reads
self.kind = kind or label
self.signature = None
self.will_execute = False
self.refresh = refresh # () -> (outputs, products), after the step ran
self.may_be_empty = set(may_be_empty) # outputs that are whole when they are empty
def plan(self, steps):
"""Which steps will actually execute: a step runs when its marker is missing or carries another
signature, when one of its outputs is missing, or when a product of it is missing that a step
executing later needs, or that was not dropped. Settled from the end backwards until nothing
changes, so that a step made necessary by a later one is caught."""
executing = {}
# the steps planned earlier in this run (the recode) count as part of this plan: their signatures and
# whether they executed carry over
steps_by_label = dict(getattr(self, "planned", {}))
steps_by_label.update({step.label: step for step in steps})
for label, step in getattr(self, "planned", {}).items():
executing[label] = step.will_execute
cache = {}
for step in steps:
step.signature = self.signature(step, steps_by_label, cache)
executing[step.label] = (not self.is_done(step)) or any(not os.path.exists(path) for path in step.outputs) \
or self.an_output_lost_its_marker(step)
if not executing[step.label] and step.label == "figures":
with open(step.outputs[0]) as handle:
executing[step.label] = any(not os.path.exists(path) for path in handle.read().split("\n") if path)
changed = True
while changed:
changed = False
# a step that executes changes its results for the steps after it: they execute too
for step in steps:
if not executing[step.label] and any(executing.get(label, False) for label in step.depends_on):
executing[step.label] = True
changed = True
for step in steps:
if executing[step.label]:
continue
for variable, paths in step.products.items():
for path in ([paths] if isinstance(paths, str) else paths):
if os.path.exists(path):
continue
needed = variable not in self.drop or any(executing.get(label, False) for label in self.consumers(variable))
if needed:
executing[step.label] = True
changed = True
break
if executing[step.label]:
break
for step in steps:
step.will_execute = executing[step.label]
# --only: run just the steps named, whatever else the plan holds. What it is for: a step whose
# marker or product was lost invalidates every step after it, so a change in one late step would
# drag the whole chain along, where only the products that change need to be made again. A step
# still refuses to run when a file it reads is not on disk, so this cannot make a product out of
# nothing; what it can do is leave the tree with old and new parts in it, so the log says plainly
# which planned steps were held back.
if self.only is not None:
held_back_here = []
for step in steps:
if step.will_execute and step.label not in self.only:
step.will_execute = False
held_back_here.append(step.label)
if held_back_here:
self.held_back.extend(held_back_here)
self.note("--only holds back %d planned steps: %s" % (len(held_back_here), ", ".join(held_back_here)))
# A step held back leaves its own products as they are, which is the point; what must not
# happen is a selected step running on products that are out of date and then recording a
# marker made with the signature of the step that did not run -- the next run would then
# find everything in order. So every step a selected step depends on,
# directly or through others, must carry the signature this run works out for it.
for step in steps:
if not step.will_execute:
continue
# the step it rests on directly is enough: a marker's signature is made of that step's
# arguments and the signatures of the steps IT rests on, so a marker that still matches
# says the whole chain above it is as it was. A product that is gone rather than out of
# date is caught by execute(), which refuses a step whose inputs are not on disk.
for label in step.depends_on:
if label in self.only:
continue
other = steps_by_label.get(label)
if other is None:
continue
recorded = self.marker_signature(label)
if recorded != other.signature:
raise FlowDivideError(
"--only would run %s while %s, which it rests on, is out of date (%s); run %s as well, "
"or leave --only out" % (step.label, label,
"it has never run" if recorded is None else "its arguments or its own inputs have changed",
label))
planned = dict(getattr(self, "planned", {}))
planned.update({step.label: step for step in steps})
self.planned = planned
return steps