docs: in-place index builds measured with the shipped operator

docs/in-place-index.md

@@ -11,7 +11,9 @@ under new names ([staging](staging.md) protocol 2) and a [cutover](local-cutover
 compares every row while the source's writes are frozen. That pause grows with the table - measured
 at 22.5 s for 300 000 rows on PostgreSQL, and a table of about 400 000 rows passes the cutover's
 30 s budget and rolls back. The in-place build of the same index took 355 ms, and the longest gap
-between two writes during it was 16 ms. Measurements and scripts:
+between two writes during it was 16 ms. Measured again with the shipped operator: at 450 000 rows
+on PostgreSQL the copy path's cutover ran into the 30 s watchdog and was rolled back, while the
+in-place build was published in 618 ms. Measurements and scripts:
 [qualification/in-place-index](qualification/in-place-index/README.md).
 
 Only adding indexes can be done in place. A new key order, a partition, other tables or another

docs/qualification/in-place-index/README.md

@@ -25,13 +25,37 @@ and `premise/premise-run.log` (100 000 and 300 000 rows).
 
 The pause through the copy is linear in the table - about 75 µs a row on PostgreSQL - because the
 cutover copies and compares every row while the source's writes are frozen. At about 400 000 rows
-it passes the cutover's 30 s budget, and the cutover rolls back: an "add an index" decision could
+on PostgreSQL it passes the cutover's 30 s budget, and the cutover rolls back: an "add an index" decision could
 not be carried out for a table of that size at all. The in-place build moves no row; the longest
 gap between two writes during it is of the order of the gaps without it. No write failed in any run.
 
 The script's usage lines were rewritten when it moved into this directory; its code is the code
 that produced these files. `premise/SHA256SUMS` covers every file here.
 
+## After: the shipped operator against the copy path
+
+`after/measure_after.py` runs what shipped in PR #83 (`f77bb7b`): `LocalCutover.index` on a signed
+`sde-index` authorization - snapshot, native build, qualification from the catalogue, watermark,
+atomic publication - against the copy path on the same table, with a process on the map in force
+writing a row every 5 ms throughout. Same laptop, same engines, load average 1.0-1.4 at the start
+and 2.6 at the end. Raw results: `after/after.json`, `after/after-run.log`.
+
+| engine | rows | in place: receipt / operator wall | longest gap between writes during the build (before it) | copy path: write pause | copy path: outcome |
+|---|---|---|---|---|---|
+| PostgreSQL | 100 000 | 346 ms / 482 ms | 13.8 ms (9.6) | 7 174 ms | success |
+| PostgreSQL | 450 000 | 618 ms / 764 ms | 80.2 ms (9.2) | interrupted by the operator's watchdog at 30 066 ms | recovery required; the resume aborted (`recovered_before_decision`), 257 ms |
+| ClickHouse | 100 000 | 657 ms / 885 ms | 20.0 ms (27.3) | 5 581 ms | success |
+| ClickHouse | 450 000 | 707 ms / 920 ms | 22.1 ms (21.2) | 27 843 ms | success, 2.2 s inside the 30 s budget |
+
+The table the copy path cannot finish is there: at 450 000 rows on PostgreSQL the cutover's pause
+ran into the operator's 30 s watchdog, the source stayed frozen until a recovery aborted the
+cutover, and the index the model decided was not in force - while the in-place build of the same
+index was built and published in 618 ms. On ClickHouse the same table still fitted, by 2.2 s. No
+write failed in any in-place build. The gap that stands out, 80 ms at 450 000 rows on PostgreSQL,
+is the longest interval between two successful writes during that build; the script records only
+the maximum, so how often a gap of that size occurred is not known, and this record does not
+explain it.
+
 ## Engine behaviours the rules are written against
 
 `premise/probe_engines.py` -> `premise/probe_engines.out.json`:

docs/qualification/in-place-index/after/SHA256SUMS

@@ -0,0 +1,5 @@
+087ef78ee4bc97abafa614f8930f1721f6cc8dcf554c72921f44a8af97e4aa45  measure_after.py
+8f2299c2c296b68f8af70b20fff36d3196c259e4df36e422d7010304a509b921  after.json
+9b54ae3bec5c50a18e5c35c9411e6ac7440968b7679e0dff916afc4e6cf2126a  after-run.log
+886f1840e5822f594fa1b0fd2cef0d47d1798d6299f7cf85d281f4570a680000  load-before.txt
+faca8f3df3b557fc706ce971c2b413993bf184913a5e30149f5c4ebc4ec231d4  load-after.txt

docs/qualification/in-place-index/after/after-run.log

@@ -0,0 +1,9 @@
+{"path": "in_place", "dialect": "postgres", "rows": 100000, "outcome": "built", "receipt_elapsed_ms": 346, "operator_wall_ms": 482, "writer_max_gap_before_ms": 9.6, "writer_max_gap_during_ms": 13.8, "writer_errors": 0, "index_in_force": true}
+{"path": "copy", "dialect": "postgres", "rows": 100000, "stage_ms": 401, "stage_outcome": "prepared", "cutover_wall_ms": 7243, "cutover_outcome": "success", "cutover_reason": "matched", "within_budget": true, "pause_ms": 7174, "recovery": null, "index_in_force": true}
+{"path": "in_place", "dialect": "postgres", "rows": 450000, "outcome": "built", "receipt_elapsed_ms": 618, "operator_wall_ms": 764, "writer_max_gap_before_ms": 9.2, "writer_max_gap_during_ms": 80.2, "writer_errors": 0, "index_in_force": true}
+{"path": "copy", "dialect": "postgres", "rows": 450000, "stage_ms": 392, "stage_outcome": "prepared", "cutover_wall_ms": 30397, "cutover_outcome": "recovery_required", "cutover_reason": "operator deadline interrupted an operation; reconnect dedicated adapters and inspect local status before resuming or retrying qualification", "within_budget": null, "pause_ms": null, "recovery": {"interrupted_after_ms": 30066, "resume_ms": 257, "resumed_outcome": "abort", "resumed_reason": "recovered_before_decision"}, "index_in_force": false}
+{"path": "in_place", "dialect": "clickhouse", "rows": 100000, "outcome": "built", "receipt_elapsed_ms": 657, "operator_wall_ms": 885, "writer_max_gap_before_ms": 27.3, "writer_max_gap_during_ms": 20.0, "writer_errors": 0, "index_in_force": true}
+{"path": "copy", "dialect": "clickhouse", "rows": 100000, "stage_ms": 641, "stage_outcome": "prepared", "cutover_wall_ms": 5782, "cutover_outcome": "success", "cutover_reason": "matched", "within_budget": true, "pause_ms": 5581, "recovery": null, "index_in_force": true}
+{"path": "in_place", "dialect": "clickhouse", "rows": 450000, "outcome": "built", "receipt_elapsed_ms": 707, "operator_wall_ms": 920, "writer_max_gap_before_ms": 21.2, "writer_max_gap_during_ms": 22.1, "writer_errors": 0, "index_in_force": true}
+{"path": "copy", "dialect": "clickhouse", "rows": 450000, "stage_ms": 652, "stage_outcome": "prepared", "cutover_wall_ms": 28054, "cutover_outcome": "success", "cutover_reason": "matched", "within_budget": true, "pause_ms": 27843, "recovery": null, "index_in_force": true}
+EXIT=0

docs/qualification/in-place-index/after/after.json

@@ -0,0 +1,119 @@
+{
+  "machine": {
+    "node": "km-ThinkPad-E570",
+    "processor": "x86_64",
+    "python": "3.12.3"
+  },
+  "sde_version": "0.1.0.dev0",
+  "results": [
+    {
+      "path": "in_place",
+      "dialect": "postgres",
+      "rows": 100000,
+      "outcome": "built",
+      "receipt_elapsed_ms": 346,
+      "operator_wall_ms": 482,
+      "writer_max_gap_before_ms": 9.6,
+      "writer_max_gap_during_ms": 13.8,
+      "writer_errors": 0,
+      "index_in_force": true
+    },
+    {
+      "path": "copy",
+      "dialect": "postgres",
+      "rows": 100000,
+      "stage_ms": 401,
+      "stage_outcome": "prepared",
+      "cutover_wall_ms": 7243,
+      "cutover_outcome": "success",
+      "cutover_reason": "matched",
+      "within_budget": true,
+      "pause_ms": 7174,
+      "recovery": null,
+      "index_in_force": true
+    },
+    {
+      "path": "in_place",
+      "dialect": "postgres",
+      "rows": 450000,
+      "outcome": "built",
+      "receipt_elapsed_ms": 618,
+      "operator_wall_ms": 764,
+      "writer_max_gap_before_ms": 9.2,
+      "writer_max_gap_during_ms": 80.2,
+      "writer_errors": 0,
+      "index_in_force": true
+    },
+    {
+      "path": "copy",
+      "dialect": "postgres",
+      "rows": 450000,
+      "stage_ms": 392,
+      "stage_outcome": "prepared",
+      "cutover_wall_ms": 30397,
+      "cutover_outcome": "recovery_required",
+      "cutover_reason": "operator deadline interrupted an operation; reconnect dedicated adapters and inspect local status before resuming or retrying qualification",
+      "within_budget": null,
+      "pause_ms": null,
+      "recovery": {
+        "interrupted_after_ms": 30066,
+        "resume_ms": 257,
+        "resumed_outcome": "abort",
+        "resumed_reason": "recovered_before_decision"
+      },
+      "index_in_force": false
+    },
+    {
+      "path": "in_place",
+      "dialect": "clickhouse",
+      "rows": 100000,
+      "outcome": "built",
+      "receipt_elapsed_ms": 657,
+      "operator_wall_ms": 885,
+      "writer_max_gap_before_ms": 27.3,
+      "writer_max_gap_during_ms": 20.0,
+      "writer_errors": 0,
+      "index_in_force": true
+    },
+    {
+      "path": "copy",
+      "dialect": "clickhouse",
+      "rows": 100000,
+      "stage_ms": 641,
+      "stage_outcome": "prepared",
+      "cutover_wall_ms": 5782,
+      "cutover_outcome": "success",
+      "cutover_reason": "matched",
+      "within_budget": true,
+      "pause_ms": 5581,
+      "recovery": null,
+      "index_in_force": true
+    },
+    {
+      "path": "in_place",
+      "dialect": "clickhouse",
+      "rows": 450000,
+      "outcome": "built",
+      "receipt_elapsed_ms": 707,
+      "operator_wall_ms": 920,
+      "writer_max_gap_before_ms": 21.2,
+      "writer_max_gap_during_ms": 22.1,
+      "writer_errors": 0,
+      "index_in_force": true
+    },
+    {
+      "path": "copy",
+      "dialect": "clickhouse",
+      "rows": 450000,
+      "stage_ms": 652,
+      "stage_outcome": "prepared",
+      "cutover_wall_ms": 28054,
+      "cutover_outcome": "success",
+      "cutover_reason": "matched",
+      "within_budget": true,
+      "pause_ms": 27843,
+      "recovery": null,
+      "index_in_force": true
+    }
+  ]
+}

docs/qualification/in-place-index/after/load-after.txt

@@ -0,0 +1 @@
+ 09:41:38 up 1 day, 13:32,  1 user,  load average: 2,60, 1,60, 1,53

docs/qualification/in-place-index/after/load-before.txt

@@ -0,0 +1 @@
+ 09:39:04 up 1 day, 13:29,  1 user,  load average: 1,01, 1,24, 1,44