Skip to content

feat: the Weather COUNT runs while a staging maintains a copy - #88

Merged
krzysztof-smartdataengines merged 1 commit into
mainfrom
feat/count-during-staging
Sep 25, 2026
Merged

krzysztof-smartdataengines merged 1 commit into
mainfrom
feat/count-during-staging

Conversation

@krzysztof-smartdataengines

@krzysztof-smartdataengines krzysztof-smartdataengines commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The Weather starter's COUNT executor now runs while a staging maintains a copy. A staging keeps the
source authoritative, and the controller now carries saved queries through a staging, so the
approved COUNT stays current on the staging's map. The executor still refused any map with a copy
("Execute the count demo before staging or after completed cutover"), which left a current COUNT
that could be neither exported nor run until the cutover. The controller's installed acceptance of
queries surviving operations found this: its export-count-query after each staging was refused by
this rule, not because the COUNT was out of date.

What changed

  • sde_demo.query_count._source names the source on a map that also maintains a copy. A query
    written for the copy is still refused ("does not describe this project's current source and
    map") until the cutover makes the copy the source.
  • Relaxing the rule alone was not enough, and the new live test shows why: run_count_query opened
    its session with the source's adapter only, and sde.Session refuses a map that names an engine
    it has no adapter for (EngineError: the placement map refers to engines that were not supplied: ['clickhouse']). The executor now connects every engine the local map names, as the application
    does during a staging, counts on the source's adapter only, and closes every adapter even when one
    cannot close (ExitStack, one callback per adapter).
  • docs/weather-starter.md: the rule, and the second connection during a staging.

Tests

  • test_the_count_runs_on_the_source_of_a_real_staging[postgres|clickhouse] (live): setup, a run,
    a real staging through the starter's own operator stage with a signed packet, a second run
    writing through both copies, then the COUNT for the staging's map counts all 8 rows. The same test
    makes one adapter's close fail and checks that the failure surfaces and both adapters were
    closed. It was red before the fix with the EngineError above.
  • test_a_count_reads_the_source_while_a_staging_maintains_a_copy[postgres|clickhouse] and
    test_a_query_written_for_the_maintained_copy_is_refused: the metadata path on the staging's map,
    loaded from a signed staging packet. They were red on the old rule with its message.
  • make check locally on this commit with both live engines: ruff and mypy clean, Python 2015
    passed and 10 skipped (the orderbook slice, which needs the engine's C++ library), TypeScript 953
    passed.

Mutations on the committed tree, with both live engines: six killed - the old rule restored, the copy's
materialization accepted as the source, a session without the copy's engine, the count on another
adapter, adapters never closed, a cleanup that stops at the first failing close - and both controls
(two comments) survived.

The controller side, private, adds a test that export-count-query after a completed staging
exports the carried revision bound to the staging's map. It was red on main with the message the
installed acceptance recorded.

🤖 Generated with Claude Code

A staging keeps the source authoritative, and the controller now carries saved queries through a
staging, so the approved COUNT stays current on the staging's map. The starter's executor still
refused any map with a copy ("Execute the count demo before staging or after completed cutover"),
so a current COUNT could be neither exported nor run until the cutover. The controller's installed
acceptance of queries surviving operations found it: its export after each staging was refused by
this rule, not because the COUNT was out of date.

The executor now names the source on such a map too; a query written for the copy stays refused
until the cutover makes the copy the source. Relaxing the rule alone was not enough: the local run
opened its session with the source's adapter only, and a session refuses a map naming an engine it
has no adapter for. It now connects every engine the map names, as the application does during a
staging, counts on the source's adapter, and closes every adapter even when one cannot close.

Tests: a real staging through the starter's own operator on both engines, a second run writing
through both copies, and the COUNT for the staging's map counting every row (red before the fix
with the session's EngineError); the same test with a failing close; the metadata path on a map
loaded from a signed staging packet, and a query for the copy refused.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@krzysztof-smartdataengines
krzysztof-smartdataengines merged commit 0bc50ea into main Sep 25, 2026
11 checks passed
@krzysztof-smartdataengines
krzysztof-smartdataengines deleted the feat/count-during-staging branch September 25, 2026 00:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant