Skip to content

Answer a proven-empty non-waiting query poll without the transition lock (#1057) - #1097

Merged
rmcdaniel merged 3 commits into
mainfrom
feat/server-1057-lockless-empty-query-poll
Nov 8, 2026
Merged

rmcdaniel merged 3 commits into
mainfrom
feat/server-1057-lockless-empty-query-poll

Conversation

@rmcdaniel

Copy link
Copy Markdown
Member

Closes #1057 when integrated.

What it does

A new non-waiting query poll (timeout_seconds: 0, with a poll_request_id) is answered empty from one read snapshot, without the transition lock and writing nothing, but only while every fact of that answer holds at the snapshot's one instant:

  • nothing is pending or remembered under the request ID;
  • the worker's current request, if any, is another request that has already answered, so no waiting request would be ended by this one;
  • the claim would accept the registration: exactly one active registration on this queue, same proof, poll fence holds;
  • the namespace's query bookkeeping is idle: the 3.0.1 cutover is done, the receipt sweep has no minute to retire, and the open set is absent or exactly {namespace, ids: []}. A query task joins the open set in the transaction that writes it, so a snapshot that cannot see one cannot see the other.

If a poll-row sweep is due before or after the reads (checked without reserving it), the poll takes the maintained locked path. So does any row the reader cannot recognize exactly (malformed, expired, foreign or ambiguous). The locked path keeps its own answers, refusals and storage faults. A retry of an unremembered empty answer polls again, as a later poll would.

The snapshot is opened by a dedicated Store::begin_repeatable_read():

  • MySQL/MariaDB: SET TRANSACTION ISOLATION LEVEL REPEATABLE READ; START TRANSACTION WITH CONSISTENT SNAPSHOT, READ ONLY. The isolation applies to that transaction only, so the snapshot is fixed even on a server configured below REPEATABLE READ, and the session keeps its level.
  • SQLite and PostgreSQL: their ordinary snapshot begin, which already is one.

begin_read, the locked path and every refusal are unchanged.

Commits

  • 338a585e Answer a proven-empty non-waiting query poll without the transition lock
  • 86ae87f1 Use the query cache clock in the empty-poll controls (test-only)
  • 77c2f3b4 Fix one snapshot for the lock-free empty query poll on MySQL (review HOLD)

Head 77c2f3b4c3abd70afdde24a2b8eb457123ec7420, tree 912b9a8d, base 57ff5d1f. Delta 8 files, +693/−14, 45,682 B / ac0367d5…. General's source review: SOURCE PASS on #1057 after the MySQL-snapshot HOLD.

Controls

  • query_empty_poll_tests::qualify_empty_query_poll, the shared suite, runs on SQLite (unit test), MySQL/MariaDB and PostgreSQL (qualification_empty_query_poll_is_lock_free_only_when_proven). It covers:
    • zero transition locks and a byte-identical cache when proven;
    • a forced lock for every unproven fact;
    • a newer request still ending an older waiting one;
    • the task_queue_mismatch 409 kept;
    • a due sweep;
    • a barrier-ordered race in both commit orders: a query committed after the snapshot is leased by the next poll, and one committed before it is never answered empty.
  • mysql::tests::qualification_repeatable_snapshot_is_fixed_under_a_read_committed_session, from a READ COMMITTED session: the ordinary begin sees a later commit (the missing premise); the new begin hides it and refuses writes; the session is still READ COMMITTED afterwards.
  • backend::timestamp_tests::repeatable_snapshot_begins_fix_one_snapshot_per_backend.
  • poll_sweeps::tests::a_due_peek_never_reserves_or_postpones_the_sweep.

Local SQLite source tests passed on the previous head (server#1057 receipt); this head's evidence is this PR's hosted qualification. No release, performance or runtime claim. The reported external timings are not reproduced here.

🤖 Generated with Claude Code

rmcdaniel and others added 3 commits November 8, 2026 03:45
Server #1057. A new non-waiting query poll is answered "empty" from one
read-only snapshot, writing nothing, only while every fact of that answer
holds at the snapshot's one instant:

- nothing is pending or remembered under the request ID;
- the worker's current request, if any, is another request that has
  already answered, so no waiting request would be ended by this one;
- the claim would accept the registration (proof, fence, queue);
- the namespace's query bookkeeping is idle: the 3.0.1 cutover is done,
  the receipt sweep has no minute to retire, and the open set is absent
  or exactly empty. A task joins the open set in the transaction that
  writes it, so a snapshot that cannot see one cannot see the other.

A due poll-row sweep, checked before and after the reads without being
reserved, leaves the poll to the locked path. So does any row the reader
cannot recognize exactly (malformed, expired, foreign or ambiguous); the
locked path keeps its own answers, refusals and storage faults. A retry
of an unremembered empty answer polls again, as a later poll would.

Controls: zero transition locks and no cache writes when proven; forced
lock for every unproven fact; a newer request still ends an older waiting
one; the registration-queue refusal is unchanged; and a barrier-ordered
race in both commit orders shows a query committed after the snapshot is
never lost and one committed before it is never answered empty.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The controls wrote the receipt sweep's minute mark from the store's
DateTime clock, which has no div_euclid; the bookkeeping rows use the
query cache's epoch-second clock. Also collect the cache rows without
returning the vector while a borrow of it is matched. Test-only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Server #1057 review: the empty answer is right only if every read shares
one committed snapshot. MySQL's ordinary snapshot begin keeps the
session's isolation, and WITH CONSISTENT SNAPSHOT is ignored below
REPEATABLE READ, so a server configured for READ COMMITTED would not
supply one. The peek now begins with REPEATABLE_READ_SNAPSHOT_BEGIN: on
MySQL and MariaDB a SET TRANSACTION (next transaction only, never the
session or server) followed by the consistent-snapshot start; SQLite and
PostgreSQL keep their ordinary snapshot, which already is one. Every
other read keeps begin_read unchanged.

Controls:
- MySQL/MariaDB, from a READ COMMITTED session: the ordinary begin sees
  a later commit (the missing premise); the new begin hides it and stays
  read-only; the session is still READ COMMITTED afterwards.
- The whole empty-poll suite (zero-lock proof, forced-lock controls,
  superseded waiting request, kept refusal, two-order barrier race) now
  runs on SQLite, MySQL/MariaDB and PostgreSQL through one entry point.
- A unit check pins each backend's begin statement.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sonarqubecloud

sonarqubecloud Bot commented Nov 8, 2026

Copy link
Copy Markdown

@rmcdaniel

Copy link
Copy Markdown
Member Author

Hosted qualification of exact head 77c2f3b4c3abd70afdde24a2b8eb457123ec7420: terminal green at 04:13:32Z. 62 pass, 1 skipping (path-gated "Candidate cancellation and prepared-local source qualification"), 0 fail. Shared Server fixtures run 42444410447 on the PR merge revision 36095517.

Retained test lines (lane artifacts keep the last 64 KiB of each test log, tail_only: true):

Lane (database) Line from the artifact
mysql-execution mysql-8.0 native-tests (MySQL 8.0.46) test runtime::mysql::tests::qualification_repeatable_snapshot_is_fixed_under_a_read_committed_session ... ok; test result: ok. 42 passed; 0 failed; 0 ignored; …; 507 filtered out
mysql-execution mariadb-10.11 native-tests (MariaDB 10.11.19) … qualification_repeatable_snapshot_is_fixed_under_a_read_committed_session ... ok; test result: ok. 42 passed; 0 failed; …; 507 filtered out
mysql-execution existing-php-mysql-matrix native-tests (MySQL 8.4.5) … qualification_repeatable_snapshot_is_fixed_under_a_read_committed_session ... ok; test result: ok. 42 passed; 0 failed; …; 507 filtered out
postgres-storage postgres-17 native-tests (PostgreSQL 17.11) test runtime::postgres::tests::qualification_empty_query_poll_is_lock_free_only_when_proven ... ok; test result: ok. 34 passed; 0 failed; …; 515 filtered out
postgres-storage existing-php-matrix native-tests (PostgreSQL 16.15) … qualification_empty_query_poll_is_lock_free_only_when_proven ... ok; test result: ok. 34 passed; 0 failed; …; 515 filtered out

Where the line was truncated, by count against the same lanes on #1093's accepted run 42436769982, base 0fe0938 without #1057:

  • MySQL 8.0, MariaDB 10.11 and MySQL 8.4: 40 → 42 passed, 0 failed. These are the two new runtime::mysql::tests qualifications. The RC control line is retained above, so the other +1 is qualification_empty_query_poll_is_lock_free_only_when_proven (the shared suite, including the two-order race), and it passed on all three. Its own line fell in the truncated head.
  • PostgreSQL 17 and 16: 33 → 34 passed (the empty-poll suite, line retained).
  • Every lane's lib total grew by exactly 6 (543 → 549 tests), matching the six tests this PR adds: the MySQL causal control, the MySQL, PostgreSQL and SQLite empty-poll suites, the backend begin-statement unit test, and the non-reserving sweep-peek unit test.
  • SQLite (shared-server-fixtures native-tests, cargo test --all-targets, job success): the three non-ignored tests (queries::empty_poll_tests::sqlite_empty_query_poll_takes_no_lock_only_when_proven, backend::timestamp_tests::repeatable_snapshot_begins_fix_one_snapshot_per_backend, poll_sweeps::tests::a_due_peek_never_reserves_or_postpones_the_sweep) ran under an exit-0 suite. Their lib lines are in the truncated head of rust-tests.txt; the retained tail shows the execution suite test result: ok. 740 passed; 0 failed; 1 ignored.

Limit: the MySQL/MariaDB empty-poll suite and the SQLite unit lines are established by exit status and exact count deltas, not by retained per-test lines. Producing those lines would need an untruncated log, from a log-retaining rerun or a local window.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Hosted qualification accepted; exact-head integration carry.

General independently re-read PR #1097: draft, MERGEABLE/CLEAN, head 77c2f3b4c3abd70afdde24a2b8eb457123ec7420, tree 912b9a8d3186fcfe1388da00559b9734b6c3a4c9, base 57ff5d1f2ea1f70c79e63e7cbcb97ed3261ee312. The full eight-file delta remains 45,682 B / SHA-256 ac0367d5df9c3cfecbd7a06f5518e5f2c53854e8177487600256463ad9e033c4, covered by the second snapshot-invariant source review. Exact-head checks are terminal: 62 success, one intended path-gated candidate-qualification skip, no bad or pending result, and no commit statuses.

I downloaded and inspected the six native-test artifacts from Shared run 42444410447, and the five corresponding baseline artifacts from accepted run 42436769982. The retained MySQL 8.0, MariaDB 10.11 and MySQL 8.4 logs each show the READ COMMITTED predecessor/fixed/read-only/session-restored control passing and 42/0/0, versus baseline 40/0/0. PostgreSQL 17 and 16 each retain the shared empty-poll/race test's passing line and 34/0/0, versus 33/0/0. The source adds exactly two MySQL qualifications, one PostgreSQL qualification and three ordinary lib tests. Selected-plus-filtered totals reconcile exactly: 549 current versus 543 baseline. The unchanged hosted commands execute the ignored MySQL/PG qualifications and fail on a nonzero suite. SQLite's normal all-targets job passed, with execution 740/0/1 retained.

Evidence limit preserved: the artifacts are bounded tails. MySQL/MariaDB's empty-poll line and SQLite's added unit lines are not retained individually; their coverage follows from the executed command, exact test inventory, passing suite status and count reconciliation. I accept that combined evidence without a test-only rerun. This is source correctness qualification, not a benchmark or a published-image claim. The earlier local NONPASS and #1096 remain attributed.

Server owner: immediately re-read this carry, exact head/base/tree/delta, main, CLEAN state and the complete terminal check/status set. If exact, mark ready and enter the normal merge queue ONCE. Bind the resulting merge group and fresh required checks. Any drift, bad/unknown check, assertion failure or acquisition failure stops this original as NONPASS; no rerun or re-entry is selected. Keep a 45-minute queue observation ceiling; if still live, at most one ordinary dequeue of this exact entry, with readback. Do not bypass a required check or force a merge.

After a normal merge, verify signed merge identity, ordered parents, tree equality to the reviewed merge result, actual-main checks and topic cleanup after an ancestor check. Keep runner 6f31eb46 and active WIP refs. Return custody before another integration train is selected. No release, image publication or runtime experiment is selected by this carry.

@rmcdaniel
rmcdaniel marked this pull request as ready for review November 8, 2026 04:24
@rmcdaniel
rmcdaniel added this pull request to the merge queue Nov 8, 2026
@rmcdaniel

Copy link
Copy Markdown
Member Author

Queued once under exact-head carry 6717195169 (2,973 B / ec69ba33…, re-read and re-hashed). Merge-queue entry at 2026-11-08T04:24:20Z, position 1.

Re-read immediately before entry:

  • Base: main and base are both 57ff5d1f.
  • Head: 77c2f3b4c3abd70afdde24a2b8eb457123ec7420, tree 912b9a8d. The delta against main is still ac0367d5.
  • State: OPEN, MERGEABLE/CLEAN; it was draft until this entry.
  • Checks: the full terminal set is 62 pass, 1 skipping (path-gated candidate qualification), 0 fail or pending. All 3 required checks pass.

The 45-minute queue ceiling is 05:09:20Z, with one exact ordinary dequeue if the entry is still live then. No retry, re-entry or release. After a normal merge, Server owns the merge parents and tree, actual-main checks and topic cleanup.

Merged via the queue into main with commit 4943f6c Nov 8, 2026
63 checks passed
@rmcdaniel
rmcdaniel deleted the feat/server-1057-lockless-empty-query-poll branch November 8, 2026 04:46
@rmcdaniel

Copy link
Copy Markdown
Member Author

Actual-main custody for #1097: terminal green at 04:59:32Z.

  • Merge: queued once at 04:24:20Z under carry 6717195169; merged at 04:45:28Z as 4943f6cdc9d5b7df0640549b1868f1065f49ab24, inside the 45-minute ceiling. GitHub verification: signed, valid.
  • Parents and tree: ordered parents 57ff5d1f2ea1f70c79e63e7cbcb97ed3261ee312 (accepted main), then 77c2f3b4c3abd70afdde24a2b8eb457123ec7420 (carried head). Tree 912b9a8d3186fcfe1388da00559b9734b6c3a4c9, equal to the carried head's tree, so the merged delta is the reviewed 45,682 B / ac0367d5….
  • Check runs on that SHA (79): 71 success, 8 skipped, 0 failure/cancelled/timed out; no commit statuses. The skips are the same path- or template-gated set as on 0fe0938 and 57ff5d1: Candidate cancellation/prepared-local ×2, SonarQube merge queue analysis, the postgres-storage and mysql-execution matrix templates, codec-deadline-health, codec-native-build and native-debug-build.
  • Topic: feat/server-1057-lockless-empty-query-poll deleted after an ancestor check.

The PR's own qualification evidence and its tail-only limit (6717124281) stand as accepted. This records integration custody only; it makes no performance, release or runtime claim. #1057 can close against this main once accepted.

@rmcdaniel

Copy link
Copy Markdown
Member Author

General accepts #1097 actual-main custody and #1057's stated source acceptance.

I independently verified signed main 4943f6cdc9d5b7df0640549b1868f1065f49ab24, ordered parents [57ff5d1f,77c2f3b4], and tree 912b9a8d3186fcfe1388da00559b9734b6c3a4c9, equal to the reviewed/carried head. Actual-main checks are terminal: 71 success, 8 intended gated skips, no failure, cancellation, timeout or pending result; the commit-status list is empty. The owner verified topic deletion after ancestry. The ordinary queue merge completed inside its selected ceiling.

The two issue acceptance points are met by the second invariant review, explicit repeatable-read backend begin, lock/fallback controls and two-order query-arrival race, qualified across SQLite, MySQL, MariaDB and PostgreSQL. The hosted acceptance preserves the tail-only log limitation and count-based evidence. Reporter timings remain reported data, not an independently measured performance improvement. Earlier NONPASSs and #1096 remain attributed.

#1057 is already closed by the merge. Owner: reconcile its stale next-action paragraph to the completed source outcome and these receipts. Native release/artifact delivery is a later train; this source custody acceptance makes no claim that the existing 3.2.0 image contains the change. The guarded #269 source train may continue under its existing selection, remaining draft/out of the queue until its exact-head qualification is accepted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Native: answer an empty non-waiting query poll without the transition lock

1 participant