Skip to content

test(database): promote 2 QA secondary-index integrity anchors#1886

Merged
kriszyp merged 5 commits into
mainfrom
kris/qa-promote-index-integrity
Jul 23, 2026
Merged

test(database): promote 2 QA secondary-index integrity anchors#1886
kriszyp merged 5 commits into
mainfrom
kris/qa-promote-index-integrity

Conversation

@kriszyp

@kriszyp kriszyp commented Jul 21, 2026

Copy link
Copy Markdown
Member

What

Two promoted exploratory-QA specs, both green on 1e1edc666, landing in integrationTests/database/. They pin what stays correct about secondary-index ↔ primary-record consistency, in corners the existing suite doesn't reach.

Spec Origin What it pins
eviction-index-atomicity-lmdb.test.ts QA-614 TTL/expiration eviction on the LMDB engine removes the primary row and its raw @indexed entry atomically — base and index both drain to 0, phantom=0/missing=0, for audit:false and audit:true, at 4 workers. This is the clean-engine counterpart that bounds the known RocksDB eviction-orphan behavior to RocksDB.
aborted-update-index-migration.test.ts QA-616 An aborted UPDATE that migrates an @indexed value A→B is fully atomic under both abort mechanisms — an in-request throw and an over-time abort (storage.maxTransactionOpenTime, monitor fire hard-asserted). The old entry survives at A, nothing leaks at B, the primary rolls back.

Why these two together

They are one theme: the index and the primary record agree, on every path that removes or moves an entry. Each is a negative result that constrains where index/primary skew can live — which is exactly the kind of property that regresses silently, because no query surface reports it.

That last point drives the method. Both read the raw secondary-index dbi directly rather than through search_by_value: transformToEntries() joins every index hit through the primary record and SKIPs when the primary is absent, so a query-surface oracle is structurally incapable of observing a dangling index entry. Each spec therefore also carries a positive control that injects a synthetic dangling entry (a real put, then a primary-store removal that bypasses updateIndices()) and asserts the oracle sees it — so "0 dangling" is earned rather than vacuous.

Deliberately not included: the RocksDB differential leg that the QA-615 scratch spec also ran. It reproduces a currently-open defect (200/200 orphaned entries), and a test that asserts broken behavior turns red the day that behavior is fixed. The RocksDB side belongs with its fix. For the same reason, the QA-616 positive control was re-based onto the synthetic injection mechanism instead of the delete-then-abort mechanism it originally borrowed from an open defect. Both notes are recorded in-file so the intent survives.

Verification

Run in a clean worktree at 1e1edc666, each spec cold:

aborted-update-index-migration.test.ts   tests 5 · pass 5 · fail 0  (~9.2s)
eviction-index-atomicity-lmdb.test.ts    tests 8 · pass 8 · fail 0  (~17.1s)

Positive controls fired in both. oxlint and prettier clean. Each spec is requires-isolation (distinct storage engine / worker count / transaction-timeout config), so they are separate files rather than appended to a shared-instance suite.

Update — a third spec was pulled from this batch

eviction-index-transient-lmdb.test.ts (QA-615) originally rode along here and is no longer part of this PR (c266e9d6b). After the restart_service race fix in a9cd33f38 cleared the other two, its part-C legs still failed on contended CI shards: 6/6 (Node.js v22) aborted at ~10.2s with [TimeoutError: The operation was aborted due to timeout], and 6/6 (Bun) at ~1.4s with TypeError: fetch failed / read ECONNRESET. Those legs drive a client-side sample loop concurrently with a live TTL eviction sweep at threads.count=4, and the individual sample requests don't survive a loaded runner.

Part C is that file's reason to exist — the transient-window re-audit is precisely what the settled-state sibling can't see — so trimming the flaky legs would leave nothing worth pinning. It goes back to the QA candidate pool for a CI-robust rework (per-sample retry/backoff, a sampling budget that degrades instead of asserting, and a separate look at whether the ECONNRESET-under-eviction is itself worth a scenario) rather than being merged flaky to keep the batch at three.

🤖 Generated with Claude Code

@claude

claude Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Reviewed; no blockers found.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces comprehensive integration tests and fixtures to verify secondary-index atomicity and eviction behavior under abort and TTL scenarios on both RocksDB and LMDB storage engines. The reviewer's feedback is highly actionable, focusing on improving test reliability and performance. Key recommendations include replacing fixed sleeps with condition-based polling loops to prevent flakiness, removing redundant sleeps, and strictly validating numeric configuration parameters (such as stallMs) to handle potential NaN values robustly.

Comment thread integrationTests/database/aborted-update-index-migration.test.ts Outdated
Comment thread integrationTests/database/aborted-update-index-migration.test.ts Outdated
Comment thread integrationTests/database/aborted-update-index-migration.test.ts Outdated
Comment thread integrationTests/database/aborted-update-index-migration/resources.js Outdated
Comment thread integrationTests/database/eviction-index-atomicity-lmdb.test.ts Outdated
Comment thread integrationTests/database/eviction-index-atomicity-lmdb.test.ts Outdated
kriszyp added a commit that referenced this pull request Jul 21, 2026
…led fixtures

restart_service is a fire-and-forget job (server/jobs/jobRunner.ts
launchJobThread does not await the job thread's actual work), so
restartHttpWorkers()'s HTTP response resolves as soon as the job is
launched, not once the workers have actually cycled. Its readiness probe
only detects this correctly when the probed route is 404 until the
restart completes (the redeploy/new-component-route scenario the helper
was designed for).

Both eviction-index-atomicity-lmdb.test.ts and
aborted-update-index-migration.test.ts call restartHttpWorkers() against
a route that's already live from boot (component pre-installed via
config.yaml's jsResource/graphqlSchema, same as the passing
eviction-secondary-index.test.ts sibling) -- so the probe passes
trivially before the real restart (tearing down + respawning every
threads.count worker) has finished, leaving it to race whichever test
runs next. On CI's more contended shard hosts this reliably landed mid-suite,
ECONNREFUSE-ing in-flight requests (both files showed up in the broad
6/6 (Node.js v22/v24/v26) and 5/6 (Bun) CI failures on PR #1886).

Since both fixtures are pre-installed and never redeployed, no restart is
needed at all here -- drop the restart_service round trip entirely and
just poll the probe route for readiness, matching the established pattern
in eviction-secondary-index.test.ts.

Refs #1886

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@kriszyp kriszyp changed the title test(database): promote 3 QA secondary-index integrity anchors test(database): promote 2 QA secondary-index integrity anchors Jul 21, 2026
kriszyp added a commit that referenced this pull request Jul 22, 2026
- aborted-update-index-migration.test.ts: assert pr.status===200
  unconditionally before checking the rolled-back category (both
  experiments), so a regression that deletes the primary row while
  leaving the old-A index entry dangling can no longer hide behind a
  404 that skips the check; also verify the original `value` field,
  not just `category`. Poll overTimeFireCount() with a bounded
  deadline instead of sampling once immediately after the response,
  since log output can be buffered past the response returning.
- eviction-index-atomicity-lmdb.test.ts: bump ItemF/ItemT expiration
  from 1s to 4s (matching the sibling fixture's proven-safe TTL) so
  seeding 750 rows plus two full precondition scans can't lose the
  race against the eviction sweep on a contended CI worker. Record
  A2's point-read status as a full bounded series requiring a stable
  streak before calling it settled, instead of exiting on the first
  404 sample and mislabeling a transient mid-rollback read as
  "reproduced".

Refs #1886

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@kriszyp
kriszyp marked this pull request as ready for review July 22, 2026 03:14
kriszyp and others added 5 commits July 22, 2026 16:09
Pins three properties of secondary-index/primary-record consistency that the
existing suite does not cover, all verified green on 1e1edc6:

- eviction-index-atomicity-lmdb (QA-614): TTL eviction on LMDB removes the
  primary row and its raw @indexed entry atomically (audit:false and
  audit:true, 4 workers) — the clean-engine counterpart that bounds the known
  RocksDB eviction-orphan behavior to RocksDB.
- eviction-index-transient-lmdb (QA-615): the same holds transiently, not just
  at settle — ~30k/35k direct-store samples across the eviction window observe
  zero dangling entries.
- aborted-update-index-migration (QA-616): an aborted UPDATE that migrates an
  @indexed value A->B is fully atomic under both an in-request throw and an
  over-time abort — the old entry survives at A, nothing leaks at B.

All three read the raw index dbi directly rather than through search_by_value,
which joins each index hit through the primary record and skips on absence and
so cannot observe a dangling entry. Each carries a positive control that
injects a synthetic dangling entry, so a "0 dangling" result is earned.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…led fixtures

restart_service is a fire-and-forget job (server/jobs/jobRunner.ts
launchJobThread does not await the job thread's actual work), so
restartHttpWorkers()'s HTTP response resolves as soon as the job is
launched, not once the workers have actually cycled. Its readiness probe
only detects this correctly when the probed route is 404 until the
restart completes (the redeploy/new-component-route scenario the helper
was designed for).

Both eviction-index-atomicity-lmdb.test.ts and
aborted-update-index-migration.test.ts call restartHttpWorkers() against
a route that's already live from boot (component pre-installed via
config.yaml's jsResource/graphqlSchema, same as the passing
eviction-secondary-index.test.ts sibling) -- so the probe passes
trivially before the real restart (tearing down + respawning every
threads.count worker) has finished, leaving it to race whichever test
runs next. On CI's more contended shard hosts this reliably landed mid-suite,
ECONNREFUSE-ing in-flight requests (both files showed up in the broad
6/6 (Node.js v22/v24/v26) and 5/6 (Bun) CI failures on PR #1886).

Since both fixtures are pre-installed and never redeployed, no restart is
needed at all here -- drop the restart_service round trip entirely and
just poll the probe route for readiness, matching the established pattern
in eviction-secondary-index.test.ts.

Refs #1886

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
eviction-index-transient-lmdb.test.ts still fails on contended CI shards
after a9cd33f: its part C legs ("tight client-driven sampling across the
full TTL-eviction sweep") drive a client-side sample loop concurrently with
a real TTL eviction sweep at threads.count=4, and on a loaded runner the
individual sample requests do not survive -- 6/6 (Node.js v22) aborted with
[TimeoutError: The operation was aborted due to timeout] at ~10.2s, and
6/6 (Bun) failed at ~1.4s with TypeError: fetch failed / read ECONNRESET.
The two other specs in the batch pass on the same shards now that the
restart_service race is gone, so this is specific to the sampling loop, not
to the batch.

Part C is the whole point of that file (the transient-window re-audit is
exactly what the settled-state sibling eviction-index-atomicity-lmdb.test.ts
cannot see), so trimming the flaky legs would leave nothing worth pinning.
Per the promotion policy a spec that flakes in its PR's CI gets pulled from
the batch rather than kept to preserve batch size -- it goes back to the QA
candidate pool for a CI-robust rework (per-sample retry/backoff, a sampling
budget that degrades gracefully instead of failing the assertion, and a
decision on whether the ECONNRESET-under-eviction itself deserves its own
scenario).

This PR keeps the two green anchors: eviction-index-atomicity-lmdb and
aborted-update-index-migration.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- aborted-update-index-migration.test.ts: assert pr.status===200
  unconditionally before checking the rolled-back category (both
  experiments), so a regression that deletes the primary row while
  leaving the old-A index entry dangling can no longer hide behind a
  404 that skips the check; also verify the original `value` field,
  not just `category`. Poll overTimeFireCount() with a bounded
  deadline instead of sampling once immediately after the response,
  since log output can be buffered past the response returning.
- eviction-index-atomicity-lmdb.test.ts: bump ItemF/ItemT expiration
  from 1s to 4s (matching the sibling fixture's proven-safe TTL) so
  seeding 750 rows plus two full precondition scans can't lose the
  race against the eviction sweep on a contended CI worker. Record
  A2's point-read status as a full bounded series requiring a stable
  streak before calling it settled, instead of exiting on the first
  404 sample and mislabeling a transient mid-rollback read as
  "reproduced".

Refs #1886

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- POSITIVE CONTROL and EXPERIMENT 1/2 in aborted-update-index-migration.test.ts:
  replace fixed sleep(400) + single read with a bounded condition-wait
  polling for the expected settled state, matching the repo's
  condition-wait guidance; a genuine defect that never settles still
  rides out the deadline and is caught by the existing assertions.
- UpdateThenStall's stallMs: guard against NaN/non-positive input
  (Number.isInteger + > 0) instead of trusting Number(b.stallMs)
  directly, which would resolve immediately on bad input.
- A2 (eviction-index-atomicity-lmdb.test.ts): drop the now-redundant
  fixed sleep(300) ahead of the status-series poll loop added in the
  prior commit.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@kriszyp
kriszyp merged commit c711c7a into main Jul 23, 2026
95 of 96 checks passed
kriszyp added a commit that referenced this pull request Jul 23, 2026
…led fixtures

restart_service is a fire-and-forget job (server/jobs/jobRunner.ts
launchJobThread does not await the job thread's actual work), so
restartHttpWorkers()'s HTTP response resolves as soon as the job is
launched, not once the workers have actually cycled. Its readiness probe
only detects this correctly when the probed route is 404 until the
restart completes (the redeploy/new-component-route scenario the helper
was designed for).

Both eviction-index-atomicity-lmdb.test.ts and
aborted-update-index-migration.test.ts call restartHttpWorkers() against
a route that's already live from boot (component pre-installed via
config.yaml's jsResource/graphqlSchema, same as the passing
eviction-secondary-index.test.ts sibling) -- so the probe passes
trivially before the real restart (tearing down + respawning every
threads.count worker) has finished, leaving it to race whichever test
runs next. On CI's more contended shard hosts this reliably landed mid-suite,
ECONNREFUSE-ing in-flight requests (both files showed up in the broad
6/6 (Node.js v22/v24/v26) and 5/6 (Bun) CI failures on PR #1886).

Since both fixtures are pre-installed and never redeployed, no restart is
needed at all here -- drop the restart_service round trip entirely and
just poll the probe route for readiness, matching the established pattern
in eviction-secondary-index.test.ts.

Refs #1886

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
kriszyp added a commit that referenced this pull request Jul 23, 2026
- aborted-update-index-migration.test.ts: assert pr.status===200
  unconditionally before checking the rolled-back category (both
  experiments), so a regression that deletes the primary row while
  leaving the old-A index entry dangling can no longer hide behind a
  404 that skips the check; also verify the original `value` field,
  not just `category`. Poll overTimeFireCount() with a bounded
  deadline instead of sampling once immediately after the response,
  since log output can be buffered past the response returning.
- eviction-index-atomicity-lmdb.test.ts: bump ItemF/ItemT expiration
  from 1s to 4s (matching the sibling fixture's proven-safe TTL) so
  seeding 750 rows plus two full precondition scans can't lose the
  race against the eviction sweep on a contended CI worker. Record
  A2's point-read status as a full bounded series requiring a stable
  streak before calling it settled, instead of exiting on the first
  404 sample and mislabeling a transient mid-rollback read as
  "reproduced".

Refs #1886

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@kriszyp
kriszyp deleted the kris/qa-promote-index-integrity branch July 23, 2026 14:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants