test(database): promote 2 QA secondary-index integrity anchors#1886
Merged
Conversation
Contributor
|
Reviewed; no blockers found. |
Contributor
There was a problem hiding this comment.
Code Review
This pull request introduces comprehensive integration tests and fixtures to verify secondary-index atomicity and eviction behavior under abort and TTL scenarios on both RocksDB and LMDB storage engines. The reviewer's feedback is highly actionable, focusing on improving test reliability and performance. Key recommendations include replacing fixed sleeps with condition-based polling loops to prevent flakiness, removing redundant sleeps, and strictly validating numeric configuration parameters (such as stallMs) to handle potential NaN values robustly.
kriszyp
added a commit
that referenced
this pull request
Jul 21, 2026
…led fixtures restart_service is a fire-and-forget job (server/jobs/jobRunner.ts launchJobThread does not await the job thread's actual work), so restartHttpWorkers()'s HTTP response resolves as soon as the job is launched, not once the workers have actually cycled. Its readiness probe only detects this correctly when the probed route is 404 until the restart completes (the redeploy/new-component-route scenario the helper was designed for). Both eviction-index-atomicity-lmdb.test.ts and aborted-update-index-migration.test.ts call restartHttpWorkers() against a route that's already live from boot (component pre-installed via config.yaml's jsResource/graphqlSchema, same as the passing eviction-secondary-index.test.ts sibling) -- so the probe passes trivially before the real restart (tearing down + respawning every threads.count worker) has finished, leaving it to race whichever test runs next. On CI's more contended shard hosts this reliably landed mid-suite, ECONNREFUSE-ing in-flight requests (both files showed up in the broad 6/6 (Node.js v22/v24/v26) and 5/6 (Bun) CI failures on PR #1886). Since both fixtures are pre-installed and never redeployed, no restart is needed at all here -- drop the restart_service round trip entirely and just poll the probe route for readiness, matching the established pattern in eviction-secondary-index.test.ts. Refs #1886 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
kriszyp
added a commit
that referenced
this pull request
Jul 22, 2026
- aborted-update-index-migration.test.ts: assert pr.status===200 unconditionally before checking the rolled-back category (both experiments), so a regression that deletes the primary row while leaving the old-A index entry dangling can no longer hide behind a 404 that skips the check; also verify the original `value` field, not just `category`. Poll overTimeFireCount() with a bounded deadline instead of sampling once immediately after the response, since log output can be buffered past the response returning. - eviction-index-atomicity-lmdb.test.ts: bump ItemF/ItemT expiration from 1s to 4s (matching the sibling fixture's proven-safe TTL) so seeding 750 rows plus two full precondition scans can't lose the race against the eviction sweep on a contended CI worker. Record A2's point-read status as a full bounded series requiring a stable streak before calling it settled, instead of exiting on the first 404 sample and mislabeling a transient mid-rollback read as "reproduced". Refs #1886 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
kriszyp
marked this pull request as ready for review
July 22, 2026 03:14
Pins three properties of secondary-index/primary-record consistency that the existing suite does not cover, all verified green on 1e1edc6: - eviction-index-atomicity-lmdb (QA-614): TTL eviction on LMDB removes the primary row and its raw @indexed entry atomically (audit:false and audit:true, 4 workers) — the clean-engine counterpart that bounds the known RocksDB eviction-orphan behavior to RocksDB. - eviction-index-transient-lmdb (QA-615): the same holds transiently, not just at settle — ~30k/35k direct-store samples across the eviction window observe zero dangling entries. - aborted-update-index-migration (QA-616): an aborted UPDATE that migrates an @indexed value A->B is fully atomic under both an in-request throw and an over-time abort — the old entry survives at A, nothing leaks at B. All three read the raw index dbi directly rather than through search_by_value, which joins each index hit through the primary record and skips on absence and so cannot observe a dangling entry. Each carries a positive control that injects a synthetic dangling entry, so a "0 dangling" result is earned. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…led fixtures restart_service is a fire-and-forget job (server/jobs/jobRunner.ts launchJobThread does not await the job thread's actual work), so restartHttpWorkers()'s HTTP response resolves as soon as the job is launched, not once the workers have actually cycled. Its readiness probe only detects this correctly when the probed route is 404 until the restart completes (the redeploy/new-component-route scenario the helper was designed for). Both eviction-index-atomicity-lmdb.test.ts and aborted-update-index-migration.test.ts call restartHttpWorkers() against a route that's already live from boot (component pre-installed via config.yaml's jsResource/graphqlSchema, same as the passing eviction-secondary-index.test.ts sibling) -- so the probe passes trivially before the real restart (tearing down + respawning every threads.count worker) has finished, leaving it to race whichever test runs next. On CI's more contended shard hosts this reliably landed mid-suite, ECONNREFUSE-ing in-flight requests (both files showed up in the broad 6/6 (Node.js v22/v24/v26) and 5/6 (Bun) CI failures on PR #1886). Since both fixtures are pre-installed and never redeployed, no restart is needed at all here -- drop the restart_service round trip entirely and just poll the probe route for readiness, matching the established pattern in eviction-secondary-index.test.ts. Refs #1886 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
eviction-index-transient-lmdb.test.ts still fails on contended CI shards after a9cd33f: its part C legs ("tight client-driven sampling across the full TTL-eviction sweep") drive a client-side sample loop concurrently with a real TTL eviction sweep at threads.count=4, and on a loaded runner the individual sample requests do not survive -- 6/6 (Node.js v22) aborted with [TimeoutError: The operation was aborted due to timeout] at ~10.2s, and 6/6 (Bun) failed at ~1.4s with TypeError: fetch failed / read ECONNRESET. The two other specs in the batch pass on the same shards now that the restart_service race is gone, so this is specific to the sampling loop, not to the batch. Part C is the whole point of that file (the transient-window re-audit is exactly what the settled-state sibling eviction-index-atomicity-lmdb.test.ts cannot see), so trimming the flaky legs would leave nothing worth pinning. Per the promotion policy a spec that flakes in its PR's CI gets pulled from the batch rather than kept to preserve batch size -- it goes back to the QA candidate pool for a CI-robust rework (per-sample retry/backoff, a sampling budget that degrades gracefully instead of failing the assertion, and a decision on whether the ECONNRESET-under-eviction itself deserves its own scenario). This PR keeps the two green anchors: eviction-index-atomicity-lmdb and aborted-update-index-migration. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- aborted-update-index-migration.test.ts: assert pr.status===200 unconditionally before checking the rolled-back category (both experiments), so a regression that deletes the primary row while leaving the old-A index entry dangling can no longer hide behind a 404 that skips the check; also verify the original `value` field, not just `category`. Poll overTimeFireCount() with a bounded deadline instead of sampling once immediately after the response, since log output can be buffered past the response returning. - eviction-index-atomicity-lmdb.test.ts: bump ItemF/ItemT expiration from 1s to 4s (matching the sibling fixture's proven-safe TTL) so seeding 750 rows plus two full precondition scans can't lose the race against the eviction sweep on a contended CI worker. Record A2's point-read status as a full bounded series requiring a stable streak before calling it settled, instead of exiting on the first 404 sample and mislabeling a transient mid-rollback read as "reproduced". Refs #1886 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- POSITIVE CONTROL and EXPERIMENT 1/2 in aborted-update-index-migration.test.ts: replace fixed sleep(400) + single read with a bounded condition-wait polling for the expected settled state, matching the repo's condition-wait guidance; a genuine defect that never settles still rides out the deadline and is caught by the existing assertions. - UpdateThenStall's stallMs: guard against NaN/non-positive input (Number.isInteger + > 0) instead of trusting Number(b.stallMs) directly, which would resolve immediately on bad input. - A2 (eviction-index-atomicity-lmdb.test.ts): drop the now-redundant fixed sleep(300) ahead of the status-series poll loop added in the prior commit. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
kriszyp
force-pushed
the
kris/qa-promote-index-integrity
branch
from
July 22, 2026 22:09
ce5f96b to
6d798bf
Compare
cb1kenobi
approved these changes
Jul 23, 2026
kriszyp
added a commit
that referenced
this pull request
Jul 23, 2026
…led fixtures restart_service is a fire-and-forget job (server/jobs/jobRunner.ts launchJobThread does not await the job thread's actual work), so restartHttpWorkers()'s HTTP response resolves as soon as the job is launched, not once the workers have actually cycled. Its readiness probe only detects this correctly when the probed route is 404 until the restart completes (the redeploy/new-component-route scenario the helper was designed for). Both eviction-index-atomicity-lmdb.test.ts and aborted-update-index-migration.test.ts call restartHttpWorkers() against a route that's already live from boot (component pre-installed via config.yaml's jsResource/graphqlSchema, same as the passing eviction-secondary-index.test.ts sibling) -- so the probe passes trivially before the real restart (tearing down + respawning every threads.count worker) has finished, leaving it to race whichever test runs next. On CI's more contended shard hosts this reliably landed mid-suite, ECONNREFUSE-ing in-flight requests (both files showed up in the broad 6/6 (Node.js v22/v24/v26) and 5/6 (Bun) CI failures on PR #1886). Since both fixtures are pre-installed and never redeployed, no restart is needed at all here -- drop the restart_service round trip entirely and just poll the probe route for readiness, matching the established pattern in eviction-secondary-index.test.ts. Refs #1886 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
kriszyp
added a commit
that referenced
this pull request
Jul 23, 2026
- aborted-update-index-migration.test.ts: assert pr.status===200 unconditionally before checking the rolled-back category (both experiments), so a regression that deletes the primary row while leaving the old-A index entry dangling can no longer hide behind a 404 that skips the check; also verify the original `value` field, not just `category`. Poll overTimeFireCount() with a bounded deadline instead of sampling once immediately after the response, since log output can be buffered past the response returning. - eviction-index-atomicity-lmdb.test.ts: bump ItemF/ItemT expiration from 1s to 4s (matching the sibling fixture's proven-safe TTL) so seeding 750 rows plus two full precondition scans can't lose the race against the eviction sweep on a contended CI worker. Record A2's point-read status as a full bounded series requiring a stable streak before calling it settled, instead of exiting on the first 404 sample and mislabeling a transient mid-rollback read as "reproduced". Refs #1886 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two promoted exploratory-QA specs, both green on
1e1edc666, landing inintegrationTests/database/. They pin what stays correct about secondary-index ↔ primary-record consistency, in corners the existing suite doesn't reach.eviction-index-atomicity-lmdb.test.ts@indexedentry atomically — base and index both drain to 0, phantom=0/missing=0, foraudit:falseandaudit:true, at 4 workers. This is the clean-engine counterpart that bounds the known RocksDB eviction-orphan behavior to RocksDB.aborted-update-index-migration.test.ts@indexedvalue A→B is fully atomic under both abort mechanisms — an in-request throw and an over-time abort (storage.maxTransactionOpenTime, monitor fire hard-asserted). The old entry survives at A, nothing leaks at B, the primary rolls back.Why these two together
They are one theme: the index and the primary record agree, on every path that removes or moves an entry. Each is a negative result that constrains where index/primary skew can live — which is exactly the kind of property that regresses silently, because no query surface reports it.
That last point drives the method. Both read the raw secondary-index dbi directly rather than through
search_by_value:transformToEntries()joins every index hit through the primary record andSKIPs when the primary is absent, so a query-surface oracle is structurally incapable of observing a dangling index entry. Each spec therefore also carries a positive control that injects a synthetic dangling entry (a real put, then a primary-store removal that bypassesupdateIndices()) and asserts the oracle sees it — so "0 dangling" is earned rather than vacuous.Deliberately not included: the RocksDB differential leg that the QA-615 scratch spec also ran. It reproduces a currently-open defect (200/200 orphaned entries), and a test that asserts broken behavior turns red the day that behavior is fixed. The RocksDB side belongs with its fix. For the same reason, the QA-616 positive control was re-based onto the synthetic injection mechanism instead of the delete-then-abort mechanism it originally borrowed from an open defect. Both notes are recorded in-file so the intent survives.
Verification
Run in a clean worktree at
1e1edc666, each spec cold:Positive controls fired in both.
oxlintandprettierclean. Each spec isrequires-isolation(distinct storage engine / worker count / transaction-timeout config), so they are separate files rather than appended to a shared-instance suite.Update — a third spec was pulled from this batch
eviction-index-transient-lmdb.test.ts(QA-615) originally rode along here and is no longer part of this PR (c266e9d6b). After therestart_servicerace fix ina9cd33f38cleared the other two, its part-C legs still failed on contended CI shards: 6/6 (Node.js v22) aborted at ~10.2s with[TimeoutError: The operation was aborted due to timeout], and 6/6 (Bun) at ~1.4s withTypeError: fetch failed/read ECONNRESET. Those legs drive a client-side sample loop concurrently with a live TTL eviction sweep atthreads.count=4, and the individual sample requests don't survive a loaded runner.Part C is that file's reason to exist — the transient-window re-audit is precisely what the settled-state sibling can't see — so trimming the flaky legs would leave nothing worth pinning. It goes back to the QA candidate pool for a CI-robust rework (per-sample retry/backoff, a sampling budget that degrades instead of asserting, and a separate look at whether the ECONNRESET-under-eviction is itself worth a scenario) rather than being merged flaky to keep the batch at three.
🤖 Generated with Claude Code