Skip to content

test(upgrade): promote 3 QA cross-version upgrade read-visibility anchors#1900

Open
kriszyp wants to merge 1 commit into
mainfrom
kris/qa-promote-upgrade
Open

test(upgrade): promote 3 QA cross-version upgrade read-visibility anchors#1900
kriszyp wants to merge 1 commit into
mainfrom
kris/qa-promote-upgrade

Conversation

@kriszyp

@kriszyp kriszyp commented Jul 22, 2026

Copy link
Copy Markdown
Member

Summary

Promotes 3 gated qa-explorer regression specs into the tracked integrationTests/upgrade/ suite, anchoring findings from the gh#1865 upgrade-boot read-visibility investigation. All three are test-only (no product code touched) and each stays its own file — all requires-isolation (they boot a real npm-provisioned harper@5.1.22 and/or restart/SIGKILL the instance).

  • cross-version-upgrade-visibility.test.ts (from QA-658 / P-437): a REAL 5.1.22 → current cross-version upgrade boot keeps every pre-upgrade component-table row visible on all six read surfaces — the gh#1865 refutation. Measured: 6.7s.
  • cross-version-upgrade-residuals.test.ts (from QA-660 / P-438): the same cross-version upgrade stays read-consistent at 55k rows through a SIGKILL-dirty shutdown and boot-boundary writes — closes the scale/shape + dirty-shutdown gap QA-658 left untested. Measured: 15.9s.
  • upgrade-boot-schema-change.test.ts (from QA-647 / P-432): restart + attribute-adding schema change never makes existing rows invisible on any read surface (6-surface matrix, two restarts) — the same-build control arm the two cross-version suites are compared against. Measured: 8.0s.

Network dependency: cross-version-upgrade-visibility and cross-version-upgrade-residuals npm-provision harper@5.1.22 at runtime (via ~/dev/tmp/qa658-harper51, overridable with HARPER_LEGACY_51_PATH) and skip cleanly if that install isn't present. This is a CI network dependency — leaving it to review whether/how to provision in CI.

Test plan

  • npm run build (known pre-existing resources/DatabaseTransaction.ts type error unrelated to this change; dist emits fine)
  • Ran each promoted file individually, foreground, with timeout:
    • cross-version-upgrade-visibility.test.ts: 4 pass, 1 skip (no-miss-to-heal), 6.7s
    • cross-version-upgrade-residuals.test.ts: 5 pass, 1 skip (no-miss-to-heal), 15.9s
    • upgrade-boot-schema-change.test.ts: 4 pass, 1 skip (no-miss-to-heal), 8.0s
  • oxlint clean, prettier --write applied

🤖 Promoted by qa-explorer (Claude Opus 4.8 orchestrator / Sonnet promotion agent)

…hors

Pins three properties of the gh#1865 upgrade-boot read-visibility investigation
that the existing suite does not cover, all verified green at 07c2bbc:

- cross-version-upgrade-visibility (QA-658/P-437): a REAL 5.1.22 -> current
  cross-version upgrade boot (real npm-provisioned legacy install, real
  dataRootDir handoff) keeps every pre-upgrade component-table row visible on
  all six read surfaces — the gh#1865 refutation's genuine cross-version arm.
- cross-version-upgrade-residuals (QA-660/P-438): the same cross-version
  upgrade stays read-consistent at 55k rows through a SIGKILL-dirty shutdown
  and writes racing the boot boundary — the scale/shape + dirty-shutdown
  residual QA-658 left untested.
- upgrade-boot-schema-change (QA-647/P-432): a same-build restart plus an
  attribute-adding schema change across the boot never makes existing rows
  invisible on any read surface (6-surface matrix, two restarts) — the
  same-build control arm the two cross-version suites are compared against.

cross-version-upgrade-visibility and cross-version-upgrade-residuals
npm-provision harper@5.1.22 at runtime and skip cleanly if that install isn't
present, so they carry a network dependency in CI.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces comprehensive integration tests and fixtures to verify read visibility across various database surfaces during cross-version upgrades and schema changes. The feedback highlights a redundancy in the cmp helper function within upgrade-boot-schema-change.test.ts, where the parameter b is passed but remains unused because the value is re-derived dynamically. Addressing this redundancy will improve code clarity and maintainability.

Comment on lines +396 to +410
const cmp = (a: string[], b: string[], name: string) => {
const wasOk = matrixAfterRestart1[name as keyof SurfaceMatrix] as unknown as string[];
log(
`Q4 boot-order check (${name}): restart#1 misses=${(wasOk as string[]).length}, ` +
`restart#2 misses=${a.length} -> ${
a.length === 0 && (wasOk as string[]).length > 0
? 'CLEARED on 2nd boot (order-dependent)'
: a.length > 0 && (wasOk as string[]).length === 0
? 'NEW MISS introduced by 2nd boot alone'
: a.length === (wasOk as string[]).length
? 'UNCHANGED across boots (stable)'
: 'CHANGED (different miss set/count)'
}`
);
};

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The cmp function's implementation is a bit confusing. It accepts a b parameter but then re-derives the same value from matrixAfterRestart1 using the name parameter, leaving b unused. This is redundant and makes the code harder to read.

For better clarity and maintainability, you should use the b parameter directly.

const cmp = (a: string[], b: string[], name: string) => {
	log(
		`Q4 boot-order check (${name}): restart#1 misses=${b.length}, ` +
			`restart#2 misses=${a.length} -> ${
				a.length === 0 && b.length > 0
					? 'CLEARED on 2nd boot (order-dependent)'
					: a.length > 0 && b.length === 0
						? 'NEW MISS introduced by 2nd boot alone'
						: a.length === b.length
							? 'UNCHANGED across boots (stable)'
							: 'CHANGED (different miss set/count)'
			}`
	);
};

log(`(b) dirty-shutdown trigger: ${acked}/${numChunks} chunk requests ACKed (client received 200) before kill`);
const proc = ctx.harper.process;
const pid = proc.pid!;
process.kill(-pid, 'SIGKILL');

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion (non-blocking): process.kill(-pid, 'SIGKILL') targets an entire process group, which only does what's intended if ctx.harper.process was spawned with detached: true (making it its own group leader). Two other tests in this repo doing the same "dirty crash" kill (server/crash-replay.test.ts:26, server/replay-stress.test.ts:167) use the simpler ctx.harper.process.kill('SIGKILL') — no negative pid. If the framework doesn't spawn detached, -pid here will likely throw ESRCH (uncaught, since there's no try/catch around this call) instead of delivering the kill. Consider matching the established ctx.harper.process.kill('SIGKILL') pattern unless there's a specific reason this test needs to also reap child processes of the Harper process itself.

@claude

claude Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Reviewed; no blockers found.

@kriszyp
kriszyp marked this pull request as ready for review July 22, 2026 18:50
resolve(process.env.HOME ?? '', 'dev/tmp/qa658-harper51/node_modules/harper/dist/bin/harper.js');
const legacy51Available = existsSync(LEGACY_51_BIN_PATH);

const skipSuite = process.platform === 'win32' || !legacy51Available;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

High: this anchor never runs in CI, so it provides no actual regression protection for #1865

skipSuite is true whenever !legacy51Available, and LEGACY_51_BIN_PATH defaults to ~/dev/tmp/qa658-harper51/node_modules/harper/dist/bin/harper.js — a path that only exists on a developer machine that manually ran the npm install harper@5.1.22 --no-save step described in the file header. Nothing in this PR provisions HARPER_LEGACY_51_PATH (or that default path) in CI: .github/workflows/integration-tests.yml already provisions a legacy build for the 4.x suite via HARPER_LEGACY_VERSION_PATH (lines 113/192/291), but has no equivalent step for HARPER_LEGACY_51_PATH. So on every CI run this whole suite silently skips — the PR title's "promote into the tracked suite" doesn't hold in practice; the test exists in the repo but never executes anywhere except a machine with that exact local install, which means it can't fail on the cross-version regression it's meant to anchor.

Suggested fix: add a CI step (mirroring the existing HARPER_LEGACY_VERSION_PATH provisioning in integration-tests.yml) that npm-installs harper@5.1.22 into a known path and sets HARPER_LEGACY_51_PATH, or explicitly document in the PR that this anchor is local-only until that's wired up.


Generated by Barber AI

process.env.HARPER_LEGACY_51_PATH ??
resolve(process.env.HOME ?? '', 'dev/tmp/qa658-harper51/node_modules/harper/dist/bin/harper.js');
const legacy51Available = existsSync(LEGACY_51_BIN_PATH);
const skipSuite = process.platform === 'win32' || !legacy51Available;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

High: same CI-provisioning gap as cross-version-upgrade-visibility.test.ts — this anchor also never runs in CI

Same !legacy51Available skip condition, same unprovisioned default path. Combined with the sibling file, that's 2 of the 3 "promoted" anchors in this PR that will perpetually skip in CI (only upgrade-boot-schema-change.test.ts — the same-build control arm, not a cross-version boot — actually runs unconditionally). The two tests that exercise the actual #1865 cross-version scenario are the ones gated off.

Suggested fix: same as the sibling file — provision HARPER_LEGACY_51_PATH in CI (or the shared default path) alongside the existing 4.x legacy-install step, so these anchors run somewhere that can actually catch a regression.


Generated by Barber AI

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants