fix(scripts): make the Postgres smoke report the code under test, not the environment - #149
Merged
Conversation
added 2 commits
August 8, 2026 11:31
Three script-only defects made smoke_postgres.sh report failures that had nothing to do with the code under test. 1. Interpreter split. PYBIN came from $PYTHON (scripts/smoke_postgres.sh:43) but the checks shelled out to a bare `jvagent` off PATH (:96, :125). On a host with more than one install those are different environments; here PATH resolved to one whose jvspatial was an editable sibling checkout without asyncpg, so bootstrap died with "ImportError: asyncpg is required for the Postgres backend" and 8 checks failed spuriously. The entrypoint is now derived from PYBIN — $(dirname $PYBIN)/jvagent, falling back to "$PYBIN -m jvagent" — and used everywhere. 2. Overlapping runs killed each other. Container name and port were fixed and teardown was `pkill -f "jvagent $APP_ROOT"` (:54, :104), which matches every concurrent run's server: an earlier run's EXIT trap terminated a later run's server mid-restart, yielding "server failed to restart", "login failed post-restart" and a bogus event-loop error count. Container name and both ports are now per-run unique (PID-seeded, then probed for a free port), and the server is tracked and killed by PID instead of by name pattern. 3. Logs deleted on failure. The EXIT trap ran `rm -rf "$WORKDIR"` unconditionally (:56), so the bootstrap.log / server_*.log a failure message pointed at were already gone. The workdir is now preserved and its path printed whenever any check failed. Verified: 15/15 with no PATH manipulation (PYTHON=.venv/bin/python), and two deliberately overlapping runs no longer interfere.
…olding replies Two follow-ups from running the concurrency/interpreter fix in anger. APP_ROOT defaulted to the cwd-relative "examples/jvagent_app". The app's .env is gitignored, so running the script from a worktree — or anywhere that is not the checkout root — silently picks up a different app, or a keyless one. Resolve it from BASH_SOURCE instead so the default always means "this repo's example app". An explicit positional argument still overrides it. The turn check tested only that the reply was non-empty. Without a usable model key the agent still answers — by echoing the prompt scaffolding it was handed — so the check passed on output that proved nothing, TURNS stayed 1, and the run came apart two steps later at the recall check with a failure that pointed at Postgres rather than at the missing key. That is how a 15/15 suite reported "14 passed, 1 failed" for a reason that had nothing to do with the database. reply_is_substantive() rejects the scaffolding markers and a too-short reply. A scaffolding reply now demotes the turn to SKIP, names the likely cause, and clears TURNS so the dependent persistence and recall checks skip with it. Verified three ways against merged main, each isolating one behavior: - keyed app, explicit root, cwd=/tmp -> 15 passed, 0 failed, 0 skipped - keyless app (worktree), default root -> 12 passed, 0 failed, 2 skipped - keyless app, default root, cwd=/tmp -> 12 passed, 0 failed, 2 skipped The last two are the point: identical results from inside and outside the repo, and zero failures where the same condition previously produced a misleading one.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
scripts/smoke_postgres.shreported failures three separate times this week that had nothing to do with Postgres, jvspatial, or jvagent. Each one cost real investigation before turning out to be the harness. Two commits, four defects, all in the script.What went wrong, and why it was hard to see
1. It tested whichever
jvagentwas onPATH. Line 43 honoured$PYTHONforPYBIN, then invoked the barejvagentconsole script. On a box with more than one install, the run exercised a different environment than the one you asked for. In practicejvagentresolved to an install whosejvspatialwas an editable sibling checkout withoutasyncpg, so step 1 died withImportError: asyncpg is required for the Postgres backendand the suite reported 8 failures across every later step. The interpreter is now derived from$PYTHON—$(dirname "$(command -v "$PYBIN")")/jvagent, falling back to"$PYBIN" -m jvagent— andImportErrorwas added to the bootstrap-failure grep so this specific cause names itself next time.2. Two runs killed each other. Fixed container name, fixed ports, and
pkill -f "jvagent $APP_ROOT"in theEXITtrap. An earlier run's cleanup terminated a later run's server mid-restart, producingserver failed to restart,login failed post-restart, and a spurious1 postgres/event-loop error in server logs— 3 failures, all pointing at the event-loop affinity code, none of them real. Container name and both ports are now per-run unique ($$-seeded,pick_portscans upward), and the server is tracked and torn down by PID.3.
APP_ROOTwas cwd-relative and the app.envis gitignored. Run the script from a worktree and you silently get a keyless app. Now resolved fromBASH_SOURCE; an explicit positional argument still overrides.4. The turn check passed on non-empty, not on substantive. This is the one worth reading twice. With no usable model key the agent still answers — by echoing the prompt scaffolding it was handed:
[ -n "$r1" ]was satisfied,turn produced a replypassed,TURNSstayed1, and the run fell over two steps later at the recall check — with a failure that reads as "Postgres did not persist the interaction". The existing "no model key → skip" guard never fired, because there was a reply.reply_is_substantive()now rejects the scaffolding markers and a too-short reply, demotes the turn toSKIPnaming the likely cause, and clearsTURNSso the dependent checks skip with it.Also: the workdir is now kept when any check fails, and its path printed. Previously the
EXITtrap ranrm -rf "$WORKDIR"unconditionally, so a failure message pointed you at abootstrap.logthat had already been deleted.Verification
Each run isolates one behaviour, all against merged
main:/tmp/tmpThe last two are the point. Identical results inside and outside the repo, and zero failures where the same condition previously produced a misleading one — the keyless case used to report 14 passed, 1 failed.
--no-dockeris unaffected: it requiresJVSPATIAL_POSTGRES_DSNexplicitly (the:?guard), so the now-dynamicPGPORTis never read on that path.Gate:
pre-commit run --all-filesclean,pytest tests/3356 passed / 4 skipped.