Skip to content

fix(scripts): make the Postgres smoke report the code under test, not the environment - #149

Merged
eldonm merged 2 commits into
mainfrom
fix/smoke-postgres-hardening
Aug 8, 2026
Merged

fix(scripts): make the Postgres smoke report the code under test, not the environment#149
eldonm merged 2 commits into
mainfrom
fix/smoke-postgres-hardening

Conversation

@eldonm

@eldonm eldonm commented Aug 8, 2026

Copy link
Copy Markdown
Member

scripts/smoke_postgres.sh reported failures three separate times this week that had nothing to do with Postgres, jvspatial, or jvagent. Each one cost real investigation before turning out to be the harness. Two commits, four defects, all in the script.

What went wrong, and why it was hard to see

1. It tested whichever jvagent was on PATH. Line 43 honoured $PYTHON for PYBIN, then invoked the bare jvagent console script. On a box with more than one install, the run exercised a different environment than the one you asked for. In practice jvagent resolved to an install whose jvspatial was an editable sibling checkout without asyncpg, so step 1 died with ImportError: asyncpg is required for the Postgres backend and the suite reported 8 failures across every later step. The interpreter is now derived from $PYTHON$(dirname "$(command -v "$PYBIN")")/jvagent, falling back to "$PYBIN" -m jvagent — and ImportError was added to the bootstrap-failure grep so this specific cause names itself next time.

2. Two runs killed each other. Fixed container name, fixed ports, and pkill -f "jvagent $APP_ROOT" in the EXIT trap. An earlier run's cleanup terminated a later run's server mid-restart, producing server failed to restart, login failed post-restart, and a spurious 1 postgres/event-loop error in server logs3 failures, all pointing at the event-loop affinity code, none of them real. Container name and both ports are now per-run unique ($$-seeded, pick_port scans upward), and the server is tracked and torn down by PID.

3. APP_ROOT was cwd-relative and the app .env is gitignored. Run the script from a worktree and you silently get a keyless app. Now resolved from BASH_SOURCE; an explicit positional argument still overrides.

4. The turn check passed on non-empty, not on substantive. This is the one worth reading twice. With no usable model key the agent still answers — by echoing the prompt scaffolding it was handed:

agent: User message: Remember the number 8675309.

[Deliver every MANDATORY directive from the system prompt in full and in order, ...]

[ -n "$r1" ] was satisfied, turn produced a reply passed, TURNS stayed 1, and the run fell over two steps later at the recall check — with a failure that reads as "Postgres did not persist the interaction". The existing "no model key → skip" guard never fired, because there was a reply. reply_is_substantive() now rejects the scaffolding markers and a too-short reply, demotes the turn to SKIP naming the likely cause, and clears TURNS so the dependent checks skip with it.

Also: the workdir is now kept when any check fails, and its path printed. Previously the EXIT trap ran rm -rf "$WORKDIR" unconditionally, so a failure message pointed you at a bootstrap.log that had already been deleted.

Verification

Each run isolates one behaviour, all against merged main:

Run Result
keyed app, explicit root, cwd=/tmp 15 passed, 0 failed, 0 skipped — incl. cross-restart recall from Postgres
keyless app (worktree), default root 12 passed, 0 failed, 2 skipped
keyless app, default root, cwd=/tmp 12 passed, 0 failed, 2 skipped
two runs launched simultaneously ports 55983 / 56023, separate containers, both completed, neither killed the other

The last two are the point. Identical results inside and outside the repo, and zero failures where the same condition previously produced a misleading one — the keyless case used to report 14 passed, 1 failed.

--no-docker is unaffected: it requires JVSPATIAL_POSTGRES_DSN explicitly (the :? guard), so the now-dynamic PGPORT is never read on that path.

Gate: pre-commit run --all-files clean, pytest tests/ 3356 passed / 4 skipped.

eldonm added 2 commits August 8, 2026 11:31
Three script-only defects made smoke_postgres.sh report failures that had
nothing to do with the code under test.

1. Interpreter split. PYBIN came from $PYTHON (scripts/smoke_postgres.sh:43)
   but the checks shelled out to a bare `jvagent` off PATH (:96, :125). On a
   host with more than one install those are different environments; here PATH
   resolved to one whose jvspatial was an editable sibling checkout without
   asyncpg, so bootstrap died with "ImportError: asyncpg is required for the
   Postgres backend" and 8 checks failed spuriously. The entrypoint is now
   derived from PYBIN — $(dirname $PYBIN)/jvagent, falling back to
   "$PYBIN -m jvagent" — and used everywhere.

2. Overlapping runs killed each other. Container name and port were fixed and
   teardown was `pkill -f "jvagent $APP_ROOT"` (:54, :104), which matches
   every concurrent run's server: an earlier run's EXIT trap terminated a later
   run's server mid-restart, yielding "server failed to restart", "login
   failed post-restart" and a bogus event-loop error count. Container name and
   both ports are now per-run unique (PID-seeded, then probed for a free port),
   and the server is tracked and killed by PID instead of by name pattern.

3. Logs deleted on failure. The EXIT trap ran `rm -rf "$WORKDIR"`
   unconditionally (:56), so the bootstrap.log / server_*.log a failure message
   pointed at were already gone. The workdir is now preserved and its path
   printed whenever any check failed.

Verified: 15/15 with no PATH manipulation (PYTHON=.venv/bin/python), and two
deliberately overlapping runs no longer interfere.
…olding replies

Two follow-ups from running the concurrency/interpreter fix in anger.

APP_ROOT defaulted to the cwd-relative "examples/jvagent_app". The app's .env is
gitignored, so running the script from a worktree — or anywhere that is not the
checkout root — silently picks up a different app, or a keyless one. Resolve it
from BASH_SOURCE instead so the default always means "this repo's example app".
An explicit positional argument still overrides it.

The turn check tested only that the reply was non-empty. Without a usable model
key the agent still answers — by echoing the prompt scaffolding it was handed —
so the check passed on output that proved nothing, TURNS stayed 1, and the run
came apart two steps later at the recall check with a failure that pointed at
Postgres rather than at the missing key. That is how a 15/15 suite reported
"14 passed, 1 failed" for a reason that had nothing to do with the database.

reply_is_substantive() rejects the scaffolding markers and a too-short reply.
A scaffolding reply now demotes the turn to SKIP, names the likely cause, and
clears TURNS so the dependent persistence and recall checks skip with it.

Verified three ways against merged main, each isolating one behavior:
- keyed app, explicit root, cwd=/tmp        -> 15 passed, 0 failed, 0 skipped
- keyless app (worktree), default root      -> 12 passed, 0 failed, 2 skipped
- keyless app, default root, cwd=/tmp       -> 12 passed, 0 failed, 2 skipped

The last two are the point: identical results from inside and outside the repo,
and zero failures where the same condition previously produced a misleading one.
@eldonm
eldonm merged commit b7dcc8f into main Aug 8, 2026
3 checks passed
@eldonm
eldonm deleted the fix/smoke-postgres-hardening branch August 8, 2026 17:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant