Skip to content

Support postgres as a Server database type - #35

Merged
eldonm merged 2 commits into
mainfrom
api/postgres-server-support
Aug 4, 2026
Merged

Support postgres as a Server database type#35
eldonm merged 2 commits into
mainfrom
api/postgres-server-support

Conversation

@eldonm

@eldonm eldonm commented Aug 4, 2026

Copy link
Copy Markdown
Member

Type of Change

  • 🐛 Bug Fix
  • 🚀 Feature Request
  • 🔄 Refactor
  • 📖 Documentation Update

Summary

What does this PR address?

Postgres is a documented jvspatial backend — docs/md/postgres-guide.md, neon-deployment.md, and aurora-serverless-deployment.md all describe deploying on it, and the JVSPATIAL_POSTGRES_* env keys are already on the allowlist — but it is unreachable from Server(...). Two independent gaps sit between db_type="postgres" and a working process:

  1. DatabaseConfigurator.initialize_graph_context() rejects the type outright.
  2. Even past that, PostgresDB's pool does not survive the loop handoff that a CLI-then-ASGI-server startup performs.

This PR closes both, so the Server path reaches the backend that already exists underneath it.


Description

Bug Fixes:

Unsupported database type: postgres

  • Bug. Server(db_type="postgres") raises ValueError: Unsupported database type: postgres at construction.
  • Root cause. initialize_graph_context() (jvspatial/api/components/database_configurator.py:177 before this change) dispatches through a hard-coded json / mongodb / sqlite / dynamodb if-chain and raises on anything else. The layer below is complete — create_database("postgres", ...) builds a fully working PostgresDB today — so only the Server-facing branch was missing.
  • Fix. Add the branch, with postgresql accepted as an alias to match db/factory.py.

Pool bound to a dead event loop

  • Bug. After the branch above, a host that bootstraps its graph inside one asyncio.run() and then serves from a second loop fails on its first server-loop query:
    ❌ Database initialization failed: cannot perform operation: another operation is in progress
    asyncpg.exceptions.ConnectionDoesNotExistError: connection was closed in the middle of operation
    
  • Root cause. _ensure_pool() memoizes self._pool — and self._pool_lock is built once in __init__. Both asyncpg.Pool and asyncio.Lock bind to the loop they were created on, so both are unusable once that loop closes. File-backed adapters never notice, which is why this went unseen.
  • Fix. Record the loop the pool belongs to; when _ensure_pool() runs on a different loop, drop the pool and the lock and rebuild. The stale pool's connections are abandoned rather than closed — close() would have to await on a loop that no longer runs — and Postgres reaps them when the sockets drop.

Feature Request:

ServerConfig.database gains postgres_dsn, postgres_min_pool_size, postgres_max_pool_size, and postgres_pooler_mode, mapped from the existing JVSPATIAL_POSTGRES_* env keys in env_adapter.py alongside the mongodb and dynamodb keys.

Motivation is the CONTRIBUTING invariant "Single source of truth for config — don't read environment variables ad-hoc from inside library code, go through the config object." Without these fields the configurator would have to hand create_database a bare db_type and let the driver read env behind the config object's back, leaving Postgres the one backend whose connection settings a host cannot set programmatically. Unset values are omitted from the kwargs so PostgresDB's own defaults (including its env reads) still apply — no behavior change for existing direct create_database users.


Changes Made

High-Level Summary:

  1. jvspatial/api/components/database_configurator.py — add the postgres / postgresql branch plus _resolve_postgres_kwargs().
  2. jvspatial/api/config_groups.py — four postgres_* fields on DatabaseConfig.
  3. jvspatial/env_adapter.py — map JVSPATIAL_POSTGRES_* onto them (string and int keys).
  4. jvspatial/db/postgres.py_pool_loop tracking and _discard_pool_from_dead_loop(); close() clears the loop handle.
  5. Tests — TestPoolLoopAffinity in tests/db/test_postgres_unit.py, postgres cases in tests/api/components/test_database_configurator.py, new tests/test_env_adapter_postgres.py.
  6. Docs — "Using Postgres with Server" and "Event loops" sections in docs/md/postgres-guide.md; CHANGELOG.md under [Unreleased].

No breaking changes. Nothing existing changes shape; the only behavior change on a previously-working path is that a PostgresDB reused across loops now works instead of raising.


Checklist

  • Code follows the project's coding guidelines.
  • Tests have been added or updated for new functionality.
  • Documentation has been updated (if applicable).
  • Existing tests pass locally with these changes. (with the caveat below)
  • Any dependencies introduced are justified and documented. (none — asyncpg remains the existing [postgres] extra)

Each new test was confirmed to fail against main before the corresponding fix landed.

Pre-existing failures on my machine, unrelated to this PR. A full local run shows 8 failures. I re-ran every one against unmodified main and they fail identically there, so this branch introduces none of them:

  • 7 × tests/storage/detect_mime_type() returns application/octet-stream (and once application/SIMH-tape-data) instead of image/png. A libmagic/db difference on this host, not a code issue.
  • 1 × tests/api/endpoints/test_walker_executor.py::TestWalkerExecutor::test_execute_direct_execution_walker — response-envelope mismatch ({'result': ...} vs {'success': True, 'data': {...}}). Order-dependent: it passes when run alone and fails under the full tests/api run — on main as well as here. Might be worth a separate look; I left it alone since it's outside this change.

Separately, and also on main: the interpreter hangs at exit after the suite finishes (Py_FinalizeExwait_for_thread_shutdown), so a non-daemon thread isn't joining. Results are complete before it hangs, but pytest ... | tail never flushes. Untouched by this PR; flagging in case it's news.


Steps to Test

docker run -d --name jvspatial-pg \
  -e POSTGRES_USER=jvspatial -e POSTGRES_PASSWORD=jvspatial -e POSTGRES_DB=jvdb \
  -p 55432:5432 postgres:16-alpine

pip install -e '.[dev,test,postgres]'
pytest tests/db/test_postgres_unit.py tests/test_env_adapter_postgres.py \
       tests/api/components/test_database_configurator.py -q

Unit coverage needs no live database. For the loop-handoff path specifically, the shape that used to fail is:

db = create_database("postgres", dsn="postgresql://jvspatial:jvspatial@localhost:55432/jvdb")
asyncio.run(bootstrap(db))   # loop 1
asyncio.run(serve(db))       # loop 2 — previously ConnectionDoesNotExistError

Additional Context

Found while bringing up jvagent on Postgres — it is exactly the two-loop host described above (CLI bootstraps the application graph under asyncio.run(), then hands off to uvicorn). Both gaps reproduce identically on 0.0.9, 0.0.12, and 0.0.15.

End-to-end verification against this branch, with no patches on the jvagent side:

  • jvagent <app> bootstrap persists the full application graph to Postgres — node / edge / object tables and their indexes created unattended.
  • Server starts clean; /health reports "database":"connected", lifecycle logs 📊 Database: PostgresDB | 🌳 Root: n.Root.root.
  • Admin bootstrap writes a user; JWT login authenticates against it.
  • Agent turns run end to end, creating the per-user memory subgraph.
  • Conversation state survives a full server restart — the restart is what exercises the loop-affinity fix, and it is the case that failed before it.

Full suite green locally, pre-commit run --all-files clean.


Questions or Concerns

One judgment call worth a reviewer's eye: the stale pool's connections are abandoned, not closed, on a loop change. Closing them would require awaiting on the dead loop, which isn't possible from the new one. In the startup-handoff case this leaks at most one pool's worth of connections once per process, and Postgres reaps them when the sockets drop — but if you'd rather see an explicit min_size=0 recommendation for that window, or a warning log instead of debug, say the word.

Happy to split this into two PRs (db/ fix and api/ support) if you'd prefer them reviewed separately — they're independent, though Postgres-on-Server needs both to actually work.

eldonm added 2 commits August 4, 2026 14:16
asyncpg pools and asyncio.Lock both bind to the loop they were created
on, but _ensure_pool() memoized the pool and lock for the lifetime of the
PostgresDB instance. A host that bootstraps inside one asyncio.run() and
then serves from a second loop -- the common CLI-then-ASGI-server startup
-- carried a pool bound to the first, now-closed loop into the server
loop, and its first query failed with "cannot perform operation: another
operation is in progress" / ConnectionDoesNotExistError.

Track the loop the pool belongs to and drop both pool and lock when the
running loop differs. The stale pool's connections are abandoned rather
than closed, since close() would have to await on a loop that no longer
runs; Postgres reaps them when the sockets drop.

File-backed adapters never hit this, which is why it went unnoticed:
it only bites the Server path on postgres.
DatabaseConfigurator.initialize_graph_context() dispatched db_type
through a hard-coded json/mongodb/sqlite/dynamodb chain and raised
"Unsupported database type: postgres" for anything else. PostgresDB and
create_database("postgres", ...) were already complete, and the
JVSPATIAL_POSTGRES_* keys were already allowlisted -- but Server() is the
only path most deployments use, so a documented backend was unreachable
from the API layer and from anything built on it.

Add the missing branch, and give ServerConfig.database the settings it
needs to describe the connection: postgres_dsn, postgres_min_pool_size,
postgres_max_pool_size, postgres_pooler_mode, mapped from env in
env_adapter alongside the mongodb and dynamodb keys. Connection settings
now reach the driver through the config object rather than only through
the driver's own env reads. Unset values are omitted so PostgresDB's
defaults still apply.

postgresql is accepted as an alias for postgres, matching the factory.
@github-actions

github-actions Bot commented Aug 4, 2026

Copy link
Copy Markdown

Benchmark comparison

Threshold: ±25% (informational, does not block merge)

benchmark baseline (s) current (s) delta status
tests/benchmarks/test_deferred_save_benchmarks.py::test_bench_deferred_save_batched_100 0.048122 0.053379 +10.9% OK
tests/benchmarks/test_deferred_save_benchmarks.py::test_bench_immediate_save_100 0.049047 0.051965 +5.9% OK
tests/benchmarks/test_jsondb_benchmarks.py::test_bench_jsondb_batched_saves_500 0.449596 0.539528 +20.0% OK
tests/benchmarks/test_jsondb_benchmarks.py::test_bench_jsondb_count_empty_query 0.959360 1.143833 +19.2% OK
tests/benchmarks/test_jsondb_benchmarks.py::test_bench_jsondb_count_filtered 1.099044 1.250062 +13.7% OK
tests/benchmarks/test_jsondb_benchmarks.py::test_bench_jsondb_find_filtered 0.839041 0.994021 +18.5% OK
tests/benchmarks/test_jsondb_benchmarks.py::test_bench_jsondb_save_throughput 0.001583 0.001716 +8.4% OK
tests/benchmarks/test_sqlite_benchmarks.py::test_bench_sqlite_count_empty 0.365094 0.370237 +1.4% OK
tests/benchmarks/test_sqlite_benchmarks.py::test_bench_sqlite_count_fallback_via_regex 0.388536 0.404907 +4.2% OK
tests/benchmarks/test_sqlite_benchmarks.py::test_bench_sqlite_count_pushdown 0.353934 0.424180 +19.8% OK
tests/benchmarks/test_sqlite_benchmarks.py::test_bench_sqlite_find_fallback_via_regex 0.385815 0.402613 +4.4% OK
tests/benchmarks/test_sqlite_benchmarks.py::test_bench_sqlite_find_pushdown 0.343674 0.366008 +6.5% OK
tests/benchmarks/test_sqlite_benchmarks.py::test_bench_sqlite_sort_limit_pushdown 0.401564 0.411326 +2.4% OK

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant