From fb6c815524a81d40b594fe8573f1e6c9e59e5da6 Mon Sep 17 00:00:00 2001 From: Eldon Marks Date: Tue, 4 Aug 2026 16:26:53 -0400 Subject: [PATCH] docs(postgres): document the PostgreSQL backend and its upstream blockers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit jvspatial ships a complete PostgresDB backend and jvagent needs no code changes to use it, but `JVSPATIAL_DB_TYPE=postgres` fails at startup: DatabaseConfigurator.initialize_graph_context() rejects the type (jvspatial database_configurator.py:177), and PostgresDB._ensure_pool() has no event-loop affinity, which jvagent's bootstrap-then-uvicorn boot trips. Both reproduce on jvspatial 0.0.9, 0.0.12, and 0.0.15 (the pin). Nothing in the docs said so. .env.example listed four backends, and .planning/reference/jvspatial-integration.md §2.5 the same four, so the only way to learn Postgres was unavailable was to try it and read a traceback. Add docs/postgres.md: the full config surface (all four JVSPATIAL_POSTGRES_* keys, verified defaults), local docker setup, both blockers with file:line and exact errors, and what was verified working once patched. Documented as blocked rather than supported -- the fix belongs upstream (CLAUDE.md section 4), and TrueSelph/jvspatial#35 is open for it. Also flag two quieter traps: the DSN reaches the driver via env only (server_config.py:126-164 threads a connection string for mongodb but has no postgres field), and the log DB silently falls back to a json file log for any type jvspatial does not recognise, postgres included. --- .env.example | 14 +++ .planning/reference/jvspatial-integration.md | 5 +- CLAUDE.md | 1 + docs/configuration.md | 1 + docs/environment-keys-reference.md | 4 + docs/postgres.md | 115 +++++++++++++++++++ 6 files changed, 139 insertions(+), 1 deletion(-) create mode 100644 docs/postgres.md diff --git a/.env.example b/.env.example index 02337952..e22565f0 100644 --- a/.env.example +++ b/.env.example @@ -80,6 +80,17 @@ TYPESENSE_API_KEY=your-typesense-api-key-here # AWS_ACCESS_KEY_ID= # AWS_SECRET_ACCESS_KEY= +# PostgreSQL — NOT USABLE YET: blocked by two jvspatial gaps (jvspatial 0.0.15). +# Setting JVSPATIAL_DB_TYPE=postgres fails at startup with +# "ValueError: Unsupported database type: postgres". See docs/postgres.md. +# Requires the asyncpg driver (pip install asyncpg) once unblocked. +# The DSN is read from the environment only — app.yaml database.uri is not wired. +# JVSPATIAL_DB_TYPE=postgres +# JVSPATIAL_POSTGRES_DSN=postgresql://user:password@localhost:5432/jvagent_db +# JVSPATIAL_POSTGRES_MIN_POOL_SIZE=2 +# JVSPATIAL_POSTGRES_MAX_POOL_SIZE=10 +# JVSPATIAL_POSTGRES_POOLER_MODE=session # use 'transaction' behind PgBouncer / RDS Proxy + # ============================================================================ # LOGGING DATABASE (jvspatial: JVSPATIAL_DB_LOGGING_* / JVSPATIAL_LOG_DB_*) # ============================================================================ @@ -90,6 +101,9 @@ TYPESENSE_API_KEY=your-typesense-api-key-here # JVSPATIAL_LOG_DB_NAME=jvagent_logs # JVSPATIAL_LOG_DB_PATH=./jvagent_logs # JVSPATIAL_LOG_RETENTION_DEFAULT_DAYS=60 # Log retention in days (0 = indefinite); shared with jvspatial load_env +# NOTE: JVSPATIAL_LOG_DB_TYPE defaults to JVSPATIAL_DB_TYPE. jvspatial has no +# postgres branch for the log DB and silently falls back to a json file log, so +# set this explicitly (json or mongodb) if the main graph DB is postgres. # DynamoDB logging (when JVSPATIAL_LOG_DB_TYPE=dynamodb) # JVSPATIAL_LOG_DB_TABLE_NAME=jvagent_logs diff --git a/.planning/reference/jvspatial-integration.md b/.planning/reference/jvspatial-integration.md index 96df085d..9b446db4 100644 --- a/.planning/reference/jvspatial-integration.md +++ b/.planning/reference/jvspatial-integration.md @@ -79,7 +79,7 @@ async def my_handler(...): ... ### 2.5 Persistence -jvspatial supports four backends, selected via env vars: +jvspatial supports four backends usable from jvagent, selected via env vars: | Backend | Use case | Env | |---|---|---| @@ -87,6 +87,9 @@ jvspatial supports four backends, selected via env vars: | **SQLite** | Single-process serverless / embedded | `JVSPATIAL_DB_TYPE=sqlite` | | **MongoDB** | Production, multi-process | `JVSPATIAL_DB_TYPE=mongodb`, `JVSPATIAL_MONGODB_URI`, `JVSPATIAL_MONGODB_DB_NAME` | | **DynamoDB** | AWS Lambda / serverless | `JVSPATIAL_DB_TYPE=dynamodb`, table + AWS creds | +| **PostgreSQL** | ⚠️ *blocked upstream* | `JVSPATIAL_DB_TYPE=postgres`, `JVSPATIAL_POSTGRES_DSN` | + +A fifth backend, `PostgresDB`, is implemented in jvspatial (`jvspatial/db/postgres.py`) and reachable via `create_database("postgres")`, but `DatabaseConfigurator.initialize_graph_context()` — the path `Server(...)` uses — rejects it with `ValueError: Unsupported database type: postgres` (`jvspatial/api/components/database_configurator.py:177`, unchanged through 0.0.15), and `PostgresDB._ensure_pool()` has no event-loop affinity, which jvagent's two-loop boot trips. Both are jvspatial's to fix per [§4](#4-the-boundary); details and verification in [`docs/postgres.md`](../../docs/postgres.md). CRUD via entity methods (no separate ORM): ```python diff --git a/CLAUDE.md b/CLAUDE.md index ed22a559..796c1a61 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -37,6 +37,7 @@ Use cases: turn-based chatbots, channel adapters (WhatsApp / Messenger / email / | **Understand memory pruning** | [`.planning/reference/memory-and-pruning.md`](.planning/reference/memory-and-pruning.md) | | **Tune / query logging** | [`.planning/reference/observability.md`](.planning/reference/observability.md) + [`docs/logging.md`](docs/logging.md) | | **Find a config key** | [`.planning/reference/configuration-keys.md`](.planning/reference/configuration-keys.md) + [`docs/environment-keys-reference.md`](docs/environment-keys-reference.md) | +| **Run on PostgreSQL** (blocked upstream) | [`docs/postgres.md`](docs/postgres.md) | | **Understand the Orchestrator pattern** | [`docs/ORCHESTRATOR.md`](docs/ORCHESTRATOR.md) + ADRs [0012](.planning/adr/0012-skill-executive-architecture.md) (architecture), [0013](.planning/adr/0013-togglable-deterministic-turn-lock.md) (turn-lock), [0014](.planning/adr/0014-identity-on-agent-replyaction-egress.md) (identity/egress), [0015](.planning/adr/0015-skill-executive-configuration-surface.md) (config surface), [0016](.planning/adr/0016-model-gearing-light-heavy.md) (model gearing), [0017](.planning/adr/0017-two-skill-specs-code-execution-substrate.md) (skill specs + code execution), [0018](.planning/adr/0018-lean-tool-surfacing.md) (lean surfacing), [0019](.planning/adr/0019-orchestrator-resumable-plan.md) (resumable plan), [0041](.planning/adr/0041-gearing-and-cost-policy-in-core.md) (gearing/cost in core), [0042](.planning/adr/0042-session-context-ground-truth.md) (session clock/channel) | | **Document conversational test scenarios (CUCS)** | [`.planning/reference/conversation-use-cases.md`](.planning/reference/conversation-use-cases.md) + [ADR-0027](.planning/adr/0027-conversation-use-case-spec.md) | | **Run jvagent locally** | [`.planning/runbooks/local-dev.md`](.planning/runbooks/local-dev.md) | diff --git a/docs/configuration.md b/docs/configuration.md index 307f36c8..a6bb315c 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -131,5 +131,6 @@ Do not commit real secret values in `app.yaml`. - [task-tracking.md](task-tracking.md) - `PROACTIVE` queue, `TaskMonitor`, scheduler bootstrap. - [proactive-messages.md](proactive-messages.md) - Canned `send_proactive_message` vs queued agentic tasks. - [environment-keys-reference.md](environment-keys-reference.md) - Canonical env key inventory. +- [postgres.md](postgres.md) - PostgreSQL backend keys and the upstream gaps that currently block it. - [integrations-environment.md](integrations-environment.md) - Integration/vendor env keys. - [scaffolding.md](scaffolding.md) - CLI app and agent scaffolding flow. diff --git a/docs/environment-keys-reference.md b/docs/environment-keys-reference.md index 1fb5c9e4..b36f5124 100644 --- a/docs/environment-keys-reference.md +++ b/docs/environment-keys-reference.md @@ -95,6 +95,10 @@ These are commonly used by `jvagent` and should be configured in `jvagent` deplo - `JVSPATIAL_DYNAMODB_TABLE_NAME` - `JVSPATIAL_DYNAMODB_REGION` - `JVSPATIAL_DYNAMODB_ENDPOINT_URL` +- `JVSPATIAL_POSTGRES_DSN` - PostgreSQL DSN. Env-only (no `app.yaml` path). Blocked upstream — see [postgres.md](postgres.md). +- `JVSPATIAL_POSTGRES_MIN_POOL_SIZE` +- `JVSPATIAL_POSTGRES_MAX_POOL_SIZE` +- `JVSPATIAL_POSTGRES_POOLER_MODE` - `session` (default) or `transaction` for PgBouncer / RDS Proxy. ### Auth/rate limit (framework side) - `JVSPATIAL_JWT_SECRET_KEY` diff --git a/docs/postgres.md b/docs/postgres.md new file mode 100644 index 00000000..a724c83e --- /dev/null +++ b/docs/postgres.md @@ -0,0 +1,115 @@ +# Running jvagent on PostgreSQL + +> **Status: blocked upstream.** jvagent itself needs no changes to run on Postgres, and jvspatial ships a complete `PostgresDB` backend — but the code path `jvagent` uses to build its `Server` does not accept `db_type=postgres`. Two upstream gaps must be fixed in jvspatial first; both are described in [§3](#3-upstream-blockers). Until then, use `json`, `sqlite`, `mongodb`, or `dynamodb`. +> +> Verified against `jvspatial==0.0.15` (the pin in [`pyproject.toml`](../pyproject.toml)), and reproduced identically on 0.0.9 and 0.0.12. + +--- + +## 1. Configuration surface + +Postgres is selected the same way as every other backend — `JVSPATIAL_DB_TYPE`, resolved with the usual precedence (env > `app.yaml` > default; see [configuration.md](configuration.md)). + +| Key | Purpose | Default | +|---|---|---| +| `JVSPATIAL_DB_TYPE` | `postgres` (alias: `postgresql`) | `json` | +| `JVSPATIAL_POSTGRES_DSN` | libpq DSN, e.g. `postgresql://user:pass@host:5432/dbname` | `postgresql://postgres:postgres@localhost:5432/jvdb` | +| `JVSPATIAL_POSTGRES_MIN_POOL_SIZE` | asyncpg pool floor | 0 (serverless) / 2 | +| `JVSPATIAL_POSTGRES_MAX_POOL_SIZE` | asyncpg pool ceiling | 3 (serverless) / 10 | +| `JVSPATIAL_POSTGRES_POOLER_MODE` | `session` or `transaction` — set `transaction` behind PgBouncer / RDS Proxy (disables asyncpg's prepared-statement cache) | `session` | + +Notes: + +- **The driver is an extra.** `pip install asyncpg` (or `jvspatial[postgres]`); it is not pulled in by jvagent's base dependencies. +- **`JVSPATIAL_DB_PATH` does not apply.** [`server_config.py:161`](../jvagent/cli/server_config.py) only exports a path for `json` / `sqlite`. +- **The DSN reaches the driver via env only.** [`server_config.py:126-164`](../jvagent/cli/server_config.py) threads a connection string into jvspatial's `DatabaseConfig` for `mongodb` and table/region for `dynamodb`; there is no Postgres field, so `JVSPATIAL_POSTGRES_DSN` must be set in the environment — an `app.yaml` `database.uri` will **not** be picked up. +- **Logging DB has no Postgres branch.** jvspatial's `logging/config.py` falls through to a `json` file log for any unrecognized type — silently. Set `JVSPATIAL_LOG_DB_TYPE=json` (or `mongodb`) explicitly so the fallback is a decision rather than a surprise. See [logging.md](logging.md). +- **PageIndex is a separate store** with its own `JVAGENT_PAGEINDEX_DB_TYPE` (`json` by default) and is unaffected by the main graph backend. + +Schema is created by the driver on first use — three tables (`node`, `edge`, `object`) plus indexes. Nothing to migrate by hand. + +--- + +## 2. Local setup + +```bash +docker run -d --name jvagent-pg \ + -e POSTGRES_USER=jvagent -e POSTGRES_PASSWORD=jvagent -e POSTGRES_DB=jvagent_demo \ + -p 55432:5432 postgres:16-alpine +``` + +```bash +pip install asyncpg +``` + +Then in your app's `.env` (see [`.env.example`](../.env.example)): + +```bash +JVSPATIAL_DB_TYPE=postgres +JVSPATIAL_POSTGRES_DSN=postgresql://jvagent:jvagent@localhost:55432/jvagent_demo +JVSPATIAL_LOG_DB_TYPE=json +``` + +```bash +jvagent examples/jvagent_app bootstrap +``` + +As of `jvspatial==0.0.15` that bootstrap **fails**: + +``` +❌ Failed to initialize GraphContext: Unsupported database type: postgres +ValueError: Unsupported database type: postgres +``` + +--- + +## 3. Upstream blockers + +Both live in jvspatial. Per [`CLAUDE.md`](../CLAUDE.md) §4 and [`adr/0006`](../.planning/adr/0006-jvspatial-dependency.md), database adapter behavior is jvspatial's to own — do **not** work around either of these inside jvagent. + +### 3.1 `Server` path rejects `postgres` + +`jvspatial/api/components/database_configurator.py:177` — `DatabaseConfigurator.initialize_graph_context()` dispatches on `db_type` through a hard-coded `json` / `mongodb` / `sqlite` / `dynamodb` if-chain and raises `ValueError: Unsupported database type: {db_type}` for anything else. + +This is the path `Server(...)` takes, and therefore the path [`create_server_from_config()`](../jvagent/cli/server_config.py) takes. The layer beneath it is complete: `jvspatial/db/factory.py` handles `("postgres", "postgresql")`, reads `JVSPATIAL_POSTGRES_*` from env, and returns a fully functional `PostgresDB`. Calling `create_database("postgres")` directly and driving `Root` / `Node` CRUD against Postgres works today. + +**Fix:** add the missing `elif db_type in ("postgres", "postgresql")` branch to `initialize_graph_context()`, delegating to `create_database`. + +### 3.2 asyncpg pool has no event-loop affinity + +`jvspatial/db/postgres.py:353` — `PostgresDB._ensure_pool()` memoizes `self._pool` (created under `self._pool_lock`, itself constructed in `__init__`) and never revalidates which event loop that pool belongs to. + +This breaks jvagent specifically, because jvagent boots across **two** loops: the CLI bootstraps the application graph inside `asyncio.run(...)` ([`cli/main.py`](../jvagent/cli/main.py)), then hands off to uvicorn, which runs its own loop. The pool created during bootstrap stays bound to the first, now-dead loop, and the first query on the server loop fails: + +``` +❌ Database initialization failed: cannot perform operation: another operation is in progress +asyncpg.exceptions.ConnectionDoesNotExistError: connection was closed in the middle of operation +``` + +File-based backends never notice; this is the same class of bug as the per-event-loop lock pattern jvagent already uses at [`core/app.py:100-124`](../jvagent/core/app.py). + +**Fix:** give `_ensure_pool()` loop affinity — record the loop the pool was created on, and when the running loop differs, drop the stale pool and its lock and rebuild. + +--- + +## 4. What was verified + +With both gaps patched at runtime (monkeypatch only — no jvagent or jvspatial source changed), against `jvspatial==0.0.15` + `asyncpg==0.31.0` + `postgres:16-alpine`: + +- `jvagent examples/jvagent_app bootstrap` completed; `App`, `Agents`, both example agents, and every installed action persisted to Postgres — 44 `node` rows, 43 `edge` rows, 3 `object` rows after bootstrap plus two conversation turns, with the driver creating all three tables and their indexes unattended. +- Server started clean — `/health` reported `"database":"connected"`, lifecycle logged `📊 Database: PostgresDB | 🌳 Root: n.Root.root`. +- Admin bootstrap wrote a `User` to Postgres; `POST /api/auth/login` returned a JWT against it. +- `POST /api/agents/{id}/interact` ran full turns on the orchestrator example agent (`OrchestratorInteractAction` → `ReplyAction`), creating `User` / `Conversation` / `Interaction` nodes. +- Conversation state survived a **full server restart** — a later turn recalled a value stated before the restart, read back out of Postgres rather than process memory. + +So the blockers are strictly at the configuration boundary. Once jvspatial accepts `postgres` in the `Server` path and makes its pool loop-aware, jvagent runs on Postgres as-is. + +--- + +## 5. Related documentation + +- [configuration.md](configuration.md) — config precedence and the `app.yaml` / env split. +- [environment-keys-reference.md](environment-keys-reference.md) — canonical env key inventory. +- [database-indexing.md](database-indexing.md) — `ensure_indexes`, `JVSPATIAL_AUTO_CREATE_INDEXES`, per-backend behavior. +- [logging.md](logging.md) — log DB selection and retention. +- [`.planning/reference/jvspatial-integration.md`](../.planning/reference/jvspatial-integration.md) §2.5 — the backend matrix and the jvagent/jvspatial boundary.