Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 2 additions & 3 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -80,10 +80,9 @@ TYPESENSE_API_KEY=your-typesense-api-key-here
# AWS_ACCESS_KEY_ID=
# AWS_SECRET_ACCESS_KEY=

# PostgreSQL — NOT USABLE YET: blocked by two jvspatial gaps (jvspatial 0.0.15).
# Setting JVSPATIAL_DB_TYPE=postgres fails at startup with
# PostgreSQL — requires jvspatial >= 0.0.16 and the asyncpg driver
# (pip install asyncpg). On 0.0.15 and earlier this fails at startup with
# "ValueError: Unsupported database type: postgres". See docs/postgres.md.
# Requires the asyncpg driver (pip install asyncpg) once unblocked.
# The DSN is read from the environment only — app.yaml database.uri is not wired.
# JVSPATIAL_DB_TYPE=postgres
# JVSPATIAL_POSTGRES_DSN=postgresql://user:password@localhost:5432/jvagent_db
Expand Down
10 changes: 5 additions & 5 deletions .planning/reference/jvspatial-integration.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
## 1. Where jvspatial lives

- **Source**: `/Users/eldonmarks/Briefcase/dev/jv/jvspatial` (sibling directory in this workspace).
- **Pip install**: declared in [`pyproject.toml`](../../pyproject.toml) as `jvspatial==0.0.12`.
- **Pip install**: declared in [`pyproject.toml`](../../pyproject.toml) as `jvspatial==0.0.16`.
- **Own docs**: jvspatial has its own [`README.md`](../../../jvspatial/README.md) and [`SPEC.md`](../../../jvspatial/SPEC.md). Treat those as authoritative for anything below.

---
Expand Down Expand Up @@ -79,17 +79,17 @@ async def my_handler(...): ...

### 2.5 Persistence

jvspatial supports four backends usable from jvagent, selected via env vars:
jvspatial supports five backends usable from jvagent, selected via env vars:

| Backend | Use case | Env |
|---|---|---|
| **JSON** | Dev, demos, tests | `JVSPATIAL_DB_TYPE=json`, `JVSPATIAL_JSONDB_PATH=./jvdb/dev` |
| **SQLite** | Single-process serverless / embedded | `JVSPATIAL_DB_TYPE=sqlite` |
| **MongoDB** | Production, multi-process | `JVSPATIAL_DB_TYPE=mongodb`, `JVSPATIAL_MONGODB_URI`, `JVSPATIAL_MONGODB_DB_NAME` |
| **DynamoDB** | AWS Lambda / serverless | `JVSPATIAL_DB_TYPE=dynamodb`, table + AWS creds |
| **PostgreSQL** | ⚠️ *blocked upstream* | `JVSPATIAL_DB_TYPE=postgres`, `JVSPATIAL_POSTGRES_DSN` |
| **PostgreSQL** | Production, multi-process; managed Postgres (Neon / Aurora) | `JVSPATIAL_DB_TYPE=postgres`, `JVSPATIAL_POSTGRES_DSN` (requires `jvspatial >= 0.0.16` + `asyncpg`) |

A fifth backend, `PostgresDB`, is implemented in jvspatial (`jvspatial/db/postgres.py`) and reachable via `create_database("postgres")`, but `DatabaseConfigurator.initialize_graph_context()` — the path `Server(...)` uses — rejects it with `ValueError: Unsupported database type: postgres` (`jvspatial/api/components/database_configurator.py:177`, unchanged through 0.0.15), and `PostgresDB._ensure_pool()` has no event-loop affinity, which jvagent's two-loop boot trips. Both are jvspatial's to fix per [§4](#4-the-boundary); details and verification in [`docs/postgres.md`](../../docs/postgres.md).
Postgres was unreachable from `Server(...)` before jvspatial 0.0.16 — the config path rejected the type, and the asyncpg pool did not survive jvagent's bootstrap-loop → uvicorn-loop handoff. Both were fixed upstream in 0.0.16; setup, the full key surface, and a smoke script are in [`docs/postgres.md`](../../docs/postgres.md).

CRUD via entity methods (no separate ORM):
```python
Expand Down Expand Up @@ -171,7 +171,7 @@ Things jvagent **owns**:

## 5. Version policy

- Minimum required jvspatial: pinned in [`pyproject.toml`](../../pyproject.toml) as `jvspatial==X.Y.Z`. Current: `==0.0.12`.
- Minimum required jvspatial: pinned in [`pyproject.toml`](../../pyproject.toml) as `jvspatial==X.Y.Z`. Current: `==0.0.16`.
- When jvspatial introduces breaking changes (e.g., walker API rename, persistence shape change), bump the pin and update this section.
- When adding a new dependency on a jvspatial feature, document the symbol + version it was introduced in. Helps downstream consumers know the floor.
- Rationale: [`adr/0006-jvspatial-dependency.md`](../adr/0006-jvspatial-dependency.md).
Expand Down
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ Use cases: turn-based chatbots, channel adapters (WhatsApp / Messenger / email /
| **Understand memory pruning** | [`.planning/reference/memory-and-pruning.md`](.planning/reference/memory-and-pruning.md) |
| **Tune / query logging** | [`.planning/reference/observability.md`](.planning/reference/observability.md) + [`docs/logging.md`](docs/logging.md) |
| **Find a config key** | [`.planning/reference/configuration-keys.md`](.planning/reference/configuration-keys.md) + [`docs/environment-keys-reference.md`](docs/environment-keys-reference.md) |
| **Run on PostgreSQL** (blocked upstream) | [`docs/postgres.md`](docs/postgres.md) |
| **Run on PostgreSQL** | [`docs/postgres.md`](docs/postgres.md) |
| **Understand the Orchestrator pattern** | [`docs/ORCHESTRATOR.md`](docs/ORCHESTRATOR.md) + ADRs [0012](.planning/adr/0012-skill-executive-architecture.md) (architecture), [0013](.planning/adr/0013-togglable-deterministic-turn-lock.md) (turn-lock), [0014](.planning/adr/0014-identity-on-agent-replyaction-egress.md) (identity/egress), [0015](.planning/adr/0015-skill-executive-configuration-surface.md) (config surface), [0016](.planning/adr/0016-model-gearing-light-heavy.md) (model gearing), [0017](.planning/adr/0017-two-skill-specs-code-execution-substrate.md) (skill specs + code execution), [0018](.planning/adr/0018-lean-tool-surfacing.md) (lean surfacing), [0019](.planning/adr/0019-orchestrator-resumable-plan.md) (resumable plan), [0041](.planning/adr/0041-gearing-and-cost-policy-in-core.md) (gearing/cost in core), [0042](.planning/adr/0042-session-context-ground-truth.md) (session clock/channel) |
| **Document conversational test scenarios (CUCS)** | [`.planning/reference/conversation-use-cases.md`](.planning/reference/conversation-use-cases.md) + [ADR-0027](.planning/adr/0027-conversation-use-case-spec.md) |
| **Run jvagent locally** | [`.planning/runbooks/local-dev.md`](.planning/runbooks/local-dev.md) |
Expand Down
2 changes: 1 addition & 1 deletion docs/environment-keys-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ These are commonly used by `jvagent` and should be configured in `jvagent` deplo
- `JVSPATIAL_DYNAMODB_TABLE_NAME`
- `JVSPATIAL_DYNAMODB_REGION`
- `JVSPATIAL_DYNAMODB_ENDPOINT_URL`
- `JVSPATIAL_POSTGRES_DSN` - PostgreSQL DSN. Env-only (no `app.yaml` path). Blocked upstream — see [postgres.md](postgres.md).
- `JVSPATIAL_POSTGRES_DSN` - PostgreSQL DSN. Env-only (no `app.yaml` path). Requires `jvspatial >= 0.0.16` — see [postgres.md](postgres.md).
- `JVSPATIAL_POSTGRES_MIN_POOL_SIZE`
- `JVSPATIAL_POSTGRES_MAX_POOL_SIZE`
- `JVSPATIAL_POSTGRES_POOLER_MODE` - `session` (default) or `transaction` for PgBouncer / RDS Proxy.
Expand Down
60 changes: 27 additions & 33 deletions docs/postgres.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# Running jvagent on PostgreSQL

> **Status: blocked upstream.** jvagent itself needs no changes to run on Postgres, and jvspatial ships a complete `PostgresDB` backend — but the code path `jvagent` uses to build its `Server` does not accept `db_type=postgres`. Two upstream gaps must be fixed in jvspatial first; both are described in [§3](#3-upstream-blockers). Until then, use `json`, `sqlite`, `mongodb`, or `dynamodb`.
> **Requires `jvspatial >= 0.0.16`** (the pin in [`pyproject.toml`](../pyproject.toml)). On 0.0.15 and earlier, starting jvagent with `JVSPATIAL_DB_TYPE=postgres` fails outright — see [§3](#3-version-requirement).
>
> Verified against `jvspatial==0.0.15` (the pin in [`pyproject.toml`](../pyproject.toml)), and reproduced identically on 0.0.9 and 0.0.12.
> jvagent needs no Postgres-specific code: the backend and every connection setting live in jvspatial. Verified end to end against the released 0.0.16 wheel — see [§4](#4-verifying-your-setup).

---

Expand Down Expand Up @@ -52,57 +52,51 @@ JVSPATIAL_LOG_DB_TYPE=json

```bash
jvagent examples/jvagent_app bootstrap
jvagent examples/jvagent_app
```

As of `jvspatial==0.0.15` that bootstrap **fails**:
The driver creates its schema on first use, so there is no migration step between those two commands.

---

## 3. Version requirement

Postgres works from `jvspatial >= 0.0.16`. On **0.0.15 and earlier**, `Server` construction fails before the app ever boots:

```
❌ Failed to initialize GraphContext: Unsupported database type: postgres
ValueError: Unsupported database type: postgres
```

---

## 3. Upstream blockers
Two defects had to be fixed upstream, both shipped in 0.0.16 ([TrueSelph/jvspatial#35](https://github.com/TrueSelph/jvspatial/pull/35)). They are worth knowing about because the second one is invisible until a restart:

Both live in jvspatial. Per [`CLAUDE.md`](../CLAUDE.md) §4 and [`adr/0006`](../.planning/adr/0006-jvspatial-dependency.md), database adapter behavior is jvspatial's to own — do **not** work around either of these inside jvagent.
1. **`Server` rejected the type.** `DatabaseConfigurator.initialize_graph_context()` dispatched `db_type` through a hard-coded `json`/`mongodb`/`sqlite`/`dynamodb` chain. The backend underneath always worked — `create_database("postgres", ...)` was fine — so only the `Server` path, the one [`create_server_from_config()`](../jvagent/cli/server_config.py) uses, was closed.

### 3.1 `Server` path rejects `postgres`
2. **The asyncpg pool had no event-loop affinity.** `PostgresDB._ensure_pool()` memoized the pool and its lock for the life of the instance. This bites jvagent specifically, because jvagent boots across **two** loops: the CLI bootstraps the graph inside `asyncio.run(...)` ([`cli/main.py`](../jvagent/cli/main.py)), then hands off to uvicorn's own loop. The pool built during bootstrap stayed bound to the first, now-dead loop, and the first query on the server loop died with:

`jvspatial/api/components/database_configurator.py:177` — `DatabaseConfigurator.initialize_graph_context()` dispatches on `db_type` through a hard-coded `json` / `mongodb` / `sqlite` / `dynamodb` if-chain and raises `ValueError: Unsupported database type: {db_type}` for anything else.
```
cannot perform operation: another operation is in progress
asyncpg.exceptions.ConnectionDoesNotExistError: connection was closed in the middle of operation
```

This is the path `Server(...)` takes, and therefore the path [`create_server_from_config()`](../jvagent/cli/server_config.py) takes. The layer beneath it is complete: `jvspatial/db/factory.py` handles `("postgres", "postgresql")`, reads `JVSPATIAL_POSTGRES_*` from env, and returns a fully functional `PostgresDB`. Calling `create_database("postgres")` directly and driving `Root` / `Node` CRUD against Postgres works today.
File-backed adapters never notice this. It is the same class of bug as the per-event-loop lock pattern jvagent already uses at [`core/app.py:100-124`](../jvagent/core/app.py).

**Fix:** add the missing `elif db_type in ("postgres", "postgresql")` branch to `initialize_graph_context()`, delegating to `create_database`.
Per [`CLAUDE.md`](../CLAUDE.md) §4 and [`adr/0006`](../.planning/adr/0006-jvspatial-dependency.md), database adapter behavior is jvspatial's to own — if Postgres misbehaves, fix it there rather than working around it in jvagent.

### 3.2 asyncpg pool has no event-loop affinity
---

`jvspatial/db/postgres.py:353` — `PostgresDB._ensure_pool()` memoizes `self._pool` (created under `self._pool_lock`, itself constructed in `__init__`) and never revalidates which event loop that pool belongs to.
## 4. Verifying your setup

This breaks jvagent specifically, because jvagent boots across **two** loops: the CLI bootstraps the application graph inside `asyncio.run(...)` ([`cli/main.py`](../jvagent/cli/main.py)), then hands off to uvicorn, which runs its own loop. The pool created during bootstrap stays bound to the first, now-dead loop, and the first query on the server loop fails:
[`scripts/smoke_postgres.sh`](../scripts/smoke_postgres.sh) drives a real app against a real database and checks what unit tests cannot:

```bash
scripts/smoke_postgres.sh # spins up its own container
scripts/smoke_postgres.sh path/to/your_app # against your app
```
❌ Database initialization failed: cannot perform operation: another operation is in progress
asyncpg.exceptions.ConnectionDoesNotExistError: connection was closed in the middle of operation
```

File-based backends never notice; this is the same class of bug as the per-event-loop lock pattern jvagent already uses at [`core/app.py:100-124`](../jvagent/core/app.py).

**Fix:** give `_ensure_pool()` loop affinity — record the loop the pool was created on, and when the running loop differs, drop the stale pool and its lock and rebuild.

---

## 4. What was verified

With both gaps patched at runtime (monkeypatch only — no jvagent or jvspatial source changed), against `jvspatial==0.0.15` + `asyncpg==0.31.0` + `postgres:16-alpine`:

- `jvagent examples/jvagent_app bootstrap` completed; `App`, `Agents`, both example agents, and every installed action persisted to Postgres — 44 `node` rows, 43 `edge` rows, 3 `object` rows after bootstrap plus two conversation turns, with the driver creating all three tables and their indexes unattended.
- Server started clean — `/health` reported `"database":"connected"`, lifecycle logged `📊 Database: PostgresDB | 🌳 Root: n.Root.root`.
- Admin bootstrap wrote a `User` to Postgres; `POST /api/auth/login` returned a JWT against it.
- `POST /api/agents/{id}/interact` ran full turns on the orchestrator example agent (`OrchestratorInteractAction` → `ReplyAction`), creating `User` / `Conversation` / `Interaction` nodes.
- Conversation state survived a **full server restart** — a later turn recalled a value stated before the restart, read back out of Postgres rather than process memory.
It bootstraps the graph, serves it, authenticates, runs an agent turn, **restarts the server**, and reads the conversation back. The restart is the point: it is the only step that exercises pool loop-affinity, and it is what failed before 0.0.16. Agent-turn checks are skipped when no model key is configured; the persistence checks still run.

So the blockers are strictly at the configuration boundary. Once jvspatial accepts `postgres` in the `Server` path and makes its pool loop-aware, jvagent runs on Postgres as-is.
Against the released `jvspatial==0.0.16` with `asyncpg==0.31.0` and `postgres:16-alpine`, all 15 checks pass on `examples/jvagent_app`: the full graph persists (39 nodes at bootstrap, 44 nodes / 43 edges / 3 objects after two turns), `/health` reports `"database":"connected"`, lifecycle logs `📊 Database: PostgresDB`, JWT login resolves a Postgres-stored user, and a post-restart turn recalls a value stated before the restart — read back out of Postgres, not process memory.

---

Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ classifiers = [
dependencies = [
"aiohttp>=3.9.0",
# CI records the resolved jvspatial version after install (see .github/workflows/test-jvagent.yaml).
"jvspatial==0.0.15",
"jvspatial==0.0.16",
"python-dotenv>=1.0.0",
"pyyaml>=6.0.0",
"httpx>=0.27.0",
Expand Down
Loading