Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,23 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Added

- **Postgres is selectable from `Server`** (`jvspatial/api/components/database_configurator.py`).
`initialize_graph_context()` now builds a `PostgresDB` prime database for
`db_type="postgres"` (alias `"postgresql"`) instead of raising
`ValueError: Unsupported database type: postgres`. The backend and
`create_database("postgres", ...)` already worked; only the `Server` path
was missing, so every API-layer deployment β€” and anything built on it β€” was
locked out of a documented backend.

- **`ServerConfig.database` carries Postgres settings** (`jvspatial/api/config_groups.py`,
`jvspatial/env_adapter.py`) β€” `postgres_dsn`, `postgres_min_pool_size`,
`postgres_max_pool_size`, `postgres_pooler_mode`, populated from the
already-allowlisted `JVSPATIAL_POSTGRES_*` env keys. Connection settings now
flow through the config object like every other backend's rather than being
readable only by the driver. Unset values still defer to `PostgresDB`'s own
defaults. Coverage: `tests/test_env_adapter_postgres.py`,
`tests/api/components/test_database_configurator.py`.

- **`resolve_sort_value(record, field)`** (`jvspatial/db/database.py`) β€” the
dotted-path resolution `finalize_find_results` uses, exported so adapters and
cursor logic resolve a sort field the same way. Added to the module's
Expand All @@ -24,6 +41,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Fixed

- **`PostgresDB` pool is event-loop aware** (`jvspatial/db/postgres.py`). The
`asyncpg` pool and its lock bind to the loop that created them, but
`_ensure_pool()` memoized both for the lifetime of the instance. A host that
bootstrapped in one `asyncio.run()` and then served from a second loop hit
`cannot perform operation: another operation is in progress` /
`ConnectionDoesNotExistError` on its first query. The pool and lock are now
rebuilt when the running loop changes. Coverage:
`tests/db/test_postgres_unit.py::TestPoolLoopAffinity`.

- **Partial-index repair log is INFO, not WARNING** (`jvspatial/db/sqlite.py`).
Dropping a non-partial index so it can be recreated with `WHERE` is expected
one-shot migration noise; log at info. Also satisfy ruff SIM110 in
Expand Down
40 changes: 40 additions & 0 deletions docs/md/postgres-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,37 @@ alice = await User.create(name="Alice", email="alice@example.com")
loaded = await User.get(alice.id)
```

### Using Postgres with `Server`

`Server` selects the prime database from `db_type`, so Postgres is available
to the API layer the same way JSON or MongoDB is:

```python
from jvspatial.api.server import Server

server = Server(
db_type="postgres",
postgres_dsn="postgresql://user:pw@localhost:5432/mydb",
)
```

`postgresql` is accepted as an alias for `postgres`. Every connection setting
also resolves from the environment through `ServerConfig`:

```bash
JVSPATIAL_DB_TYPE=postgres
JVSPATIAL_POSTGRES_DSN=postgresql://user:pw@localhost:5432/mydb
```

Settings you leave unset fall through to `PostgresDB`'s own defaults, so the
DSN is the only value most deployments need. `JVSPATIAL_DB_PATH` does not
apply β€” it is a file-backend setting.

Note that the logging database is a separate store and has no Postgres
backend; if `JVSPATIAL_LOG_DB_TYPE` is unset it inherits `JVSPATIAL_DB_TYPE`
and falls back to a JSON file log. Set it explicitly when the graph runs on
Postgres.

## Schema

Each collection becomes one table. The shape:
Expand Down Expand Up @@ -152,6 +183,15 @@ JVSPATIAL_POSTGRES_MIN_POOL_SIZE=5
JVSPATIAL_POSTGRES_MAX_POOL_SIZE=25
```

### Event loops

The pool belongs to the event loop that created it. If a host bootstraps
inside one `asyncio.run()` and then serves from a second loop β€” the common
CLI-then-ASGI-server startup β€” `PostgresDB` notices the loop change and
rebuilds the pool on first use in the new loop. The abandoned pool's
connections are dropped rather than closed, since closing them would mean
awaiting on a loop that no longer runs.

### Pooler compatibility (PgBouncer / RDS Proxy)

Transaction-mode poolers (PgBouncer transaction-pooling, AWS RDS Proxy) reuse
Expand Down
25 changes: 25 additions & 0 deletions jvspatial/api/components/database_configurator.py
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,25 @@ def _resolve_mongodb_connection(self) -> Tuple[str, str]:
db_name = (db.db_database_name or "").strip() or "jvdb"
return uri, db_name

def _resolve_postgres_kwargs(self) -> Dict[str, Any]:
"""Resolve PostgresDB connection kwargs from server configuration.

Only settings the operator actually supplied are returned, so
anything left unset falls through to ``PostgresDB``'s own env
defaults rather than being pinned to a value here.
"""
db = self.config.database
kwargs: Dict[str, Any] = {}
if db.postgres_dsn:
kwargs["dsn"] = db.postgres_dsn
if db.postgres_min_pool_size is not None:
kwargs["min_size"] = db.postgres_min_pool_size
if db.postgres_max_pool_size is not None:
kwargs["max_size"] = db.postgres_max_pool_size
if db.postgres_pooler_mode:
kwargs["pooler_mode"] = db.postgres_pooler_mode
return kwargs

def _resolve_observability_kwargs(self) -> Dict[str, Any]:
"""Resolve optional DB observability wrapper kwargs from env."""
raw_enabled = str(os.environ.get("JVSPATIAL_OBSERVABILITY_ENABLED", "")).lower()
Expand Down Expand Up @@ -173,6 +192,12 @@ def initialize_graph_context(self) -> Optional[GraphContext]:
aws_secret_access_key=self.config.database.dynamodb_secret_access_key,
**observability_kwargs,
)
elif db_type in ("postgres", "postgresql"):
prime_db = create_database(
db_type="postgres",
**self._resolve_postgres_kwargs(),
**observability_kwargs,
)
else:
raise ValueError(f"Unsupported database type: {db_type}")

Expand Down
17 changes: 17 additions & 0 deletions jvspatial/api/config_groups.py
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,23 @@ class DatabaseConfig(BaseModel):
default=None, validation_alias="AWS_SECRET_ACCESS_KEY"
)

# PostgreSQL Configuration (only used if db_type is "postgres"/"postgresql").
# Unset values fall through to PostgresDB's own env defaults.
postgres_dsn: Optional[str] = Field(
default=None, validation_alias="JVSPATIAL_POSTGRES_DSN"
)
postgres_min_pool_size: Optional[int] = Field(
default=None, validation_alias="JVSPATIAL_POSTGRES_MIN_POOL_SIZE"
)
postgres_max_pool_size: Optional[int] = Field(
default=None, validation_alias="JVSPATIAL_POSTGRES_MAX_POOL_SIZE"
)
postgres_pooler_mode: Optional[str] = Field(
default=None,
validation_alias="JVSPATIAL_POSTGRES_POOLER_MODE",
description="'session' (default) or 'transaction' for PgBouncer / RDS Proxy.",
)


class SecurityConfig(BaseModel):
"""Security configuration group."""
Expand Down
37 changes: 37 additions & 0 deletions jvspatial/db/postgres.py
Original file line number Diff line number Diff line change
Expand Up @@ -337,6 +337,10 @@ def __init__(

self._pool: Optional["Pool"] = None
self._pool_lock = asyncio.Lock()
# The loop ``_pool`` (and ``_pool_lock``) belong to. asyncpg pools
# and asyncio primitives bind to the loop they were created on, so
# a pool carried into a second loop is unusable.
self._pool_loop: Optional[asyncio.AbstractEventLoop] = None

# Collections we've already created the table + base indexes for.
# Avoids running CREATE TABLE IF NOT EXISTS on the hot path.
Expand All @@ -350,8 +354,39 @@ def __init__(

# ---- pool lifecycle ----------------------------------------------------

def _discard_pool_from_dead_loop(self) -> None:
"""Drop a pool that belongs to a different event loop.

Hosts that bootstrap inside one ``asyncio.run()`` and then serve
from a second loop (the CLI-then-ASGI-server pattern) would
otherwise reuse a pool bound to the first, now-closed loop, and
every query fails with ``cannot perform operation: another
operation is in progress``. The stale pool's connections are
abandoned rather than closed β€” ``close()`` would have to await on
the dead loop β€” and Postgres reaps them when the sockets drop.
"""
if self._pool is None:
return
try:
running = asyncio.get_running_loop()
except RuntimeError: # pragma: no cover - callers are async
return
if self._pool_loop is running:
return

logger.debug(
"PostgresDB: event loop changed; rebuilding pool (was %r, now %r)",
self._pool_loop,
running,
)
self._pool = None
# ``asyncio.Lock`` binds to its loop too, so it has to go as well.
self._pool_lock = asyncio.Lock()
self._collections_bootstrapped.clear()

async def _ensure_pool(self) -> "Pool":
"""Lazily create the asyncpg pool. Idempotent + concurrency-safe."""
self._discard_pool_from_dead_loop()
if self._pool is not None:
return self._pool
async with self._pool_lock:
Expand All @@ -377,13 +412,15 @@ async def _ensure_pool(self) -> "Pool":
self.pooler_mode,
)
self._pool = await asyncpg.create_pool(**create_kwargs)
self._pool_loop = asyncio.get_running_loop()
return self._pool

async def close(self) -> None:
"""Close the connection pool. Safe to call multiple times."""
if self._pool is not None:
await self._pool.close()
self._pool = None
self._pool_loop = None
self._collections_bootstrapped.clear()

# ---- tenant scope (C6) -------------------------------------------------
Expand Down
8 changes: 8 additions & 0 deletions jvspatial/env_adapter.py
Original file line number Diff line number Diff line change
Expand Up @@ -124,9 +124,17 @@ def server_config_overrides_from_env() -> Dict[str, Any]:
("JVSPATIAL_DYNAMODB_TABLE_NAME", "dynamodb_table_name"),
("JVSPATIAL_DYNAMODB_REGION", "dynamodb_region"),
("JVSPATIAL_DYNAMODB_ENDPOINT_URL", "dynamodb_endpoint_url"),
("JVSPATIAL_POSTGRES_DSN", "postgres_dsn"),
("JVSPATIAL_POSTGRES_POOLER_MODE", "postgres_pooler_mode"),
):
if (t := _opt_str(ek)) is not None:
db[dk] = t
for ek, dk in (
("JVSPATIAL_POSTGRES_MIN_POOL_SIZE", "postgres_min_pool_size"),
("JVSPATIAL_POSTGRES_MAX_POOL_SIZE", "postgres_max_pool_size"),
):
if (n := _opt_int(ek)) is not None:
db[dk] = n
if db:
o["database"] = db

Expand Down
84 changes: 84 additions & 0 deletions tests/api/components/test_database_configurator.py
Original file line number Diff line number Diff line change
Expand Up @@ -249,6 +249,90 @@ async def test_initialize_graph_context_dynamodb(self):
slow_query_ms=100.0,
)

@pytest.mark.asyncio
async def test_initialize_graph_context_postgres(self):
"""Test GraphContext initialization with PostgreSQL."""
config = ServerConfig()
config.database.db_type = "postgres"
config.database.postgres_dsn = "postgresql://u:p@localhost:5432/testdb"
config.database.postgres_min_pool_size = 3
config.database.postgres_max_pool_size = 7
config.database.postgres_pooler_mode = "transaction"
configurator = DatabaseConfigurator(config)

with patch(
"jvspatial.api.components.database_configurator.create_database"
) as mock_create:
mock_db = MagicMock()
mock_create.return_value = mock_db

with patch(
"jvspatial.api.components.database_configurator.get_database_manager"
) as mock_get_manager:
mock_manager = MagicMock()
mock_manager.get_current_database.return_value = mock_db
mock_get_manager.side_effect = RuntimeError()

with patch(
"jvspatial.api.components.database_configurator.set_database_manager"
):
with patch(
"jvspatial.api.components.database_configurator.GraphContext"
) as mock_context:
mock_ctx_instance = MagicMock()
mock_context.return_value = mock_ctx_instance

with patch(
"jvspatial.api.components.database_configurator.set_default_context"
):
result = configurator.initialize_graph_context()

assert result == mock_ctx_instance
mock_create.assert_called_once_with(
db_type="postgres",
dsn="postgresql://u:p@localhost:5432/testdb",
min_size=3,
max_size=7,
pooler_mode="transaction",
observe=False,
slow_query_ms=100.0,
)

@pytest.mark.asyncio
async def test_initialize_graph_context_postgresql_alias(self):
"""``postgresql`` is accepted as an alias for ``postgres``."""
config = ServerConfig()
config.database.db_type = "postgresql"
configurator = DatabaseConfigurator(config)

with patch(
"jvspatial.api.components.database_configurator.create_database"
) as mock_create:
mock_create.return_value = MagicMock()
with patch(
"jvspatial.api.components.database_configurator.get_database_manager",
side_effect=RuntimeError(),
):
with patch(
"jvspatial.api.components.database_configurator.set_database_manager"
):
with patch(
"jvspatial.api.components.database_configurator.GraphContext",
return_value=MagicMock(),
):
with patch(
"jvspatial.api.components.database_configurator.set_default_context"
):
configurator.initialize_graph_context()

# Unset connection settings are omitted so PostgresDB's own env
# defaults still apply.
mock_create.assert_called_once_with(
db_type="postgres",
observe=False,
slow_query_ms=100.0,
)

@pytest.mark.asyncio
async def test_initialize_graph_context_no_db_type(self, configurator):
"""Test that None is returned when db_type is not set."""
Expand Down
Loading