Dev

.github/dependabot.yml

@@ -0,0 +1,50 @@
+# Dependabot configuration.
+#
+# Two ecosystems are watched:
+# * pip -- the project's runtime + dev dependencies as declared in
+#   pyproject.toml. Weekly cadence keeps the noise low while still
+#   catching CVEs on a reasonable schedule.
+# * github-actions -- the CI workflow actions (actions/checkout,
+#   actions/setup-python, etc.). These have a long history of
+#   breaking changes and tag drift; weekly is enough.
+#
+# We mark security-impacting bumps separately so a maintainer can
+# fast-track them without wading through routine version bumps.
+
+version: 2
+updates:
+  - package-ecosystem: "pip"
+    directory: "/"
+    schedule:
+      interval: "weekly"
+      day: "monday"
+      time: "09:00"
+    open-pull-requests-limit: 5
+    labels:
+      - "dependencies"
+      - "pip"
+    commit-message:
+      prefix: "deps"
+      include: "scope"
+    # Group routine version bumps so we don't drown in tiny PRs.
+    groups:
+      pip-minor-and-patch:
+        patterns:
+          - "*"
+        update-types:
+          - "minor"
+          - "patch"
+
+  - package-ecosystem: "github-actions"
+    directory: "/"
+    schedule:
+      interval: "weekly"
+      day: "monday"
+      time: "09:00"
+    open-pull-requests-limit: 5
+    labels:
+      - "dependencies"
+      - "ci"
+    commit-message:
+      prefix: "ci"
+      include: "scope"

.github/workflows/benchmarks.yml

@@ -0,0 +1,186 @@
+name: Benchmarks
+
+# Runs the pytest-benchmark suite on every pull request and compares
+# against a baseline captured from main. A regression > REGRESSION_PCT
+# is reported as a comment on the PR but does NOT (yet) fail the build
+# -- shared GitHub-runner CI has too much variance to make hard gating
+# trustworthy. Treat the comment as a signal to investigate.
+#
+# To capture a new baseline (e.g. after intentional perf work):
+# - merge the PR to main
+# - the workflow's `update-baseline` job below runs on push to main
+#   and uploads the fresh result as the `bench-baseline` artifact
+
+on:
+  pull_request:
+    paths:
+      - 'jvspatial/**'
+      - 'tests/benchmarks/**'
+      - 'pyproject.toml'
+      - '.github/workflows/benchmarks.yml'
+  push:
+    branches:
+      - main
+    paths:
+      - 'jvspatial/**'
+      - 'tests/benchmarks/**'
+      - 'pyproject.toml'
+      - '.github/workflows/benchmarks.yml'
+
+permissions:
+  contents: read
+  pull-requests: write
+
+env:
+  # When a PR is more than this percent slower than the baseline on
+  # any single benchmark, post a regression comment. Threshold is
+  # deliberately loose to absorb runner variance; tighten as the
+  # benchmark suite stabilizes.
+  REGRESSION_PCT: '25'
+
+jobs:
+  run-benchmarks:
+    name: Run benchmark suite
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v5
+
+      - name: Set up Python
+        uses: actions/setup-python@v6
+        with:
+          python-version: '3.11'
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -e '.[dev,test]'
+
+      - name: Run benchmarks (PR or main push)
+        run: |
+          mkdir -p .benchmarks/current
+          pytest tests/benchmarks \
+            --benchmark-only \
+            --benchmark-json=.benchmarks/current/result.json \
+            --benchmark-min-rounds=5 \
+            --benchmark-warmup=on
+
+      - name: Upload current results
+        uses: actions/upload-artifact@v6
+        with:
+          name: bench-current
+          path: .benchmarks/current/result.json
+          retention-days: 30
+
+      # On a push to main, this run becomes the new baseline.
+      - name: Publish as baseline (main only)
+        if: github.event_name == 'push' && github.ref == 'refs/heads/main'
+        uses: actions/upload-artifact@v6
+        with:
+          name: bench-baseline
+          path: .benchmarks/current/result.json
+          retention-days: 90
+
+  compare-to-baseline:
+    name: Compare against baseline
+    if: github.event_name == 'pull_request'
+    runs-on: ubuntu-latest
+    needs: run-benchmarks
+    steps:
+      - uses: actions/checkout@v5
+
+      - name: Download current results
+        uses: actions/download-artifact@v5
+        with:
+          name: bench-current
+          path: .benchmarks/current
+
+      - name: Download baseline (best-effort)
+        id: baseline
+        continue-on-error: true
+        uses: dawidd6/action-download-artifact@v6
+        with:
+          workflow: benchmarks.yml
+          branch: main
+          name: bench-baseline
+          path: .benchmarks/baseline
+          if_no_artifact_found: warn
+
+      - name: Check baseline presence
+        # The download action returns outcome=success even when no
+        # artifact was found (because if_no_artifact_found=warn). The
+        # only reliable check is whether the file actually landed.
+        id: have_baseline
+        run: |
+          if [ -f .benchmarks/baseline/result.json ]; then
+            echo "present=true" >> "$GITHUB_OUTPUT"
+          else
+            echo "present=false" >> "$GITHUB_OUTPUT"
+            echo "::notice::No baseline artifact yet -- benchmark comparison skipped. After this PR merges and the benchmarks workflow runs on main, the produced 'bench-baseline' artifact will be used for future PR comparisons."
+          fi
+
+      - name: Compare
+        if: steps.have_baseline.outputs.present == 'true'
+        id: compare
+        run: |
+          python <<'PY'
+          import json, os, sys
+          from pathlib import Path
+
+          cur = json.loads(Path('.benchmarks/current/result.json').read_text())
+          base = json.loads(Path('.benchmarks/baseline/result.json').read_text())
+
+          threshold = float(os.environ.get('REGRESSION_PCT', '25'))
+
+          # Build {name: mean_seconds}
+          def index(report):
+              out = {}
+              for b in report.get('benchmarks', []):
+                  out[b['fullname']] = b['stats']['mean']
+              return out
+
+          cur_idx = index(cur)
+          base_idx = index(base)
+
+          rows = []
+          regressions = []
+          for name, cur_mean in sorted(cur_idx.items()):
+              base_mean = base_idx.get(name)
+              if base_mean is None:
+                  rows.append((name, None, cur_mean, None, 'NEW'))
+                  continue
+              pct = (cur_mean - base_mean) / base_mean * 100.0
+              status = 'OK'
+              if pct > threshold:
+                  status = f'REGRESSION (+{pct:.1f}%)'
+                  regressions.append((name, base_mean, cur_mean, pct))
+              elif pct < -threshold:
+                  status = f'IMPROVED ({pct:.1f}%)'
+              rows.append((name, base_mean, cur_mean, pct, status))
+
+          # Render markdown summary
+          md = ['## Benchmark comparison\n']
+          md.append(f'Threshold: ±{threshold:.0f}% (informational, does not block merge)\n')
+          md.append('| benchmark | baseline (s) | current (s) | delta | status |')
+          md.append('|---|---:|---:|---:|---|')
+          for row in rows:
+              if row[1] is None:
+                  md.append(f'| `{row[0]}` | — | {row[2]:.6f} | — | {row[4]} |')
+              else:
+                  delta = f'{row[3]:+.1f}%'
+                  md.append(f'| `{row[0]}` | {row[1]:.6f} | {row[2]:.6f} | {delta} | {row[4]} |')
+
+          summary = '\n'.join(md) + '\n'
+          Path('comment.md').write_text(summary)
+          print(summary)
+
+          if regressions:
+              print(f'::warning::Detected {len(regressions)} benchmark(s) regressing > {threshold}%')
+          PY
+
+      - name: Comment on PR
+        if: steps.have_baseline.outputs.present == 'true'
+        uses: peter-evans/create-or-update-comment@v4
+        with:
+          issue-number: ${{ github.event.pull_request.number }}
+          body-path: comment.md
+          edit-mode: replace

.github/workflows/security.yml

@@ -0,0 +1,71 @@
+name: Security audit
+
+# Runs pip-audit weekly and on any change to dependency declarations.
+# Currently non-blocking (continue-on-error: true) so an upstream CVE
+# disclosure doesn't block unrelated PRs while a maintainer triages.
+# Flip to ``false`` once the dependency hygiene baseline is clean.
+
+on:
+  pull_request:
+    paths:
+      - 'pyproject.toml'
+      - 'requirements*.txt'
+      - '.github/workflows/security.yml'
+  push:
+    branches:
+      - main
+    paths:
+      - 'pyproject.toml'
+      - 'requirements*.txt'
+  schedule:
+    # Mondays at 06:00 UTC. Catches CVEs disclosed over the weekend
+    # before adopters hit them.
+    - cron: '0 6 * * 1'
+
+permissions:
+  contents: read
+
+jobs:
+  pip-audit:
+    name: pip-audit
+    runs-on: ubuntu-latest
+    continue-on-error: true
+    steps:
+      - uses: actions/checkout@v5
+
+      - name: Set up Python
+        uses: actions/setup-python@v6
+        with:
+          python-version: '3.11'
+
+      - name: Install project + audit tooling
+        run: |
+          python -m pip install --upgrade pip
+          pip install pip-audit
+          # Resolve the project's full dependency set so the audit
+          # sees the same packages adopters would install.
+          pip install '.[dev,test,otel]'
+
+      - name: Run pip-audit
+        run: |
+          # We audit an explicit requirements list rather than the
+          # environment because jvspatial itself (0.0.x) isn't on PyPI
+          # yet -- env-mode pip-audit fails on it both as a regular
+          # install ("Dependency not found on PyPI") and as an editable
+          # install ("distribution marked as editable"; --skip-editable
+          # is unreliable across pip-audit versions). Generating a
+          # filtered freeze-list and feeding it via -r side-steps the
+          # whole detection path.
+          #
+          # --strict: fail on any vulnerability in the resolved deps.
+          # The whole job is wrapped in continue-on-error today so a
+          # CVE in a transitive dep doesn't block unrelated PRs while
+          # a maintainer triages.
+          pip list --format=freeze \
+            | grep -iv '^jvspatial' \
+            | grep -iv '^-e' \
+            > /tmp/audit-reqs.txt
+          echo "::group::Auditing the following requirements"
+          cat /tmp/audit-reqs.txt
+          echo "::endgroup::"
+          pip-audit --strict -r /tmp/audit-reqs.txt --progress-spinner off

.github/workflows/test-jvspatial.yaml

@@ -34,7 +34,11 @@ jobs:
     # coverage threshold once on 3.12.
     - name: Run tests (with coverage)
       if: matrix.python-version == '3.12'
-      run: pytest tests/ -v --tb=short --cov=jvspatial --cov-report=xml --cov-report=term-missing --cov-fail-under=55
+      # Coverage floor bumped to 60% after Phase A1-A5/B1-B3/C1-C2 added
+      # ~80 new tests covering durability, transactions, push-down,
+      # caching, observability, retry, bulk APIs, multipart S3.
+      # Tighten further once flake risk is understood under the new suite.
+      run: pytest tests/ -v --tb=short --cov=jvspatial --cov-report=xml --cov-report=term-missing --cov-fail-under=60
 
     - name: Run tests (no coverage)
       if: matrix.python-version != '3.12'

CHANGELOG.md

@@ -7,22 +7,50 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.0.7] - 2026-05-08
+
 ### Security
 
 - **BREAKING**: JWT secret must be set explicitly when authentication is enabled. The server now fails fast with a clear error if `JVSPATIAL_JWT_SECRET_KEY` is not set or uses a placeholder value. Set via environment or `Server(auth=dict(jwt_secret="..."))`.
 - Add security headers middleware (X-Content-Type-Options, X-Frame-Options, X-XSS-Protection) applied to all responses by default. Configurable via `Server(security=dict(security_headers_enabled=True))`.
 - AuthConfig `jwt_secret` default changed from `"your-secret-key"` to empty string; explicit setting required when auth is enabled.
 - Remove duplicate `/auth/register` from auth exempt paths.
+- Add weekly `pip-audit` workflow (`.github/workflows/security.yml`).
 
 ### Added
 
+- **IO durability:** `JsonDB` writes now go through an atomic `temp + fsync + rename + fsync(dir)` helper (`jvspatial/db/_atomic.py`); a process crash, kernel panic, or power loss can never leave a partial record on disk. Orphan `*.jvtmp` files left by prior crashed processes are reaped on startup (skipped under serverless mode).
+- **Per-path locking:** `JsonDB` uses a bounded-LRU `PathLockManager` (`jvspatial/db/_path_locks.py`) so concurrent writes to different files run in parallel while same-file writes serialize. Cross-thread safe.
+- **Atomic version metadata** in `LocalFileInterface.create_version()` and `save_file()` via the same helper.
+- **Native `Database.count()`** with filter pushdown across all backends. SQLite gains a Mongo→SQL query translator (`jvspatial/db/_sqlite_translate.py`) that pushes `$eq`/`$ne`/`$gt`/`$gte`/`$lt`/`$lte`/`$in`/`$nin`/`$exists`, top-level AND, and `$and`/`$or` (recursive) into `WHERE … json_extract()` clauses with `LIMIT`/`ORDER BY` pushdown. MongoDB gains native `count_documents`/`estimated_document_count`. DynamoDB gains `Select="COUNT"`. JsonDB gains a dirent-only fast path for empty queries.
+- **Bulk APIs:** `Database.find_many(ids)` and `Database.bulk_save(records)` with native overrides on every backend (Mongo `$in` + `bulk_write`, SQLite single-transaction `IN`/`executemany`, DynamoDB `BatchGetItem`/`BatchWriteItem`, JsonDB parallel reads/writes).
+- **Capability flags:** `Database.supports_transactions` (False default; True on MongoDB) so callers can branch without sniffing adapter classes.
+- **Read-through cache wrapper:** opt-in via `create_database(cache_get_size=N, cache_get_ttl=S)`. LRU + TTL, invalidates on save/delete, refreshes on `bulk_save`/`find_one_and_update`, skipped under serverless.
+- **Observability layer:** opt-in via `create_database(observe=True, slow_query_ms=N, metrics=...)`. Emits a structured log line per DB op (`backend`/`op`/`collection`/`duration_ms`/`success`/`result_count`) with WARNING-elevation at the slow-query threshold, plus four metrics (`jvspatial.db.op.duration_seconds`, `.count`, `.slow_count`, `.result_count`).
+- **`MetricsRecorder` Protocol** with `NullMetricsRecorder` default. Optional **OpenTelemetry adapter** under `pip install jvspatial[otel]` (`jvspatial/observability/otel.py`).
+- **Shared retry helper** (`jvspatial/utils/retry.py`): async `retry_async()` and `@retry()` decorator with exponential backoff + full jitter, configurable retryable predicate, optional `on_retry` hook. Used by MongoDB connection-error recovery, DynamoDB throttle errors, S3 SlowDown/5xx errors.
+- **S3 multipart uploads** at ≥ 8 MiB (configurable via constructor or `JVSPATIAL_S3_MULTIPART_THRESHOLD` env). Uses boto3's `TransferManager` for splitting, parallel parts, and resume-on-failure.
+- **DynamoDB throttle retry:** `save`/`get`/`delete` auto-retry on `ProvisionedThroughputExceededException`, `ThrottlingException`, `RequestLimitExceeded`, `TooManyRequestsException`, `TransactionConflictException`.
+- **DeferredSave auto-flush:** new `max_pending_saves` class attr (default `None`/disabled) bounds in-memory dirty state for callers who forget to flush.
+- **`@experimental` and `@deprecated` decorators** (`jvspatial/utils/{stability,deprecation}.py`) with once-per-process warnings, async support, serverless suppression. `JsonDBTransaction(best_effort=True)` is now wired to emit the experimental warning.
+- **PEP 561 typing:** `jvspatial/py.typed` marker shipped via `pyproject.toml` package data so mypy/pyright treat jvspatial as typed.
+- **Benchmark suite:** `tests/benchmarks/` with 13 benches across JsonDB / SQLite / DeferredSave guarding the new IO wins. New `.github/workflows/benchmarks.yml` posts a regression-comparison comment on every PR (vs. the latest `bench-baseline` artifact published from `main`).
+- **Community-readiness scaffolding:** `CONTRIBUTING.md` (root), `CODE_OF_CONDUCT.md` (Contributor Covenant 2.1 by reference), `RELEASING.md` aligned with the `version.py`-driven publish workflow.
+- **Stability contract:** `docs/md/stability.md` declaring public/internal/experimental tiers and the deprecation policy.
+- **Observability + benchmarks docs:** `docs/md/observability.md`, `docs/md/benchmarks.md`.
+- **Dependabot:** `.github/dependabot.yml` for `pip` and `github-actions` ecosystems, weekly with grouped minor/patch bumps.
+- `Database.drop_deprecated_indexes()` optional hook (default no-op) for named-index cleanup; MongoDB implementation drops listed names. Documented in [Custom Database guide](docs/md/custom-database-guide.md) and [optimization](docs/md/optimization.md#declarative-database-indexing) (index creation timing, partial indexes, MongoDB conflict handling).
 - `SecurityConfig` with `security_headers_enabled` option.
 - [Production Deployment Guide](docs/md/production-deployment.md) with security checklist.
-- CI now runs test coverage with `--cov-fail-under=50`.
 
 ### Changed
 
+- **BREAKING (limited):** `JsonDBTransaction(db).save/get/delete/find()` now raises `NotImplementedError` by default instead of silently no-op'ing. Pass `best_effort=True` to opt into the buffered-commit semantics, or check `Database.supports_transactions` and fall back to non-transactional writes. Audited downstream consumers (`jvagent`, `integral`) — neither uses this surface, so no coordinated change required.
+- **`QueryEngine` optimization cache** is now bounded by an LRU (default 1024 entries, configurable via `QueryEngine(cache_size=...)`). Was unbounded.
+- **MongoDB retries** now go through the shared retry helper (`utils/retry.py`); behavior preserved (one retry on connection-error with reset).
+- CI coverage gate raised from 55 → 60% to reflect the new tested code.
 - Security headers are applied automatically (enabled by default).
+- `pyproject.toml` adds `[otel]` extra and a `benchmark` pytest marker; default `pytest` invocation now skips `tests/benchmarks/` (run with `pytest tests/benchmarks --benchmark-only`).
 
 ## [0.0.5] - Previous