Dev

[eldonm]

Type of Change

What type of change does this PR introduce? Mark all that apply:

• 🐛 Bug Fix (concurrency bug in JsonDB orphan-tmp sweep; honest transaction semantics replacing silent no-op)
• 🚀 Feature Request (atomic writes, native count + pushdown, bulk APIs, caching, observability, retry, multipart S3, benchmarks)
• 🔄 Refactor (MongoDB retry consolidation; SQLite query translator; QueryEngine LRU bound)
• 📖 Documentation Update (observability, benchmarks, stability, contributing, releasing)
• 🔧 Other: community-readiness scaffolding (Dependabot, pip-audit CI, py.typed, CoC, security policy)

---

Summary

What does this PR address?

Multi-phase community-readiness pass to harden jvspatial's IO layer, expose first-class observability and resilience primitives, ship contributor-facing scaffolding, and fix a concurrency bug discovered in flight. After this PR, jvspatial is durable under crashes, faster on every backend, opt-in observable, and structurally ready for outside contributors.

• Adds crash-safe atomic writes everywhere (JsonDB, LocalFileInterface)
• Adds native count() + filter/sort/limit pushdown across MongoDB, SQLite, DynamoDB, JsonDB
• Adds Database.find_many() and Database.bulk_save() with native per-backend overrides
• Adds opt-in CachingDatabase (LRU + TTL, negative caching) and ObservableDatabase (structured log + metrics) wrappers, composable via create_database() kwargs
• Adds MetricsRecorder Protocol + zero-overhead NullMetricsRecorder default + optional OpenTelemetry adapter under jvspatial[otel]
• Adds shared async retry helper with exponential-backoff-plus-full-jitter; refactors MongoDB retries to use it; adds DynamoDB throttle retries; adds S3 SlowDown/5xx retries
• Adds S3 multipart upload at ≥ 8 MiB
• Makes JsonDBTransaction strict by default (raises NotImplementedError); adds opt-in best_effort=True buffered mode marked @experimental
• Adds Database.supports_transactions capability flag
• Adds DeferredSaveMixin.max_pending_saves auto-flush safety net
• Bounds QueryEngine optimization cache with LRU
• Adds @experimental and @deprecated decorators with once-per-process semantics
• Adds pytest-benchmark regression suite (13 benches) + CI workflow with PR-comment comparison
• Adds CONTRIBUTING.md, RELEASING.md, CODE_OF_CONDUCT.md, docs/md/{stability,observability,benchmarks}.md, py.typed
• Adds Dependabot config + weekly pip-audit workflow; raises CI coverage gate to 60%
• Fixes a JsonDB concurrency bug where the lazy orphan-tmp sweep raced concurrent writers

---

Description

Bug Fixes

1. JsonDB orphan-tmp sweep race condition (uncovered during Phase A4 development)

• Bug: Concurrent bulk_save calls would intermittently fail with ENOENT on rename. The first writer to call _get_collection_dir() triggered an rglob("*.jvtmp") sweep that unlinked in-flight temp files of other concurrent writers.
• Root cause: The lazy sweep ran inside the per-call directory-resolution path with no synchronization.
• Fix: Added a threading.Lock (_init_lock) gating the sweep so it runs exactly once before any concurrent write can proceed. Fast-path check outside the lock keeps steady-state cost at one branch.

2. Silent-no-op JsonDBTransaction

• Bug: JsonDBTransaction(db).save/get/delete/find() silently delegated to the underlying database without any transactional guarantee, then commit() returned successfully. Callers thought writes were transactional; they weren't.
• Root cause: Original implementation had no enforcement; the docstring claimed "simulated."
• Fix: Strict-by-default — every IO method raises NotImplementedError with a clear message pointing to Database.supports_transactions or the best_effort=True opt-in. The opt-in mode buffers writes/deletes in memory and applies on commit (with documented semantics — atomic against single-process readers, not against process crashes mid-commit). Marked @experimental so adopters see a once-per-process warning.

Feature Request

Phase A1 — IO durability

• jvspatial/db/_atomic.py: atomic_write_bytes / atomic_write_text (temp + fsync + rename + dir-fsync). Adopters get crash safety on JsonDB and LocalFileInterface for free. Skips dir-fsync under serverless mode where the FS is tmpfs.
• jvspatial/db/_path_locks.py: PathLockManager — bounded LRU per-path locks. Concurrent writes to different files run in parallel; same-file writes serialize. Cross-thread safe (still works from asyncio.run-in-side-thread callers like DBLogHandler).

Phase A2 — Native count + filter pushdown

• Database.count() is now native on every backend.
  • SQLite: SELECT COUNT(*) plus a Mongo→SQL query translator (jvspatial/db/_sqlite_translate.py) supporting $eq/$ne/$gt/$gte/$lt/$lte/$in/$nin/$exists, top-level AND, and $and/$or (recursive) into json_extract() WHERE clauses with LIMIT/ORDER BY pushdown. Falls back to legacy in-Python filter for $regex/$elemMatch/etc. Field paths validated against a strict regex; values are bound parameters — no SQL-injection surface.
  • MongoDB: count_documents / estimated_document_count.
  • DynamoDB: Select="COUNT" via Scan or GSI Query.
  • JsonDB: dirent fast path for empty queries; streaming match for filtered queries (no result-list materialization).
• Entity-layer Object.count() and Object.find() benefit transparently.

Phase A3 — Caching

• CachingDatabase wrapper (jvspatial/db/_cache.py): opt-in LRU+TTL get-cache, negative caching, save/delete/find-one-and-* invalidation. Skipped under serverless mode. Wired via create_database(cache_get_size=, cache_get_ttl=).
• Bounded the QueryEngine optimization cache with an LRU (default 1024, configurable, set to 0 to disable).

Phase A4 — Bulk APIs

• Database.find_many(ids) and Database.bulk_save(records) added to the protocol. Native overrides:
  • MongoDB: find({"_id": {"$in": ids}}) / bulk_write with ordered=False.
  • SQLite: chunked WHERE id IN (...) SELECT (500-id chunks) / single-transaction executemany (all-or-nothing).
  • DynamoDB: surface existing BatchGetItem / BatchWriteItem.
  • JsonDB: parallel reads via asyncio.gather / parallel atomic per-file writes.
• CachingDatabase splits find_many between cache hits and a single backend call for misses.
• DeferredSaveMixin gains max_pending_saves class var (default None/disabled) — a safety net that triggers auto-flush if N deferred saves accumulate without an explicit flush.

Phase A5 — Resilience

• jvspatial/utils/retry.py: shared async retry_async() and @retry() decorator. Exponential backoff + full jitter (per AWS Architecture Blog's analysis), configurable retryable predicate (exception class, tuple, or callable), on_retry hook for log/metric integration.
• DynamoDB save/get/delete auto-retry on ProvisionedThroughputExceededException, ThrottlingException, RequestLimitExceeded, TooManyRequestsException, TransactionConflictException.
• S3 save_file/get_file/delete_file retry on SlowDown, RequestTimeout, ServiceUnavailable, 5xx.
• S3 multipart upload at ≥ 8 MiB (configurable via constructor or JVSPATIAL_S3_MULTIPART_THRESHOLD env). Uses boto3's TransferManager for splitting + parallel parts + resume-on-failure.

Phase B1/B2 — Observability

• MetricsRecorder Protocol (jvspatial/observability/metrics.py): record_duration / increment_counter / record_value with **labels. Runtime-checkable. NullMetricsRecorder default (zero overhead).
• Optional OpenTelemetryMetricsRecorder (jvspatial/observability/otel.py) under pip install jvspatial[otel]. Targets the OTel meter API; if no SDK is configured, calls become no-ops by design.
• ObservableDatabase wrapper (jvspatial/db/_observable.py) emits a structured log line per DB op with backend/op/collection/duration_ms/success/result_count. WARNING-elevation at the slow_query_ms threshold (default 100ms). Plus four metrics: jvspatial.db.op.{duration_seconds, count, slow_count, result_count}.
• Composes after the cache so log timings reflect user-visible latency (including cache hits/misses).

Phase B3 — Benchmarks

• tests/benchmarks/: 13 benches across JsonDB / SQLite / DeferredSaveMixin guarding the IO wins.
• .github/workflows/benchmarks.yml runs on PRs, compares against the latest bench-baseline artifact (published from main), posts a markdown comparison comment. Hard-gating off (CI variance too high); informational warning at >25% regression.

Refactor Request

• MongoDB retries: the previous per-method try/except blocks for ConnectionFailure + "Event loop is closed" RuntimeError were duplicated 8 ways. Refactored save/get/delete/find to share a _run_with_reconnect(coro_factory) helper that uses the new retry_async with an on_retry hook that resets _client/_db. Behavior preserved exactly (one retry on transient connection error with state reset).
• SQLite single-connection model documented. Confirmed via audit that the existing single-persistent-connection pattern is correct (WAL gives concurrent reads, asyncio.Lock serializes writes); added a docstring block explaining the trade-off and the single-event-loop assumption rather than building a pool that wasn't needed.
• QueryEngine.optimize_query cache changed from unbounded dict to OrderedDict-backed LRU. New cache_evictions counter in _optimization_stats.

---

Changes Made

High-Level Summary

1. IO durability layer (_atomic.py, _path_locks.py) wired into JsonDB, LocalFileInterface
2. Honest transactions: JsonDBTransaction strict by default, opt-in best-effort marked experimental, Database.supports_transactions capability flag
3. Native count() everywhere + Mongo→SQL query translator for SQLite (_sqlite_translate.py)
4. Bulk APIs (find_many, bulk_save) with native overrides on all four backends
5. Caching wrapper CachingDatabase (LRU+TTL, negative-cache, invalidation, serverless-skipped)
6. Observability wrapper ObservableDatabase + MetricsRecorder Protocol + Null default + optional OTel adapter
7. Resilience: shared async retry helper; refactored MongoDB retries; added DynamoDB throttle retries; S3 SlowDown/5xx retries; S3 multipart at ≥ 8 MiB
8. DeferredSaveMixin gains max_pending_saves auto-flush safety net
9. @experimental + @deprecated decorators with once-per-process semantics
10. Benchmark suite + PR-comparison workflow
11. Community scaffolding: CONTRIBUTING.md, RELEASING.md, CODE_OF_CONDUCT.md, docs/md/{stability,observability,benchmarks}.md, py.typed PEP-561 marker
12. CI hardening: Dependabot for pip + GitHub Actions, weekly pip-audit workflow, coverage gate raised 55 → 60%
13. Concurrency bug fix: JsonDB lazy orphan-tmp sweep gated by threading.Lock to stop racing concurrent writers
14. Bounded QueryEngine optimization cache (LRU, default 1024)
15. README + CHANGELOG updated to surface all of the above

File counts: 62 files changed, +7,340 / -306. Breakdown: 4 new internal modules (_atomic, _path_locks, _cache, _observable, _sqlite_translate), 3 new utility modules (stability, deprecation, retry), 3 new observability modules (__init__, metrics, otel), 12 new test files (~110 cases), 4 new doc pages, 4 new community files, 2 new CI workflows.

---

Checklist

Mark all that apply:

• Code follows the project's coding guidelines (pre-commit clean: black, isort, flake8 with bugbear/simplify/comprehensions/annotations/naming/docstrings, mypy, detect-secrets, check-yaml, check-json, trailing-whitespace).
• Tests have been added or updated for new functionality (~110 new test cases across atomic writes, path locks, transaction semantics, query translator, SQLite pushdown, JsonDB count, caching database, observable database, metrics recorder, OpenTelemetry adapter, bulk APIs, deferred-save auto-flush, retry helper, S3 multipart, stability marker, deprecation marker, local storage atomicity).
• Documentation has been updated (docs/md/{observability,benchmarks,stability}.md new; CONTRIBUTING.md, RELEASING.md, CODE_OF_CONDUCT.md new at repo root; README.md cross-links; CHANGELOG.md [Unreleased] block).
• Existing tests pass locally with these changes (full suite green; all 13 new benches runnable via pytest tests/benchmarks --benchmark-only).
• Any dependencies introduced are justified and documented (pytest-benchmark added to dev+test extras for the regression suite; opentelemetry-api added as a new optional [otel] extra — not pulled into core).

---

Steps to Test

1. Quality bar — same as CI:

   pre-commit run --all-files
   pytest tests/ -v --tb=short --cov=jvspatial --cov-fail-under=60

2. New tests in isolation:

   pytest tests/db/test_atomic.py \
          tests/db/test_path_locks.py \
          tests/db/test_transaction_semantics.py \
          tests/db/test_query_cache_lru.py \
          tests/db/test_caching_database.py \
          tests/db/test_observable_database.py \
          tests/db/test_bulk_apis.py \
          tests/db/test_jsondb_count.py \
          tests/db/test_sqlite_translate.py \
          tests/db/test_sqlite_pushdown.py \
          tests/storage/test_local_version_atomicity.py \
          tests/storage/test_s3_multipart.py \
          tests/utils/test_retry.py \
          tests/utils/test_deprecation.py \
          tests/utils/test_stability.py \
          tests/observability/test_metrics_recorder.py \
          tests/observability/test_otel_adapter.py \
          tests/core/test_deferred_save_auto_flush.py -v

3. Benchmarks:

   pytest tests/benchmarks --benchmark-only

4. Smoke a full opt-in stack (cache + observe + OTel):

   from jvspatial.db import create_database
   from jvspatial.observability.otel import OpenTelemetryMetricsRecorder

   db = create_database(
       "sqlite",
       db_path=":memory:",
       cache_get_size=2048,
       cache_get_ttl=30.0,
       observe=True,
       slow_query_ms=50.0,
       metrics=OpenTelemetryMetricsRecorder(),
   )
   await db.save("node", {"id": "x", "v": 1})
   assert (await db.get("node", "x"))["v"] == 1

Expect a structured log line per op via the jvspatial.db.observable logger.

5. Confirm downstream compatibility — jvagent and integral test suites should pass against the new commit without any source change. (Audit during this PR confirmed zero callers of the only meaningfully-breaking surface, JsonDBTransaction direct usage.)

6. Regression check — run the benchmark suite before and after, confirm no >25% slowdown on any of the 13 benches.

---

Additional Context

Stability tiers. docs/md/stability.md declares which APIs are public/stable, internal, or experimental — and the deprecation policy. New public surface: Database.find_many, Database.bulk_save, Database.supports_transactions, MetricsRecorder Protocol, NullMetricsRecorder, OpenTelemetryMetricsRecorder. New experimental surface: JsonDBTransaction(best_effort=True) (emits ExperimentalWarning on first use per process). Internal-only: _atomic, _path_locks, _sqlite_translate, _cache, _observable — used via create_database() factory kwargs.

Serverless safety. Every new layer that has a "long-lived process" assumption (cache, auto-flush, orphan-tmp sweep, dir-fsync, once-per-process warnings) checks is_serverless_mode() and degrades cleanly. Same pattern as the existing DeferredSaveMixin.

Performance impact (rough):

• User.count() on a 100K-row SQLite table: was O(100K) JSON parse + Python filter; now O(1) server-side.
• User.find({"active": True}) on the same: was full-table load + filter; now WHERE-pushdown.
• DynamoDB count() without GSI on a 100K-item table: was 100K item bodies pulled across the wire; now Select=COUNT returns just the count.
• JsonDB empty count(): was N reads + N JSON parses; now N dirent entries.
• S3 100MB upload: was single put_object; now ~25 parallel multipart uploaders with resume-on-failure.

Breaking change inventory (all surfaces):

• JsonDBTransaction(db).save/get/delete/find() raises NotImplementedError by default (was silent no-op). Migration: pass best_effort=True for the buffered semantics, OR check Database.supports_transactions and fall back to non-transactional writes.

That is the only meaningfully-breaking change. Audited downstream consumers (jvagent, integral) — neither uses this surface, so no coordinated change required.

---

Questions or Concerns

1. Should [otel] go into the default install? Currently it's a separate extra (pip install jvspatial[otel]). Keeps core dep tree small. If you'd rather have OTel ride along by default, easy switch — move it into dependencies.

2. CI coverage gate at 60%. Bumped from 55%. Could go higher given the new tested code lands but I'd rather measure first to avoid CI flake. Open to tightening to 65–70% in a follow-up.

3. Mongo supports_transactions = True is the adapter claim, not a deployment guarantee (still requires a replica set). I documented this in the docstring; should it surface more loudly somewhere — e.g. a runtime check in begin_transaction() that raises a clearer error than the underlying pymongo one?

4. Per-process cache under autoscaling. CachingDatabase is local to each worker. For a service running 8 replicas behind a load balancer, a write to one worker doesn't invalidate the others' cached copies until TTL expires. Documented in docs/md/observability.md but worth a louder warning if anyone tries it in production. Open to adding a "use Redis-backed cache instead" pointer in the constructor docstring.

5. OpenTelemetry tracing not included in this PR. Only metrics. A Tracer Protocol parallel to MetricsRecorder would be a natural follow-up; happy to do it as a separate PR if you'd rather not stack it on this one.