Skip to content

perf(postgres): reduce round trips and retain native adapter controls - #822

Merged
cofin merged 10 commits into
mainfrom
feat/postgres-adapters-optimization
Sep 28, 2026
Merged

cofin merged 10 commits into
mainfrom
feat/postgres-adapters-optimization

Conversation

@cofin

@cofin cofin commented Sep 25, 2026 •

Copy link
Copy Markdown
Member

Reduce ADK store round trips with RETURNING, bound psycopg COPY buffering, and release failed psqlpy initialization attempts. Preserve native PostgreSQL configuration controls and quoted typing boundaries.

  • Forward asyncpg statement timeouts consistently across queries, batches, scripts, stacks, and streams, including explicit zero/None values. PgBouncer mode disables both implicit caching and explicit stack preparation while retaining cancellation-safe transaction cleanup. Outside PgBouncer mode, asyncpg stacks execute sequentially with prepared-statement reuse; observer metadata now reflects that behavior.
  • Preserve asyncpg custom codecs and psycopg connection-scoped JSON codecs. Support psycopg native null pools without forwarding the SQLSpec flag to libpq or discarding native concurrency limits.
  • Forward supported CockroachDB startup settings with validated boolean/integer values. Preserve retry classification and cycle protection.
  • Coerce explicit psqlpy VECTOR parameters with its native PgVector encoder, independent of the Python pgvector package. Keep parse-once execution and targeted lifecycle fixes.

The psqlpy pipeline optimization remains deferred pending cancellation, preflight, and error-attribution fixes. Schema metadata caching remains deferred until DDL/search-path invalidation is defined. No gateway_region connection option is advertised; it is not a supported session setting.

Validation: 10 focused no-service native-option and cancellation tests passed; scoped Pyright reported 0 errors; scoped Ruff and git diff --check passed. Database-service integration, native compiled-wheel behavior, and documentation builds were not run locally. Hosted CI supplies the broader validation boundary.

A previous hosted integration attempt stopped before tests when Docker timed out contacting https://icr.io/v2/ while pulling a service image. This is an infrastructure validation block; no passing integration result is claimed.

Rebase validation: based on main a1067c856, with branch source changes preserved. Reconciled changelog entries, checked release-history preservation, and passed the focused documentation audit and whitespace checks. No runtime suites were rerun for this documentation-only update.

CI repair: register the consumed BigQuery storage_write_stream_type setting in the shared feature contract (exact regression passed). Docker pre-pulls retry up to three times with bounded backoff and preserve failure after exhaustion. YAML, shell syntax, scoped zizmor, and fake-command success/retry/failure checks passed. No local database services or broad test suites were run. Fresh hosted checks are pending on the updated head.

@cofin
cofin force-pushed the feat/postgres-adapters-optimization branch from f90cc88 to 6105085 Compare September 25, 2026 00:53
@codecov-commenter

codecov-commenter commented Sep 25, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 76.19048% with 65 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.06%. Comparing base (0007c1f) to head (7463ad1).

Files with missing lines Patch % Lines
sqlspec/adapters/psycopg/driver.py 46.66% 10 Missing and 6 partials ⚠️
sqlspec/adapters/psqlpy/adk/store.py 53.12% 14 Missing and 1 partial ⚠️
sqlspec/adapters/psqlpy/config.py 68.42% 6 Missing ⚠️
sqlspec/adapters/psqlpy/type_converter.py 61.53% 4 Missing and 1 partial ⚠️
sqlspec/adapters/asyncpg/config.py 73.33% 2 Missing and 2 partials ⚠️
sqlspec/adapters/asyncpg/core.py 69.23% 4 Missing ⚠️
sqlspec/adapters/asyncpg/driver.py 86.36% 1 Missing and 2 partials ⚠️
sqlspec/adapters/asyncpg/litestar/store.py 66.66% 1 Missing and 2 partials ⚠️
sqlspec/adapters/cockroach_psycopg/adk/store.py 66.66% 0 Missing and 2 partials ⚠️
sqlspec/adapters/cockroach_psycopg/core.py 93.75% 1 Missing and 1 partial ⚠️
... and 3 more
Additional details and impacted files
@@           Coverage Diff            @@
##             main     #822    +/-   ##
========================================
  Coverage   81.06%   81.06%            
========================================
  Files         521      521            
  Lines       77879    78026   +147     
  Branches    11188    11224    +36     
========================================
+ Hits        63129    63250   +121     
- Misses      11121    11140    +19     
- Partials     3629     3636     +7     
Flag Coverage Δ
integration 62.46% <55.31%> (-0.03%) ⬇️
py3.10 79.29% <75.09%> (+0.01%) ⬆️
py3.11 79.29% <75.09%> (+<0.01%) ⬆️
py3.12 79.29% <75.09%> (+<0.01%) ⬆️
py3.13 79.29% <75.09%> (+<0.01%) ⬆️
py3.14 80.40% <73.74%> (-0.01%) ⬇️
unit 71.43% <47.25%> (+0.04%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
sqlspec/adapters/cockroach_asyncpg/config.py 92.92% <100.00%> (+0.10%) ⬆️
sqlspec/adapters/cockroach_asyncpg/core.py 95.79% <100.00%> (+1.00%) ⬆️
sqlspec/adapters/cockroach_psycopg/config.py 90.67% <100.00%> (-0.24%) ⬇️
sqlspec/adapters/psycopg/_typing.py 92.98% <100.00%> (+0.12%) ⬆️
sqlspec/adapters/psycopg/config.py 90.61% <100.00%> (+0.61%) ⬆️
sqlspec/adapters/asyncpg/adk/store.py 65.08% <75.00%> (-0.09%) ⬇️
sqlspec/adapters/cockroach_psycopg/adk/store.py 56.67% <66.66%> (ø)
sqlspec/adapters/cockroach_psycopg/core.py 96.94% <93.75%> (-1.04%) ⬇️
sqlspec/adapters/psqlpy/core.py 77.09% <33.33%> (-0.30%) ⬇️
sqlspec/adapters/psqlpy/litestar/store.py 86.11% <81.81%> (-0.56%) ⬇️
... and 8 more

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@cofin
cofin force-pushed the feat/postgres-adapters-optimization branch 7 times, most recently from ba2740b to bebc624 Compare September 27, 2026 18:44
@cofin cofin changed the title refactor(adapters): clean up PostgreSQL and CockroachDB adapters perf(postgres): reduce store round trips and bound COPY buffering Sep 27, 2026
@cofin cofin changed the title perf(postgres): reduce store round trips and bound COPY buffering perf(postgres): reduce round trips and retain native adapter controls Sep 27, 2026
@cofin
cofin force-pushed the feat/postgres-adapters-optimization branch from 83ff02a to caf9e24 Compare September 27, 2026 22:06
cofin and others added 10 commits September 28, 2026 14:11
- AsyncPG: remove dead execution helpers, use singleton RNG and exception chain depth limit, optimize connection pool configs and ADK session round-trips.
- Psycopg: simplify connection and driver hierarchy, remove dead passthrough methods, add typed query conversion helper without byte encoding, streamline ADK and Litestar session stores.
- PsqlPy: remove obsolete driver wrappers and query cache indirection, consolidate connection pool configs, optimize Arrow conversion helpers.
- CockroachDB (AsyncPG & Psycopg): inherit from PostgreSQL base driver and exception handlers, fix follower read transaction commit leak, specialize data dictionaries with per-connection version caching, align multi-region configuration options.
@cofin
cofin force-pushed the feat/postgres-adapters-optimization branch from 54eafca to 7463ad1 Compare September 28, 2026 14:13
@cofin
cofin merged commit 6bf25f1 into main Sep 28, 2026
23 checks passed
@cofin
cofin deleted the feat/postgres-adapters-optimization branch September 28, 2026 19:22
cofin added a commit that referenced this pull request Sep 29, 2026
## Summary

Prepares the v0.65.0 release:
- Bumps project version to `0.65.0` in `pyproject.toml` and updates
`uv.lock`.
- Updates `docs/changelog.rst` with the `v0.65.0` release notes covering
all 14 PRs merged since `v0.64.0`:
- IBM Db2 database adapter, SQLGlot dialect, and extension support
(#811, #823).
- Spanner and Spangres SQLGlot dialect isolation and AST support (#813).
  - Spanner adapter & store optimizations and release fixes (#814).
- MySQL family adapter consolidation and data-dictionary reorganization
(#821).
  - OracleDB adapter and store cleanup (#817, #823).
- Native controls and performance optimizations for BigQuery (#812),
PostgreSQL (#822), SQLite (#820), and MSSQL (#819).
  - Native Arrow capabilities and parse-once bindings (#818).
- Fixture JSON export/load, conflict keys normalization, and upsert
options (#824, #825, #826, #827).
- Quoted and schema-qualified migration `version_table` identifiers
(#815, #828, #829).
  - Builder CTE and column pruning ownership fixes (#829).

## Validation

- `make docs` passed without warnings or errors.
- `make lint` (ruff, prek, mypy, pyright, slotscheck, zizmor) passed
with 0 errors.
- Targeted unit tests passed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants