Skip to content

fix: preserve native adapter features and correct main regressions - #829

Merged
cofin merged 10 commits into
mainfrom
fix/recent-merged-review
Sep 28, 2026
Merged

cofin merged 10 commits into
mainfrom
fix/recent-merged-review

Conversation

@cofin

@cofin cofin commented Sep 27, 2026 •

Copy link
Copy Markdown
Member

Fix connection cleanup, fixture JSON round trips, quoted migration identifiers, and shared SQL expression ownership on main. Preserve supported native adapter features and avoid redundant SQLGlot tree copies.

Changes

  • Preserve JSON fixture strings such as "true", "null", and "[1]" across export/load, including SQLite. Match unknown columns by exact name while retaining sparse rows and upsert controls. Older pre-encoded object/array strings must be represented as fixture objects/arrays; the testing guide explains this ambiguity and migration.
  • Preserve quoted migration table names through DDL and tracking queries, including spaces and mixed case. Reuse parsed identifier components and distinguish explicit table quotes from optimizer-generated quotes.
  • Preserve MySQL Connector native async pooling and direct connections, with optional-version compatibility and cleanup on failed initialization. Restore native MySQL-family shutdown defaults and retire PyMySQL connections after rollback failure. Preserve asyncmy native LOCAL INFILE handoff bindings.
  • Preserve Oracle synchronous Thick-mode configuration while rejecting unsupported async Thick-mode requests. Keep native pool close guards, custom LOB conversion, JSON callback chaining, and established typing boundaries.
  • Fix Db2 pool/session cleanup during failures and cancellation, preserve unknown native rowcounts, and retain native string-search start/occurrence arguments.
  • Fix Spanner schema reflection bindings, preserve JOIN hints and ordinary comments, and handle qualified sequence names and IF NOT EXISTS without adding a parsing stage. Invalidate cached row converters when the configured JSON deserializer changes, preserving mutable settings behavior.
  • Isolate CTE nodes when finalizing builders and prevent callers from mutating the column-pruning cache. Avoid repeated copies when building subqueries, count queries, Db2 SQL, and disposable adapter, fixture, and migration expressions. Retain defensive copies at caller/cache boundaries and preserve SQLGlot cached generator dispatch.
  • Retain SQL Server migration schema restoration state when ALTER USER or commit fails, allowing cleanup to retry. Document that the default schema belongs to the database user and requires a dedicated migration principal without concurrent sharing.
  • Correct asyncpg stack telemetry and documentation: execution is sequential with cached prepared statements, not native pipelining.

Retains the supported native adapter features and existing compiled-helper boundaries. No broad API redesign or dependency change. Includes focused regressions, fixture guidance, and unreleased notes.

Validation

  • Focused no-service tests passed for fixture round trips, MySQL/Oracle/Db2 lifecycle handling, Spanner SQL and metadata, and quoted builder/migration identifiers.
  • Scoped Mypy, Pyright, Ruff, formatting, and diff checks passed for the initial fixes. Follow-up Oracle and MySQL changes passed 22 and 7 focused no-service tests respectively, plus scoped Pyright and Ruff. Asyncpg metadata changes passed Ruff and source review. A hosted unit failure exposed a removed asyncmy loader binding; all 20 affected cases passed locally after restoring it, with scoped Pyright and Ruff clean.
  • Independent correctness review followed by quality review completed for each area.
  • Changed guide/changelog documentation audits passed, including the final mode/pooling notes.

Local verification deliberately excludes the full test matrix, database services, cloud runs, and a compiled-wheel rebuild. Hosted CI must validate those paths. The fixture format still represents SQL NULL and JSON null identically; this existing limitation is unchanged.

Asyncpg batching research

Asyncpg does not expose heterogeneous result-bearing pipelining. Native executemany repeats one SQL template and fetchmany flattens results; neither preserves arbitrary per-statement results and failure indices. A simple-query multi-command string also loses those results and changes transaction behavior. Keep the current stack semantics; no private protocol workaround or unsupported single-trip claim is introduced. See the upstream pipeline request and native API.

Rebase validation: based on main a1067c856, with branch source changes preserved. Reconciled changelog entries, checked release-history preservation, and passed the focused documentation audit and whitespace checks. No runtime suites were rerun for this documentation-only update.

SQLGlot follow-up validation

Scoped ownership and generation tests passed: 304 core/builder/count cases, 67 builder cases, 2 cross-dialect immutability regressions, 47 Db2 cases, 143 adapter cases, and 49 fixture/migration cases. Scoped Ruff, formatting, Pyright, and whitespace checks passed. Independent correctness and quality reviews approved the final change. No benchmark speedup or compiled-wheel result is claimed; no database services or full suites were run. Existing adapter rewrite-cache misses and Spanner script classification still have separate parsing paths and were not redesigned by this copy-ownership change.

SQL Server restoration follow-up: reproduced ALTER/commit failures for both adapters before the fix; 15 focused migration-schema tests passed after it. Scoped Ruff, formatting, Pyright, and whitespace checks passed, followed by independent correctness and quality review. Existing public migration hooks remain; broader driver API removals are review proposals only.

Spanner cache follow-up: one regression reproduced the stale deserializer with reused native metadata, then passed after callable-identity invalidation. Scoped mypy, Ruff, formatting, and whitespace checks passed; independent correctness and quality review approved. No settings freeze, new public API, or additional cache was introduced.

CI repair: register the consumed BigQuery storage_write_stream_type setting in the shared feature contract (exact regression passed). Narrow the parsed SELECT and set-operation types in the Db2 ownership test; scoped mypy and Pyright pass. Docker pre-pulls retry up to three times with bounded backoff and preserve failure after exhaustion. YAML, shell syntax, scoped zizmor, and fake-command success/retry/failure checks passed. No local database services or broad test suites were run. Fresh hosted checks are pending on the updated head.

@codecov-commenter

codecov-commenter commented Sep 27, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 93.37017% with 24 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.15%. Comparing base (6bf25f1) to head (825edc7).
⚠️ Report is 3 commits behind head on main.

Files with missing lines Patch % Lines
sqlspec/core/statement.py 25.00% 5 Missing and 1 partial ⚠️
sqlspec/builder/_parsing_utils.py 84.37% 2 Missing and 3 partials ⚠️
sqlspec/adapters/mysqlconnector/config.py 91.42% 3 Missing ⚠️
sqlspec/adapters/pymysql/pool.py 60.00% 1 Missing and 1 partial ⚠️
sqlspec/adapters/aiomysql/config.py 66.66% 1 Missing ⚠️
sqlspec/adapters/asyncmy/config.py 66.66% 1 Missing ⚠️
sqlspec/adapters/oracledb/_typing.py 87.50% 0 Missing and 1 partial ⚠️
sqlspec/adapters/oracledb/config.py 93.75% 0 Missing and 1 partial ⚠️
sqlspec/builder/_ddl.py 75.00% 0 Missing and 1 partial ⚠️
sqlspec/builder/_dml.py 87.50% 0 Missing and 1 partial ⚠️
... and 2 more
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #829      +/-   ##
==========================================
- Coverage   81.31%   81.15%   -0.16%     
==========================================
  Files         521      521              
  Lines       78026    78073      +47     
  Branches    11224    11235      +11     
==========================================
- Hits        63443    63362      -81     
+ Misses      11138    11094      -44     
- Partials     3445     3617     +172     
Flag Coverage Δ
integration 62.43% <61.87%> (-0.05%) ⬇️
py3.10 79.38% <92.81%> (+0.19%) ⬆️
py3.11 79.38% <92.81%> (+0.18%) ⬆️
py3.12 79.37% <92.81%> (+0.18%) ⬆️
py3.13 79.38% <92.81%> (+0.18%) ⬆️
py3.14 80.50% <92.77%> (+0.19%) ⬆️
unit 71.60% <91.43%> (+0.17%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
sqlspec/adapters/adbc/core.py 84.50% <100.00%> (ø)
sqlspec/adapters/aiomysql/_typing.py 94.82% <100.00%> (+0.59%) ⬆️
sqlspec/adapters/asyncmy/_typing.py 96.15% <100.00%> (+1.86%) ⬆️
sqlspec/adapters/bigquery/core.py 78.03% <100.00%> (ø)
sqlspec/adapters/db2/_typing.py 91.58% <100.00%> (-0.31%) ⬇️
sqlspec/adapters/db2/core.py 97.31% <100.00%> (ø)
sqlspec/adapters/db2/pool.py 98.18% <100.00%> (+0.08%) ⬆️
sqlspec/adapters/duckdb/core.py 87.10% <100.00%> (ø)
sqlspec/adapters/mssql_python/driver.py 89.03% <100.00%> (ø)
sqlspec/adapters/mysqlconnector/_typing.py 94.44% <100.00%> (+0.39%) ⬆️
... and 33 more

... and 18 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@cofin cofin changed the title fix: address adapter, fixture, and migration review regressions fix: preserve native adapter features and correct main regressions Sep 27, 2026
@cofin
cofin force-pushed the fix/recent-merged-review branch 2 times, most recently from 31fb643 to 7adaf4f Compare September 28, 2026 14:11
@cofin
cofin force-pushed the fix/recent-merged-review branch from 7adaf4f to 825edc7 Compare September 28, 2026 19:25
@cofin
cofin merged commit 8148112 into main Sep 28, 2026
18 of 20 checks passed
@cofin
cofin deleted the fix/recent-merged-review branch September 28, 2026 19:29
cofin added a commit that referenced this pull request Sep 29, 2026
## Summary

Prepares the v0.65.0 release:
- Bumps project version to `0.65.0` in `pyproject.toml` and updates
`uv.lock`.
- Updates `docs/changelog.rst` with the `v0.65.0` release notes covering
all 14 PRs merged since `v0.64.0`:
- IBM Db2 database adapter, SQLGlot dialect, and extension support
(#811, #823).
- Spanner and Spangres SQLGlot dialect isolation and AST support (#813).
  - Spanner adapter & store optimizations and release fixes (#814).
- MySQL family adapter consolidation and data-dictionary reorganization
(#821).
  - OracleDB adapter and store cleanup (#817, #823).
- Native controls and performance optimizations for BigQuery (#812),
PostgreSQL (#822), SQLite (#820), and MSSQL (#819).
  - Native Arrow capabilities and parse-once bindings (#818).
- Fixture JSON export/load, conflict keys normalization, and upsert
options (#824, #825, #826, #827).
- Quoted and schema-qualified migration `version_table` identifiers
(#815, #828, #829).
  - Builder CTE and column pruning ownership fixes (#829).

## Validation

- `make docs` passed without warnings or errors.
- `make lint` (ruff, prek, mypy, pyright, slotscheck, zizmor) passed
with 0 errors.
- Targeted unit tests passed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants