Skip to content

store: Postgres over the host's connector (no own socket; ticketed open) - #19

Closed
MattJackson wants to merge 4 commits into
devfrom
lane-dg-storebridge
Closed

MattJackson wants to merge 4 commits into
devfrom
lane-dg-storebridge

Conversation

@MattJackson

@MattJackson MattJackson commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Draft: depends on busbar lane-dg-storebridge@1c9e599003ad2b2789b271482824b52f390c3200 (not yet on GitHub). CI can't resolve the busbar pin until that commit is pushed. Once it is, re-run scripts/fleet/repin.sh (and cargo xtask fleet render busbar-store-postgres upstream if fleet check flags the rendered files), then mark this ready.

What

Ports the Postgres store onto the host's connector, per ARCHITECT rulings 2026-10-03: Q-L14-1, Q-L16-2, and at 12:10Z STORE-KEEP and TLS. Background (busbar THE DESIGN): every call is Ready or Pending(wake), and no plugin opens its own socket.

  • No socket of its own. The postgres driver (its own sockets and blocking I/O) is gone from the logic crate. src/pgwire.rs is a small async Postgres frontend built on the sans-IO postgres-protocol (messages, SCRAM-SHA-256, md5) and postgres-types (the same binary ToSql/FromSql the 1.5.5 driver used). Every byte goes through busbar's store SDK wire.
  • One need. store_door!(…, needs: NEEDS): one outbound tcp need, egress class operator-infrastructure. The target is the DSN's host:port.
  • Ticketed open + connect step. open only parses the settings, using the same refusal texts as before. StoreSlots::connect connects, authenticates and runs migrate() (advisory lock and all), so an unreachable or refusing server still fails the load at boot, in the driver's words: error connecting to server: …, password scrubbed.
  • One kept connection, as in 1.5.5 (STORE-KEEP).
    • The instance holds busbar's store-SDK kept set (wire::Pool), bounded at KEPT_CONNECTIONS = 1. That matches 1.5.5's one mutex-guarded connection; the 1.5.5 store has no pool setting.
    • The connect step and every op run under wire::drive_kept. Only a newly established connection does TLS + startup/auth; ops reuse the kept one and take turns on it, as they took the mutex.
    • A connection is kept only while its session is idle: last ReadyForQuery status I, nothing unread, no rollback pending (pgwire::Client::release). Any other end (wire failure, protocol doubt, a transaction left open or failed) discards it, and the next op connects afresh. 1.5.5 needed a restart there.
    • No retry was added that 1.5.5 did not have.
    • Dropping the store closes its instance and its connection.
  • Op bodies are 1.5.5's. They gain only .await, on a Session instead of the mutex-guarded client. Unchanged:
    • RepeatableRead snapshots, advisory locks, SQLSTATE checks (42P01)
    • render_pg_error/scrub wording, durable op_id dedupe
    • a Transaction dropped uncommitted is rolled back before the connection's next statement, as the driver's was
    • a query is Parse/Describe/Sync then Bind/Execute/Sync, all values binary
  • TLS through the host's TLS wrap. sslmode=require|verify-ca|verify-full sends SSLRequest; on S it secures the connection with Wire::upgrade_secure (name = the DSN host). A server answering N fails the load with error performing TLS handshake: server does not support TLS. disable|allow|prefer stay plaintext, as 1.5.5's NoTls did. Behaviour note: require now verifies the server certificate and name, unlike libpq's require. The host's TLS always verifies.
  • Plugin crate. Exports only the door (export_door!); the cold-lane registration is deleted.
  • Pins. busbar 1c9e599003ad2b2789b271482824b52f390c3200 in every manifest, .busbar-ref, the rendered workflows and Cargo.lock (--locked builds offline).

Tests

The logic crate's live tests and the plugin crate's tests/common open the store through the real loader on a dispatcher, with its connections on busbar's loader test table (tcp_conns::TcpConns). Raw verification SQL uses an independent postgres driver connection (dev-dependency).

New live tests (store-postgres/src/tests/tls.rs):

  • verify_full_secures_the_connection_through_the_host: an in-test TLS proxy in front of the live server, its CA minted per run (rcgen). The proxy answers the SSLRequest with S, accepts TLS and forwards plaintext. The store opens through the loader with TcpConns::with_roots and sslmode=verify-full (host localhost), then writes and reads back.
  • an_untrusted_server_certificate_fails_the_load_in_the_drivers_words: no trust, so the load fails with error performing TLS handshake.
  • the_store_keeps_one_connection_across_its_ops: one backend pid (pg_stat_activity) across the connect step and ten ops, and dropping the store closes it.

rustls, rustls-pki-types and rcgen are dev-dependencies only.

Run against a local Postgres 17 (scram-sha-256) at its default max_connections = 100, with the real busbar release binary from the busbar tree at the pin (BUSBAR_CHECKOUT), BUSBAR_TEST_POSTGRES_URL=postgres://busbar:…@127.0.0.1:55432/busbar_pg_l16:

cargo test --workspace --locked:

Target Result
lib 87 passed
admin_api_e2e 1 passed (install_over_admin_api_then_mint_a_key_and_verify_postgres_directly)
conformance 1 passed
e2e 6 passed, including load_and_exercise_postgres_plugin_via_file_drop (1.47 s when run alone)

No store connections remain on the server after the run. cargo clippy --workspace --all-targets --locked -- -D warnings and cargo fmt --all --check are clean.

Test-only changes worth reviewing:

  • metering_splits_on_priced_from_ms_and_carries_open_classes moved to bucket 20_270_611. It shared 20_270_601 with the purge test, whose whole-bucket purge raced it.
  • is_undefined_table_matches_only_the_real_sqlstate now classifies the store's own driver errors. Two snapshot_consistent_tx tests run the helper's exact BEGIN ISOLATION LEVEL REPEATABLE READ on an independent connection.

Not exercised: TLS through the process's real connector (the TLS test uses the loader's test table, which stands in for the connector's TLS wrap); a real TLS-enabled Postgres server.

ARCHITECT rulings 2026-10-03 on Q-L14-1 and Q-L16-2 (busbar THE DESIGN: every call Ready or
Pending(wake); no plugin opens its own socket). Depends on busbar lane-dg-storebridge@c7cc4f6696.

- The logic crate drops the `postgres` driver. src/pgwire.rs is an async Postgres frontend on the
  sans-IO postgres-protocol (messages, SCRAM-SHA-256, md5) and postgres-types (the same binary
  ToSql/FromSql the 1.5.5 driver used), every byte over the op's one connection (store SDK
  wire::drive / Op::checkout). Parse/Describe/Sync then Bind/Execute/Sync, binary values; an
  uncommitted Transaction is rolled back before the connection's next statement.
- store_door!(.., needs: NEEDS): one outbound `tcp` need, operator-infrastructure, target = the
  DSN's host:port. `open` parses the settings (same refusal texts); StoreSlots::connect connects,
  authenticates and migrates, so an unreachable or refusing server still fails the load at boot in
  the driver's words ("error connecting to server: ...", password scrubbed).
- Every op is one connection: startup + auth, the 1.5.5 SQL body (now on a Session, with .await),
  Terminate. RepeatableRead snapshots, advisory locks, SQLSTATE checks, render_pg_error/scrub and
  the durable op_id dedupe are unchanged. sslmode=require|verify-* -> SSLRequest + upgrade_secure
  through the host; the other modes stay plaintext as 1.5.5's NoTls.
- The plugin crate exports only the door; the cold-lane registration is deleted.
- busbar pinned at c7cc4f6696d53654ba1ac4f8e8c4fad0b9cd9df8 (manifests, .busbar-ref, workflows).

Tests open the store through the real loader with busbar's test connection table (tcp_conns),
raw verification on an independent `postgres` connection (dev-dependency). The metering test moves
to bucket 20_270_611: it raced the purge test on 20_270_601.
…root connector wakes a plugin's pending connection and drives dispatcher-worker sockets on its own I/O thread; the real-binary e2e tests need it)
…ion); TLS test through the host; busbar pin 180d92c205 -> 8faed15ca4

ARCHITECT rulings 2026-10-03 12:10Z (STORE-KEEP, TLS).

- STORE-KEEP: the instance holds the store SDK's kept set (wire::Pool) bounded at KEPT_CONNECTIONS = 1,
  1.5.5's one mutex-guarded connection (the 1.5.5 store has no pool setting). The connect step and
  every op run under wire::drive_kept: a kept connection skips TLS + startup/auth; the connection is
  kept only if the session is idle (last ReadyForQuery status I, nothing unread, no rollback
  pending; pgwire::Client::release), and a client dropped any other way (early return, protocol
  failure, open or failed transaction) discards it, so the next op connects afresh. No retry 1.5.5
  did not have.
- TLS: unchanged mapping (disable/allow/prefer plaintext as 1.5.5 NoTls; require/verify-* SSLRequest,
  'S', upgrade_secure on the DSN host; 'N' fails the load). Docs say the host always verifies.
- Tests: src/tests/tls.rs (live): verify_full_secures_the_connection_through_the_host (in-test TLS
  proxy with a per-run rcgen CA in front of the live server; TcpConns::with_roots),
  an_untrusted_server_certificate_fails_the_load_in_the_drivers_words,
  the_store_keeps_one_connection_across_its_ops (one backend pid across the connect step and ten
  ops). The harness closes each test store's instance on drop. rustls/rustls-pki-types/rcgen are
  dev-dependencies only.
- Pin: busbar 8faed15ca46ec48d5818dc0bd0c22a9c15218de1 (manifests, .busbar-ref, workflows, Cargo.lock).
…dStore closes its instance on drop)

The harness's own close-on-drop workaround is struck: busbar's LoadedStore now closes its instance
when dropped, and its kept connection with it. the_store_keeps_one_connection_across_its_ops again
asserts that dropping the store closes its backend connection. The full live suite runs at the
server's default max_connections (100).
@MattJackson

Copy link
Copy Markdown
Contributor Author

Replaced by #20 (same commits plus the RED-arm split and Cargo.lock re-derived from busbar's lock at the pin).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant