[lesnik512]

Problem

The blackhole drill used to disconnect the primary from the Docker network. That made pg-primary stop resolving, so every connect to it failed fast with a DNS error. A real dead host behind a firewall or in a cloud VPC still resolves, and its packets are silently dropped. The drill now drops them with iptables in the dr container. That surfaced three things, all reproduced in the drill:

1. asyncpg multi-host connect never fails over to the next host. In asyncpg 0.31, one timeout (60s by default) covers the loop over every host, and it moves to the next host only on OSError. A connect to a dropped host hangs until the timeout runs out, so the replica is never tried. A probe in the drill cluster with the primary listed first: timeout=10 gave TimeoutError after 10.0s; timeout=200 connected after 133.3s, once the kernel gave up on SYN retries. Upstream: Timeout configuration for multi-host config MagicStack/asyncpg#1155 and the open per-host connect_timeout PR Add connect_timeout: a per-host deadline for multi-host connect MagicStack/asyncpg#1393.
2. The LISTEN connection ignored the engine's connect configuration. _open_listen_connection rebuilt connect kwargs from the URL with dialect.create_connect_args and called asyncpg.connect itself. That skipped async_creator, so the workaround for (1) could not cover LISTEN. It also skipped connect_args such as ssl or server_settings.
3. Application sessions hang on stale pooled connections. After a silent loss, idle pooled connections still point at the old primary. AsyncSession.__aexit__ shields close() from cancellation, so an asyncio.timeout around a publish does not end the hang: the rollback on the dead socket runs until TCP gives up.

Changes

Library (subscriber/usecase.py):

• LISTEN connects through the engine: engine.raw_connection(), take driver_connection, then detach(). URL, connect_args, async_creator and pool events now all apply. Detaching returns the slot to the pool and keeps the pool from ever handing the LISTEN connection to another checkout. driver_connection is read before detach(), which clears it.
• LISTEN is opened before the fetch connection. Workers start first and check out their writer connections eagerly. In a pool sized exactly max_workers + 1 (the minimum broker.start() accepts), opening LISTEN after the fetch connection left it waiting out pool_timeout, which fell back to polling.
• add_listener is capped at query_timeout. A timeout there raises instead of falling back to polling. The checkout can hand back an idle socket that went silent with the old primary, and polling would leave NOTIFY off until the fetch loop next failed. Other LISTEN failures (permissions, PgBouncer, pool exhaustion) still fall back to polling.
• A connection whose LISTEN setup failed is closed with the bounded _close_listen_connection, not a bare close() that could hang on a dead socket.

Drill and docs:

• tests_dr/Dockerfile adds iptables for the dr service, which now gets NET_ADMIN. The main Dockerfile is unchanged.
• The drill engine uses an async_creator that connects to each host under its own 3s timeout with target_session_attrs="primary" and command_timeout=3. DR_CLUSTER_DSN is gone; the drill skips unless PRIMARY_DSN is set, which only the compose file does.
• The NOTIFY check now starts its 3s window when the final publish returns. That publish can pay one command_timeout on a remaining stale pooled connection, which says nothing about NOTIFY.
• docs/usage/subscriber.md "Database failover" documents the gap and the per-host async_creator pattern, including why command_timeout matters for application sessions. Connection budget, troubleshooting and the checklist now describe LISTEN as a detached pool connection.

Verification

• New tests, red before the change:

  • test_listen_connection_uses_engine_async_creator: the URL host is unreachable.invalid, so only the creator reaches Postgres. It failed with the gaierror fallback to polling.
  • test_listen_connection_uses_engine_connect_args: checks application_name from connect_args on the LISTEN backend in pg_stat_activity. It failed with 0 rows.
  • test_listen_connection_fits_a_pool_sized_to_the_budget: pool_size=3, max_overflow=0, max_workers=2. It failed with QueuePool limit of size 3 overflow 0 reached before LISTEN moved ahead of the fetch connection.
  • test_open_listen_connection_raises_on_hung_add_listener and test_open_listen_connection_detaches_pooled_connection (unit).

• async_creator arrived in SQLAlchemy 2.0.16, and the floor on Python 3.11 and 3.12 is 2.0.0. The library does not use it, so test_listen_connection_uses_engine_async_creator skips where SQLAlchemy is older, and the docs pattern names the minimum version. The floors suite on 3.11 against SQLAlchemy 2.0.0 and asyncpg 0.29.0: 827 passed, 3 skipped.

• just test: 828 passed, 100% coverage.

• just test-dr, twice:

  Scenario	Publish recovered	Subscriber recovered	Redeliveries
  kill	1.0s / 0.9s	3.1s / 10.8s	0
  blackhole (packets dropped)	9.3s / 9.3s	31.6s / 32.1s	0

  Before this PR, the packet-drop blackhole did not recover within the 90s budget.

• just lint: ruff is clean. ty shows only the 6 warnings main already has.

Notes

• With the LISTEN connection now coming from the pool, pool connect and checkout events fire for it, and pool_pre_ping applies to its checkout.
• One drill run failed before any of this, because the replica exited during pg_basebackup. A clean start worked. I think the primary's healthcheck passes while the entrypoint's init server is still running on the Unix socket only. It is not addressed here.

[github-actions]

Benchmark gate

✅ gate passed

scenario	msg/s	delete/msg	WALrec/msg	WALB/msg	fpi	upd	del	dead_tup
consumer/w1/b10	1023	1.000	6.71	920	242	5000	5000	10000
consumer/w1/b100	1600	1.000	7.11	1154	243	5000	5000	10000
consumer/w2/b10	1748	1.000	6.71	921	242	5000	5000	10000
consumer/w2/b100	2461	1.000	7.11	1154	243	5000	5000	10000
consumer/w4/b10	1346	1.000	6.73	946	242	5000	5000	10000
consumer/w4/b100	1838	1.000	7.16	1191	263	5000	5000	10000
consumer/w1/b100/tfbs100	8886	0.010	6.11	1132	244	5000	5000	10000
consumer/w1/b100/interval1-10	1113	1.000	7.11	1154	243	5000	5000	10000
producer/w1/b100	4331	0.000	3.04	597	1	0	0	0
producer/w1/b100/ppt1	1102	0.000	4.04	635	0	0	0	0

Gated (fails the build): delete_calls + tuple counters (upd/del/ins) + the producer's insert_calls, exact; select_calls and other_calls within +2; wal_records within a 10% band; fetch_calls within 2x. msg/s, WAL bytes and total calls are informational (timing/FPI noise).