Investigate repeated MQTT ping-response timeouts during dev ingestion

[n30nex]

During native Pi 5 dev-subscriber validation, the server logged repeated pingresp not received, disconnecting events from both MQTT workers and then reconnected. The final combined build recorded 15 connection-loss events over roughly ten minutes, with no application restart, no packet-list HTTP 500s and successful materialized-view refreshes. Ingestion resumed after reconnection.

This is separate from #111, #112, #113 and #115. The cause has not been established. The current worker performs ingestion synchronously from its subscription callback; the message deadline is 30 seconds, keepalive is 30 seconds and ping timeout is 10 seconds. Fixed client IDs derived only from broker name also deserve a parallel-instance check.

Proposed next investigation: measure callback duration, DB/pool wait and incoming/processed cadence, then distinguish broker/network behavior from receive-loop blocking. Any concurrency change should have bounded work and explicit delivery/acknowledgement semantics, with a regression test under slow database work and reconnect.

No credentials, packet bodies or node locations are included in this report.

[n30nex]

Continuing the investigation now. I will compare the existing subscriber against a counting-only subscriber using unique client IDs, collect aggregate callback/database-wait evidence, and reproduce the behavior with a local MQTT fixture. The aim is to keep protocol handling responsive with bounded work, without introducing unlimited per-message goroutines or silently acknowledging discarded work. Each candidate will be tested on the Pi and the public preview before a focused PR is submitted.

[n30nex]

The client-isolation fix is submitted as #117 and is running in the development preview. Its tests inspect actual CONNECT packets, catch the duplicate name-only IDs, and verify stable IDs across reconnects.

The previous application logged 22 ping-response disconnects in 15 minutes. The candidate passed a ten-minute Pi 5 soak with zero disconnects, API 500 log entries or restarts and 1,585 new retained observations across both brokers. The corresponding source and build identifiers are available from the preview's source page.

I am leaving this issue open: the longer callback diagnostic also captured occasional 15–17 second calls. Client isolation does not fix slow database work, and this short comparison does not establish long-duration reliability or identify an external colliding connection from broker administration logs. Those remain separate follow-up checks.