Close SQLite connections so PHC acquisition cannot stall silently - #6
Merged
Conversation
added 3 commits
July 27, 2026 08:57
A sqlite3.Connection used directly as a context manager commits or rolls back the transaction but never closes the handle. Both the PHC ring store and the experiment recorder relied on that, so every recorded sample leaked a descriptor pair. At the Sync cadence the collector reached its 1024-descriptor limit in under three minutes; after that /dev/ptp* could not be opened either, take_phc_sample() returned None, and the loop spun silently forever. systemd still reported the unit active, so the only visible symptom was the Observatory reporting 0.0 Hz acquired and waiting for direct PHC samples. - add an owning _session() context manager to both stores that commits or rolls back and then always closes (10 call sites) - log unavailable samples instead of skipping silently, and exit for a supervisor restart after a sustained run of cycles delivering nothing - add regression tests asserting descriptors stay flat across sustained writes and recorded samples Verified: 1800 writes at a 256-descriptor limit produce zero descriptor growth; the same load leaks on the previous code.
The watchdog added in the previous commit could not tell a broken collector from one that had not started yet. On a cold boot the cascade is not running, so there are no mapped PHCs, every cycle legitimately yields nothing, and the collector exited every ten seconds into a supervisor restart loop. Exit only once a sample has been delivered and delivery then stops, which is the condition that actually indicates an unrepairable state. A process that has never acquired now waits quietly for the cascade instead. Also report the idle/resumed transitions once each rather than logging every cycle, which was filling the journal at the Sync cadence.
The first pass migrated only the plain "with self._connect()" form and missed four sites that acquire the lock in the same statement, including record_phc, which runs once per acquired sample. On hardware the collector therefore still leaked two descriptors per sample through experiments.sqlite3 while the PHC ring store stayed flat. The regression test also passed for the wrong reason: record_phc returns early without an active run, so it exercised nothing. It now starts a run, asserts the rows were actually written, and collects garbage before counting descriptors so the measurement is not sensitive to platform GC timing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Raw PHC acquisition stopped a few minutes after the collector started. The
Observatory showed
0.0 / 1.0 Hz acquiredand "Waiting for direct PHC samples",while
systemctlstill reported the unit asactive (running).A
sqlite3.Connectionused directly as a context manager commits or rolls backthe transaction but never closes the handle. Both the PHC ring store and the
experiment recorder relied on that, so every recorded sample leaked a descriptor
pair. Measured on the reference host at 1 Hz:
experiments.sqlite3+ 337 × its WAL + 169 ×phc-samples.sqlite3+ 168 × its WALOnce the limit was hit,
/dev/ptp*could no longer be opened either, sotake_phc_sample()returnedNoneand the loop no-opped indefinitely at 0% CPUwith nothing further logged. That is why the failure was silent and why a
restart appeared to "fix" it — a fresh process buys another ~168 s.
Changes
_session()context manager to both stores that commits orrolls back and then always closes. Applied to all 12 call sites, including
the four
with self._lock, self._connect() as connection:variants (one ofwhich is
record_phc, on the per-sample path).been filling the journal at the Sync cadence.
stopped, which is the condition that indicates an unrepairable state
(exhausted descriptors, revoked store path, PHC removed). A process that has
never acquired now waits quietly for the cascade instead of restart-looping on
a cold boot, where no namespaces exist yet.
Tests
tests/test_phc_store.pygains descriptor-lifetime coverage for both the ringstore and the experiment recorder. Two notes on making them meaningful:
record_phcreturns early without an active run, so the test starts a run andasserts the rows were actually written — otherwise it passes while exercising
nothing.
gc.collect()before counting, so a descriptor stillreachable after collection is a genuine leak rather than platform GC timing.
Both fail on the previous code and pass here. Full suite: 81 tests.
Verification on the reference host
Seven-stage cascade, 1 Hz, collector running this branch:
achieved_rate_hzhealthy: true, 0 gapsAt the observed uptime the previous code would already have accumulated ~440
descriptors.