Skip to content

Close SQLite connections so PHC acquisition cannot stall silently - #6

Merged
ahmadexp merged 3 commits into
mainfrom
fix/sqlite-fd-leak
Jul 27, 2026
Merged

Close SQLite connections so PHC acquisition cannot stall silently#6
ahmadexp merged 3 commits into
mainfrom
fix/sqlite-fd-leak

Conversation

@ahmadexp

Copy link
Copy Markdown
Owner

Problem

Raw PHC acquisition stopped a few minutes after the collector started. The
Observatory showed 0.0 / 1.0 Hz acquired and "Waiting for direct PHC samples",
while systemctl still reported the unit as active (running).

A sqlite3.Connection used directly as a context manager commits or rolls back
the transaction but never closes the handle. Both the PHC ring store and the
experiment recorder relied on that, so every recorded sample leaked a descriptor
pair. Measured on the reference host at 1 Hz:

  • ~4 descriptors leaked per sample, reaching the 1024 soft limit in ~168 s
  • the store contained exactly 168 rows at that point
  • 337 × experiments.sqlite3 + 337 × its WAL + 169 × phc-samples.sqlite3 + 168 × its WAL

Once the limit was hit, /dev/ptp* could no longer be opened either, so
take_phc_sample() returned None and the loop no-opped indefinitely at 0% CPU
with nothing further logged. That is why the failure was silent and why a
restart appeared to "fix" it — a fresh process buys another ~168 s.

Changes

  • Add an owning _session() context manager to both stores that commits or
    rolls back and then always closes. Applied to all 12 call sites, including
    the four with self._lock, self._connect() as connection: variants (one of
    which is record_phc, on the per-sample path).
  • Log the idle/resumed transitions once each instead of every cycle, which had
    been filling the journal at the Sync cadence.
  • Exit for a supervisor restart only after acquisition has worked and then
    stopped
    , which is the condition that indicates an unrepairable state
    (exhausted descriptors, revoked store path, PHC removed). A process that has
    never acquired now waits quietly for the cascade instead of restart-looping on
    a cold boot, where no namespaces exist yet.

Tests

tests/test_phc_store.py gains descriptor-lifetime coverage for both the ring
store and the experiment recorder. Two notes on making them meaningful:

  • record_phc returns early without an active run, so the test starts a run and
    asserts the rows were actually written — otherwise it passes while exercising
    nothing.
  • The probe calls gc.collect() before counting, so a descriptor still
    reachable after collection is a genuine leak rather than platform GC timing.

Both fail on the previous code and pass here. Full suite: 81 tests.

Verification on the reference host

Seven-stage cascade, 1 Hz, collector running this branch:

before after
descriptors 52 → 204 in 75 s 11, flat over 2.5 min
sqlite handles 193 and climbing 0
restarts dies ~every 168 s 0
achieved_rate_hz 0.0 1.0000, healthy: true, 0 gaps

At the observed uptime the previous code would already have accumulated ~440
descriptors.

Ahmad Byagowi added 3 commits July 27, 2026 08:57
A sqlite3.Connection used directly as a context manager commits or rolls back
the transaction but never closes the handle. Both the PHC ring store and the
experiment recorder relied on that, so every recorded sample leaked a
descriptor pair. At the Sync cadence the collector reached its 1024-descriptor
limit in under three minutes; after that /dev/ptp* could not be opened either,
take_phc_sample() returned None, and the loop spun silently forever. systemd
still reported the unit active, so the only visible symptom was the Observatory
reporting 0.0 Hz acquired and waiting for direct PHC samples.

- add an owning _session() context manager to both stores that commits or
  rolls back and then always closes (10 call sites)
- log unavailable samples instead of skipping silently, and exit for a
  supervisor restart after a sustained run of cycles delivering nothing
- add regression tests asserting descriptors stay flat across sustained
  writes and recorded samples

Verified: 1800 writes at a 256-descriptor limit produce zero descriptor
growth; the same load leaks on the previous code.
The watchdog added in the previous commit could not tell a broken collector
from one that had not started yet. On a cold boot the cascade is not running,
so there are no mapped PHCs, every cycle legitimately yields nothing, and the
collector exited every ten seconds into a supervisor restart loop.

Exit only once a sample has been delivered and delivery then stops, which is
the condition that actually indicates an unrepairable state. A process that has
never acquired now waits quietly for the cascade instead. Also report the
idle/resumed transitions once each rather than logging every cycle, which was
filling the journal at the Sync cadence.
The first pass migrated only the plain "with self._connect()" form and missed
four sites that acquire the lock in the same statement, including record_phc,
which runs once per acquired sample. On hardware the collector therefore still
leaked two descriptors per sample through experiments.sqlite3 while the PHC ring
store stayed flat.

The regression test also passed for the wrong reason: record_phc returns early
without an active run, so it exercised nothing. It now starts a run, asserts the
rows were actually written, and collects garbage before counting descriptors so
the measurement is not sensitive to platform GC timing.
@ahmadexp
ahmadexp merged commit 6e63303 into main Jul 27, 2026
1 check passed
@ahmadexp
ahmadexp deleted the fix/sqlite-fd-leak branch July 27, 2026 17:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant