Skip to content

Boot quiescence: stop paying ~185 rows written + ~9.7k read per hive wake - #23

Open
tieguy wants to merge 3 commits into
daniloc:mainfrom
tieguy:fix/boot-quiescence
Open

Boot quiescence: stop paying ~185 rows written + ~9.7k read per hive wake#23
tieguy wants to merge 3 commits into
daniloc:mainfrom
tieguy:fix/boot-quiescence

Conversation

@tieguy

@tieguy tieguy commented Aug 14, 2026

Copy link
Copy Markdown

The problem

initializeSchema runs in the HiveDO constructor — on every cold start / wake from hibernation, not once per deploy. Two of its steps are reconciliation (registering the kernel patterns in _objects; drop-and-recreating every audit trigger so the captured column list tracks the schema), and both ran unconditionally.

Measured on a live instance: 185 rows written and ~7.7k rows read per no-op wake. The wake rate is set by client reconnects (each MCP handshake, each /ws connect), so the cost has no relationship to actual use. On a free-tier account this consumed 90% of the 100k/day Durable Objects rows_written allowance — and 92% of the 5M/day rows_read allowance — on a day with six agent writes, then tipped over the cap entirely and took the instance down (every hive-backed route 500s once DO writes are refused). Cloudflare's hourly analytics made the attribution unambiguous: idle-hour write totals are near-exact multiples of 185 (191 = one boot, 573 = three), and reads track writes at a constant ~52.5:1 across a 50× activity range — the same cold-start population, not traffic.

The fix

Two gates, same shape as the codebase's other self-enforcing declarations:

  1. bootFingerprint (stamped in _meta): derived from the kernel declarations plus the live per-pattern column signature — never hand-maintained. The _objects registration and the audit-trigger rebuild run only when it moves. Fail closed: a throw, a missing stamp, or any difference reconciles. Stored whole rather than hashed — a collision would silently resurrect the stale-trigger bug the always-rebuild deliberately fixed (a trigger built before a column existed omits it from _mutation_log forever), and an exact compare costs one row write per real schema change. evolution.ts still calls ensureAuditTriggers directly, so a live schema change never waits on a boot. Also: the _objects upsert gains WHERE … IS NOT excluded.… — SQLite rewrites the row on a no-op DO UPDATE, which billed a write per kernel pattern per wake independently of the gate.

  2. runOnce (completion stamps in _completed_migrations): the pre-born-hashing token scrub was "idempotent, a near-no-op after the first cold start" — true of its writes, false of its reads: six full scans of _mutation_log (6,000 rows) on every wake forever, to re-discover there was nothing to do. A migration whose am-I-done? question is a table scan can't answer it per boot. Stamped only on success, so a part-way failure retries rather than being silently marked done. The scrub moves from hive.ts into schema.ts so the whole boot sequence is initializeSchema + applyTokenHashScrub and the oracle can measure both halves.

The oracle

boot-idempotence.test.ts counts actual rowsWritten/rowsRead through a proxied SqlStorage:

  • a second consecutive boot writes zero rows (fails on any future unconditional write, whichever line introduces it);
  • a quiescent boot's read count is identical at two data volumes — a property, not a threshold, so no magic constant, and any future boot-time table scan breaks it.

Verified by negative control: removing the runOnce gate fails it by name (boot reads grew with data volume (619 → 8219)).

One settling boot is allowed — reconciliation and the GC sweeps exist to do work; the pathology was paying that cost every time, forever.

The final commit regenerates the coherence artifacts with this repo's pinned v0.27 harness (my fork runs v0.31; the cherry-picked commits carried v0.31-generated artifacts).

Observed effect: boots go from 185 writes / ~7.7k reads to 0 written / ~224 read.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Rf5RTB7iigiMpoArYzdUKN

tieguy and others added 3 commits August 14, 2026 08:03
initializeSchema runs in the HiveDO constructor, so it executes on every
cold start / wake from hibernation — a rate set by client reconnects, not
by the owner. Two of its steps are reconciliation (register kernel patterns
in _objects; drop-and-recreate every audit trigger so the captured column
list tracks the schema) and both ran unconditionally.

Measured: 185 rows written per no-op wake. Cloudflare's
durableObjectsPeriodicGroups for 2026-08-13 attributes 87,372 of 95,720
account rows_written to HiveDO (91%) on a day with six agent writes, and
the idle overnight hours are near-exact multiples of 185 (191 = one boot,
573 = three), rising to ~9.5k/hr — one cold start every ~70s.

Gate both steps on a bootFingerprint stamped in _meta, derived from the
kernel declarations plus the live per-pattern column signature. Fail
closed: a throw, a missing stamp, or any difference reconciles. Stored
whole rather than hashed — a collision would resurrect the stale-trigger
bug the always-rebuild deliberately fixed, and exact compare costs one row
write per real schema change. evolution.ts still calls ensureAuditTriggers
directly, so a live schema change never waits on a boot.

Also add WHERE ... IS NOT excluded.... to the _objects upsert: SQLite
rewrites the row on a no-op DO UPDATE, which billed a write per kernel
pattern per wake independently of the gate.

The oracle is behavioural, not structural: boot-idempotence.test.ts counts
actual rowsWritten through a proxied SqlStorage and asserts a second
consecutive boot writes zero, so any future unconditional write fails the
build regardless of which line introduces it. One settling boot is allowed
— reconciliation and the GC sweeps exist to do work; the pathology was
paying that cost every time, forever.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rf5RTB7iigiMpoArYzdUKN
The write fix (56c056a) left the read ceiling untouched. Cloudflare's
hourly figures show reads tracking writes at a constant ~52.5:1 across a
50x range of activity — three significant figures — which identifies both
as the same cold-start population rather than request traffic: ~9,700 rows
read per HiveDO wake, 4.59M/day against the 5M/day free tier.

Most of it was the same audit-trigger rebuild already gated in 56c056a
(DROP TRIGGER scans sqlite_schema, 116 rows a statement, six per pattern):
a settle boot measures 7,754 reads, a quiescent one 224.

The separate offender was the pre-born-hashing token scrub. "Idempotent, a
near-no-op after the first cold start" was true of its WRITES and false of
its READS: six full scans of _mutation_log, 6,000 rows, zero writes, on
every wake forever, to re-discover there was nothing to do. That is the
general trap — a migration whose am-I-done question is a table scan cannot
answer it per boot. runOnce gives such work a completion stamp in
_completed_migrations, stamped only on success so a part-way failure
retries rather than being silently marked done.

applyTokenHashScrub moves from hive.ts into schema.ts so the whole boot
sequence is initializeSchema + applyTokenHashScrub and the oracle can
measure BOTH halves — a test covering only the first would have passed
while the scrub burned the read budget. It also puts the migration with
the other migrations and keeps HiveDO a wiring shell.

The read oracle asserts a property, not a threshold: a quiescent boot's
read count must be IDENTICAL at two data volumes, so it needs no magic
constant and catches any future boot-time scan. Verified by negative
control — removing the gate failed it by name ("boot reads grew with data
volume (619 -> 8219)"), then restored to green.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rf5RTB7iigiMpoArYzdUKN
The cherry-picked commits carried artifacts generated by v0.31 (the fork
runs a newer harness); regenerated here so the freshness gate compares
like with like.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rf5RTB7iigiMpoArYzdUKN
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant