Boot quiescence: stop paying ~185 rows written + ~9.7k read per hive wake - #23
Open
tieguy wants to merge 3 commits into
Open
Boot quiescence: stop paying ~185 rows written + ~9.7k read per hive wake#23tieguy wants to merge 3 commits into
tieguy wants to merge 3 commits into
Conversation
initializeSchema runs in the HiveDO constructor, so it executes on every cold start / wake from hibernation — a rate set by client reconnects, not by the owner. Two of its steps are reconciliation (register kernel patterns in _objects; drop-and-recreate every audit trigger so the captured column list tracks the schema) and both ran unconditionally. Measured: 185 rows written per no-op wake. Cloudflare's durableObjectsPeriodicGroups for 2026-08-13 attributes 87,372 of 95,720 account rows_written to HiveDO (91%) on a day with six agent writes, and the idle overnight hours are near-exact multiples of 185 (191 = one boot, 573 = three), rising to ~9.5k/hr — one cold start every ~70s. Gate both steps on a bootFingerprint stamped in _meta, derived from the kernel declarations plus the live per-pattern column signature. Fail closed: a throw, a missing stamp, or any difference reconciles. Stored whole rather than hashed — a collision would resurrect the stale-trigger bug the always-rebuild deliberately fixed, and exact compare costs one row write per real schema change. evolution.ts still calls ensureAuditTriggers directly, so a live schema change never waits on a boot. Also add WHERE ... IS NOT excluded.... to the _objects upsert: SQLite rewrites the row on a no-op DO UPDATE, which billed a write per kernel pattern per wake independently of the gate. The oracle is behavioural, not structural: boot-idempotence.test.ts counts actual rowsWritten through a proxied SqlStorage and asserts a second consecutive boot writes zero, so any future unconditional write fails the build regardless of which line introduces it. One settling boot is allowed — reconciliation and the GC sweeps exist to do work; the pathology was paying that cost every time, forever. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rf5RTB7iigiMpoArYzdUKN
The write fix (56c056a) left the read ceiling untouched. Cloudflare's hourly figures show reads tracking writes at a constant ~52.5:1 across a 50x range of activity — three significant figures — which identifies both as the same cold-start population rather than request traffic: ~9,700 rows read per HiveDO wake, 4.59M/day against the 5M/day free tier. Most of it was the same audit-trigger rebuild already gated in 56c056a (DROP TRIGGER scans sqlite_schema, 116 rows a statement, six per pattern): a settle boot measures 7,754 reads, a quiescent one 224. The separate offender was the pre-born-hashing token scrub. "Idempotent, a near-no-op after the first cold start" was true of its WRITES and false of its READS: six full scans of _mutation_log, 6,000 rows, zero writes, on every wake forever, to re-discover there was nothing to do. That is the general trap — a migration whose am-I-done question is a table scan cannot answer it per boot. runOnce gives such work a completion stamp in _completed_migrations, stamped only on success so a part-way failure retries rather than being silently marked done. applyTokenHashScrub moves from hive.ts into schema.ts so the whole boot sequence is initializeSchema + applyTokenHashScrub and the oracle can measure BOTH halves — a test covering only the first would have passed while the scrub burned the read budget. It also puts the migration with the other migrations and keeps HiveDO a wiring shell. The read oracle asserts a property, not a threshold: a quiescent boot's read count must be IDENTICAL at two data volumes, so it needs no magic constant and catches any future boot-time scan. Verified by negative control — removing the gate failed it by name ("boot reads grew with data volume (619 -> 8219)"), then restored to green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rf5RTB7iigiMpoArYzdUKN
The cherry-picked commits carried artifacts generated by v0.31 (the fork runs a newer harness); regenerated here so the freshness gate compares like with like. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rf5RTB7iigiMpoArYzdUKN
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
initializeSchemaruns in the HiveDO constructor — on every cold start / wake from hibernation, not once per deploy. Two of its steps are reconciliation (registering the kernel patterns in_objects; drop-and-recreating every audit trigger so the captured column list tracks the schema), and both ran unconditionally.Measured on a live instance: 185 rows written and ~7.7k rows read per no-op wake. The wake rate is set by client reconnects (each MCP handshake, each
/wsconnect), so the cost has no relationship to actual use. On a free-tier account this consumed 90% of the 100k/day Durable Objectsrows_writtenallowance — and 92% of the 5M/dayrows_readallowance — on a day with six agent writes, then tipped over the cap entirely and took the instance down (every hive-backed route 500s once DO writes are refused). Cloudflare's hourly analytics made the attribution unambiguous: idle-hour write totals are near-exact multiples of 185 (191 = one boot, 573 = three), and reads track writes at a constant ~52.5:1 across a 50× activity range — the same cold-start population, not traffic.The fix
Two gates, same shape as the codebase's other self-enforcing declarations:
bootFingerprint(stamped in_meta): derived from the kernel declarations plus the live per-pattern column signature — never hand-maintained. The_objectsregistration and the audit-trigger rebuild run only when it moves. Fail closed: a throw, a missing stamp, or any difference reconciles. Stored whole rather than hashed — a collision would silently resurrect the stale-trigger bug the always-rebuild deliberately fixed (a trigger built before a column existed omits it from_mutation_logforever), and an exact compare costs one row write per real schema change.evolution.tsstill callsensureAuditTriggersdirectly, so a live schema change never waits on a boot. Also: the_objectsupsert gainsWHERE … IS NOT excluded.…— SQLite rewrites the row on a no-opDO UPDATE, which billed a write per kernel pattern per wake independently of the gate.runOnce(completion stamps in_completed_migrations): the pre-born-hashing token scrub was "idempotent, a near-no-op after the first cold start" — true of its writes, false of its reads: six full scans of_mutation_log(6,000 rows) on every wake forever, to re-discover there was nothing to do. A migration whose am-I-done? question is a table scan can't answer it per boot. Stamped only on success, so a part-way failure retries rather than being silently marked done. The scrub moves fromhive.tsintoschema.tsso the whole boot sequence isinitializeSchema+applyTokenHashScruband the oracle can measure both halves.The oracle
boot-idempotence.test.tscounts actualrowsWritten/rowsReadthrough a proxiedSqlStorage:Verified by negative control: removing the
runOncegate fails it by name (boot reads grew with data volume (619 → 8219)).One settling boot is allowed — reconciliation and the GC sweeps exist to do work; the pathology was paying that cost every time, forever.
The final commit regenerates the coherence artifacts with this repo's pinned v0.27 harness (my fork runs v0.31; the cherry-picked commits carried v0.31-generated artifacts).
Observed effect: boots go from 185 writes / ~7.7k reads to 0 written / ~224 read.
🤖 Generated with Claude Code
https://claude.ai/code/session_01Rf5RTB7iigiMpoArYzdUKN