Skip to content

feat(memory-sync): escalate when the queue outlives its threshold - #76

Merged
LanNguyenSi merged 2 commits into
masterfrom
task/11424b5e-remote-escalation
Aug 5, 2026
Merged

feat(memory-sync): escalate when the queue outlives its threshold#76
LanNguyenSi merged 2 commits into
masterfrom
task/11424b5e-remote-escalation

Conversation

@LanNguyenSi

Copy link
Copy Markdown
Owner

Task 11424b5e: a permanently wrong remote (bad remoteUrl, renamed repo path, host without git-upload-pack) queued cleanly with exit 0 on every tick, forever — indistinguishable from a laptop offline; the system looked healthy and never synced.

What changed

  • Age-based escalation, zero new persisted state: after every enqueue, the oldest queued snapshot's manifest createdAt is compared against queueEscalationThresholdMs (default 24h = 96 missed ticks at the shipped 15-minute launchd/systemd intervals, cited against the actual plist/unit files). Above threshold the tick throws RemoteQueueEscalationError (exit 6, added to the canonical architecture.md table); below threshold nothing changes (byte-for-byte offline parity measured by the reviewer against master: identical stdout/stderr/exit/queue layout over 3 offline ticks).
  • The error class deliberately does not extend RemoteUnavailableError, so the existing queueing discrimination cannot swallow it; no message heuristics anywhere.
  • Config key wired end-to-end (file, env, config get/set) after the reachabilityTimeoutMs pattern; explicit null disables escalation (mirroring reachabilityCheckCommand's convention). Wiring is pinned by tests that only pass if the override reaches the check (2h backdate: under the default, over the override) — mutation-verified red when the key is unwired.
  • Clock-skew guard: an implausible age (>30x threshold, e.g. after an NTP correction on a machine that enqueued under a wrong-in-the-past clock) skips escalation with a diagnostic note instead of turning a healthy machine crash-loud.
  • Scheduled runs (run --schedule) catch the escalation, continue remaining ticks, report full JSON, and exit 6 at the end; single runs exit 6 immediately (unchanged).
  • Pins from the same review: the deliberate crash-loud fetch boundary in prepareWorkingCopy (stub git failing on fetch; mutation-verified to red exactly that test when fetch is reclassified into the queueing path) and the precheck-bypass path (unsupported-scheme remote failing at lookupRemoteHead still queues), plus escalation via the git-failure catch path.
  • Docs: README queue-escalation section with honest systemd burst arithmetic (crash cycle ~11s; raised StartLimit pairing; systemctl reset-failed recovery; macOS ThrottleInterval divergence), machine-setup's "clean no-op" claim carved out for the post-threshold case, both plist template headers incl. KeepAlive respawn analysis (edge-triggered, no tight loop — verified against chokidar ignoreInitial).

Verification

  • Reviewer re-measured everything first-hand (accept_with_notes, 6 MEDIUM + 5 LOW, zero HIGH/CRITICAL): 3 out-of-tree mutations (fetch-boundary pin discriminating, escalation-call removal reds exactly the two behavioral tests, config-unwiring gap), master-vs-HEAD offline parity, no-local-change tick still escalates (measured exit 6), --schedule death (measured, now fixed), exit-code space (6 free), hand-built CLI scenarios for config/env/invalid/round-trip.
  • All 6 MEDIUMs and 4 LOWs fixed in 70e49bb (9 new tests); orchestrator re-ran the config-unwiring mutation post-fix: the new wiring test goes red under mutation, green restored. Consciously deferred (pre-existing): run.ts inline config-type re-declaration, unbounded queue growth after escalation begins.
  • Suite 157/157 with coverage 96.14/78.13/95.20 vs gates 86/65/86; typecheck clean; the known watch-mirror-delete teardown hang (tracked as c71de504) did not reproduce on the final runs.

Lan Nguyen Si added 2 commits August 5, 2026 20:04
A permanently wrong remote (bad remoteUrl, renamed repo path, host
without git-upload-pack) queued cleanly with exit 0 on every tick,
forever — indistinguishable from a laptop offline. The tick now checks
the oldest queued snapshot's already-persisted manifest createdAt after
every enqueue and throws RemoteQueueEscalationError (exit 6) once it
exceeds queueEscalationThresholdMs (default 24h, i.e. 96 missed ticks
at the shipped 15-minute launchd/systemd intervals). Below threshold
nothing changes: local commit, queued push, exit 0. The error class
deliberately does not extend RemoteUnavailableError so the existing
queueing discrimination cannot swallow it. Config key wired end-to-end
(file, env, config get/set) after the reachabilityTimeoutMs pattern.

Pins from the same review: the deliberate crash-loud fetch boundary in
prepareWorkingCopy (stub git failing on fetch, non-zero exit,
mutation-verified) and the precheck-bypass path (unsupported-scheme
remote failing at lookupRemoteHead still queues cleanly).

Docs: README queue-escalation section, machine-setup bullet, both plist
template headers incl. the KeepAlive respawn analysis.

Task: 11424b5e
Canonical exit-code table gains code 6. Config wiring is now pinned by
tests that only pass if the override actually reaches
checkQueueEscalation (2h backdate: under the 24h default, over the 1h
override), plus an invalid-env exit-3 case. Clock-skew guard: an
implausible age (>30x threshold) skips escalation with a diagnostic
note instead of turning a healthy machine crash-loud after an NTP
correction; wall-clock dependency documented. Scheduled runs catch the
escalation, continue remaining ticks, report full JSON, and exit 6 at
the end (single runs unchanged). Null disables escalation end-to-end,
mirroring reachabilityCheckCommand's convention. README states the
systemd burst arithmetic honestly with reset-failed recovery and a
raised StartLimit pairing; machine-setup's no-op claim carves out the
post-threshold case. Escalation message trimmed to the actionable core,
queue count via direct dir listing, corrupt-manifest one-way bias
documented and pinned (surviving older manifest still escalates), and
the git-failure catch path has its own escalation test.

Task: 11424b5e
@LanNguyenSi LanNguyenSi added review:tests-pass merge-approval gate prerequisite review:checklist-complete merge-approval gate prerequisite review:comments-resolved merge-approval gate prerequisite review:scope-matches-task merge-approval gate prerequisite review:evidence-logged merge-approval gate prerequisite labels Aug 5, 2026
@LanNguyenSi
LanNguyenSi merged commit e25bd09 into master Aug 5, 2026
5 of 7 checks passed
@LanNguyenSi
LanNguyenSi deleted the task/11424b5e-remote-escalation branch August 5, 2026 18:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review:checklist-complete merge-approval gate prerequisite review:comments-resolved merge-approval gate prerequisite review:evidence-logged merge-approval gate prerequisite review:scope-matches-task merge-approval gate prerequisite review:tests-pass merge-approval gate prerequisite

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant