feat(memory-sync): escalate when the queue outlives its threshold

packages/agent-memory-sync/README.md

@@ -131,14 +131,23 @@ Environment=AGENT_MEMORY_SYNC_BRANCH=main
 ExecStart=/usr/local/bin/agent-memory-sync watch --verbose
 Restart=on-failure
 RestartSec=5s
-StartLimitIntervalSec=300
-StartLimitBurst=10
+StartLimitIntervalSec=1800
+StartLimitBurst=30
 
 [Install]
 WantedBy=multi-user.target
 ```
 
-The `StartLimitIntervalSec` / `StartLimitBurst` pair caps systemd's restart loop for the failures that still exit non-zero — a config/data error raised before the remote working copy is prepared (e.g. a required `syncPaths` entry missing), or any other git-level failure while preparing/committing that working copy (a full disk, a corrupted git config, a broken commit hook, ...) — so a persistently broken cause does not crashloop forever; a remote that is merely unreachable or rejecting the push (see below) no longer exits at all, so it never spends this budget. Inspect `journalctl -u agent-memory-sync-watch.service` for the `snapshot push failed: ...` line `watch` writes to stderr before exiting on one of those failures.
+The `StartLimitIntervalSec` / `StartLimitBurst` pair caps systemd's restart loop for the failures that still exit non-zero — a config/data error raised before the remote working copy is prepared (e.g. a required `syncPaths` entry missing), or any other git-level failure while preparing/committing that working copy (a full disk, a corrupted git config, a broken commit hook, ...) — so a persistently broken cause does not crashloop forever; a remote that is merely unreachable or rejecting the push (see below) no longer exits at all, so it never spends this budget. The one exception is [queue escalation](#queue-escalation-a-permanently-broken-remote-does-not-queue-forever): once the queue has been failing to drain past `queueEscalationThresholdMs` (default 24h), a tick DOES exit non-zero again — but only on a real local edit (`watch` is edge-triggered), so it does not spend this budget any faster than this machine's memory actually changes while the remote stays broken. Inspect `journalctl -u agent-memory-sync-watch.service` for the `snapshot push failed: ...` line `watch` writes to stderr before exiting on one of those failures.
+
+Honest arithmetic, measured: one crash-restart cycle (a failed start plus `RestartSec`) is ~11s. Under the original `StartLimitBurst=10` / `StartLimitIntervalSec=300` pairing shown in earlier revisions of this doc, 10 crashes exhausted the budget in ~110s — well inside a single ordinary "edit the config, restart, still broken, edit again" debugging session. Once the burst is exhausted, systemd does not just pause the restart loop, it marks the unit `failed` and **stops trying entirely**, even after the underlying cause is fixed, until the failure counter is explicitly cleared:
+
+```bash
+systemctl reset-failed agent-memory-sync-watch.service
+systemctl restart agent-memory-sync-watch.service   # reset-failed only clears the counter, it does not start the unit
+```
+
+The sample unit above raises the pairing to `StartLimitIntervalSec=1800` / `StartLimitBurst=30` (~30 crashes × ~11s ≈ 330s, under 6 minutes of continuous crash-looping) so ordinary iterative config editing has realistic headroom before landing in `failed`, while a genuinely broken cause still gets capped well short of looping forever. macOS's `ThrottleInterval` (see `docs/launchd/com.agent-memory-sync.watch.plist.template`) is not a direct analogue: it only enforces a minimum gap between respawns and has no burst counter or give-up state at all, so a broken `watch` LaunchAgent keeps retrying indefinitely instead of ever reaching a terminal `failed` state that needs a manual reset.
 
 macOS equivalent (LaunchAgent instead of systemd): see
 [`docs/launchd/com.agent-memory-sync.watch.plist.template`](docs/launchd/com.agent-memory-sync.watch.plist.template)
@@ -173,6 +182,35 @@ fallback-machine setup, not an optional extra — see
 [docs/machine-setup.md](docs/machine-setup.md) for the launchd/systemd
 companion jobs.
 
+##### Queue escalation: a permanently broken remote does not queue forever
+
+The queue-instead-of-crash handling above is deliberately silent for a
+remote that is merely *offline* — a laptop closed overnight, on a flight, or
+disconnected for a weekend. But a remote that is *correctly* classified
+`RemoteUnavailableError` can still be **permanently** wrong (a bad
+`remoteUrl`, a renamed repository path, a host that accepts an SSH/TCP
+connection but cannot serve the repository) — without a second signal, that
+looks identical to a laptop on a plane and would queue cleanly, exit `0`,
+forever, never syncing again.
+
+Every enqueue therefore checks the age of the OLDEST currently-queued
+snapshot (`stateDir/queue/<id>/manifest.json`'s `createdAt`, already written
+on every enqueue — no new state) against `queueEscalationThresholdMs`
+(config file / `AGENT_MEMORY_SYNC_QUEUE_ESCALATION_THRESHOLD_MS`, default
+24h). Below the threshold, behavior is unchanged: silent, exit `0`, every
+tick. Once the oldest queued snapshot is older than the threshold — meaning
+the remote has been *continuously* unreachable for that long, not just on
+this one tick, since a successful push clears the whole queue at once — the
+tick throws instead: a clear message on stderr and a non-zero exit (`6`),
+the same supervisor-restart surface a non-network failure already uses. The
+snapshot itself is never lost; it stays queued and is replayed automatically
+once the remote is reachable again. 24h is sized against this package's own
+committed periodic-sync tick interval (900s / 15min — see
+[docs/machine-setup.md](docs/machine-setup.md) and the launchd/systemd
+templates) — 96 missed ticks, comfortably longer than an overnight or
+weekend offline window, still bounding how long a genuinely broken remote
+can hide to about a day.
+
 ##### Push authentication
 
 `watch` (and `run --mode push`) invoke the system `git` binary; authentication is whatever `git` itself is configured to use, e.g. an SSH key, an OS credential helper, or a `https://x-access-token:$TOKEN@github.com/...` URL.
@@ -260,6 +298,7 @@ The `--config` flag overrides the default path.
   "outputFormat": "text",
   "verbose": false,
   "reachabilityTimeoutMs": 4000,
+  "queueEscalationThresholdMs": 86400000,
   "syncPaths": [
     { "source": "MEMORY.md", "destination": "MEMORY.md", "kind": "file" },
     { "source": "logs", "destination": "logs", "kind": "directory" }
@@ -309,6 +348,22 @@ Priority order (highest to lowest): CLI flags > environment variables > config f
   separate on/off switch, this is the supported way to opt out
 - failed pushes (including ones skipped by the reachability precheck) are queued locally in
   `stateDir/queue` and replayed on the next successful push
+- if the OLDEST queued snapshot is older than `queueEscalationThresholdMs`
+  (config file / `AGENT_MEMORY_SYNC_QUEUE_ESCALATION_THRESHOLD_MS`, default 24h) — i.e. the
+  remote has been *continuously* unreachable for that long, not just on this one tick — the
+  tick throws instead of returning a clean "queued" result: a message on stderr and exit code
+  `6`, so a permanently misconfigured remote (wrong `remoteUrl`, a renamed repository path, a
+  host that accepts a connection but cannot serve the repository) does not queue silently
+  forever. Below the threshold nothing changes: silent, exit `0`, every tick — a merely offline
+  machine is unaffected. See [watch's "Queue escalation" section](#queue-escalation-a-permanently-broken-remote-does-not-queue-forever)
+  for the full rationale. Set `queueEscalationThresholdMs` to `null` (config file, or
+  `config set queueEscalationThresholdMs null`) to disable this check entirely — mirrors
+  `reachabilityCheckCommand`'s null-is-a-real-value convention above. The queue then keeps
+  queuing silently, exit `0`, forever, regardless of age; a computed age past a 30x-threshold
+  sanity ceiling is also never escalated even with a finite threshold configured, since an age
+  that implausible more likely reflects this machine's clock having been wrong when the
+  snapshot was queued than a genuinely stuck remote — a diagnostic note is emitted on that
+  otherwise-silent "queued" outcome instead
 - append-only concurrent edits are merged automatically; other conflicts default to inline conflict markers
 - `--dry-run` previews the result without changing local files or the remote repository
 

packages/agent-memory-sync/docs/architecture.md

@@ -114,6 +114,7 @@ Commands signal failure by raising/returning an error - they never call `os.exit
 | `3` | Configuration error (bad config file, missing required setting) |
 | `4` | Runtime error (external service unavailable, permission denied) |
 | `5` | Not found (resource the command expected does not exist) |
+| `6` | Queue escalation (the local queue has not drained within queueEscalationThresholdMs) |
 
 Error messages follow the pattern: `error: <what went wrong>. <how to fix it>.`
 

packages/agent-memory-sync/docs/launchd/com.agent-memory-sync.sync.plist.template

@@ -53,6 +53,19 @@
     no-op — `run --mode sync`'s own reachability precheck (default 4s
     bound; see src/memory-sync/reachability.ts) means this never hangs or
     piles up overlapping ssh attempts.
+
+    Queue escalation: this template's 900s StartInterval is also the tick
+    cadence the queue-escalation default threshold is sized against — see
+    src/memory-sync/state-store.ts's DEFAULT_QUEUE_ESCALATION_THRESHOLD_MS
+    (24h = 96 of these ticks) and docs/machine-setup.md's "Queue escalation"
+    bullet. If the oldest snapshot in stateDir/queue is older than that
+    threshold — i.e. the remote has been unreachable for the whole 24h, not
+    just this one tick — a `run --mode sync` tick exits non-zero (6) instead
+    of its usual clean 0, with a clear stderr message; that failure is
+    visible via `launchctl print`'s last-exit-status and this job's
+    StandardErrorPath log, without launchd respawning anything (this
+    template has no KeepAlive — it is a plain periodic tick, so a failed run
+    just tries again at the next StartInterval).
 -->
 <plist version="1.0">
 <dict>

packages/agent-memory-sync/docs/launchd/com.agent-memory-sync.watch.plist.template

@@ -81,6 +81,25 @@
     so pair this job with the periodic-sync companion,
     com.agent-memory-sync.sync.plist.template — see its header and
     docs/machine-setup.md for why that companion is required, not optional.
+
+    Queue escalation, and why it DOES respawn (unlike a merely-offline
+    tick): the queue-instead-of-crash handling above only stays silent
+    while the queue keeps draining within a bounded time. If the OLDEST
+    snapshot in stateDir/queue is older than queueEscalationThresholdMs
+    (default 24h — src/memory-sync/state-store.ts's
+    DEFAULT_QUEUE_ESCALATION_THRESHOLD_MS, sized against this template's own
+    companion's 900s StartInterval), the remote has been unreachable for the
+    WHOLE 24h, not just this one tick — a permanently broken remote (bad
+    remoteUrl, a renamed repo path, a host that accepts a connection but
+    can't serve the repository) rather than a laptop offline overnight. That
+    tick then exits non-zero (6) with a clear stderr message instead of the
+    usual clean 0, which — per KeepAlive/SuccessfulExit=false above — DOES
+    trigger a respawn. This is not a tight crash loop: `watch` only ticks on
+    a real local edit, so it only re-throws (and respawns again) the next
+    time this machine's memory actually changes while the remote is still
+    broken, each time writing a fresh, visible line to
+    watch.<profile>.err.log — see docs/machine-setup.md's "Queue escalation"
+    bullet and README.md's "Queue escalation" section under `watch`.
 -->
 <plist version="1.0">
 <dict>

packages/agent-memory-sync/docs/machine-setup.md

@@ -56,6 +56,35 @@ This document wires together the pieces already documented individually
   git-level failure while that working copy is being prepared or committed
   (a full disk, a corrupted git config, a broken commit hook, ...) still
   exit non-zero.
+- **Queue escalation: silent-forever is not the same as offline.** The
+  queue-instead-of-crash behavior above is deliberately quiet for a machine
+  that is merely offline (closed overnight, on a flight, a weekend without
+  connectivity) — but a remote that is `RemoteUnavailableError` for a
+  *permanent* reason (a typo'd `remoteUrl`, a renamed bare-repo path on the
+  mini, an SSH host that accepts the connection but can no longer serve
+  `git-upload-pack`) would otherwise look identical: queued, exit `0`,
+  every tick, forever, never actually syncing again. Every enqueue now
+  checks the age of the OLDEST currently-queued snapshot
+  (`stateDir/queue/<id>/manifest.json`'s `createdAt`, already written on
+  every enqueue — no new persisted state) against
+  `queueEscalationThresholdMs` (config file /
+  `AGENT_MEMORY_SYNC_QUEUE_ESCALATION_THRESHOLD_MS`, default 24h). Below the
+  threshold, nothing changes. Once the oldest queued snapshot is older than
+  the threshold — meaning the remote has been *continuously* unreachable for
+  that long, not merely on this one tick, since a successful push clears the
+  whole queue at once — the tick throws instead of returning a clean
+  "queued" result: a clear message on stderr and a non-zero exit (`6`), the
+  same supervisor-restart surface a non-network failure already uses (see
+  the previous bullet). The queued snapshot itself is never lost either way;
+  it stays queued and replays automatically once the remote is reachable
+  again. The 24h default is sized against this document's own committed
+  periodic-sync tick interval — 900s / 15min, see
+  `docs/launchd/com.agent-memory-sync.sync.plist.template`'s `StartInterval`
+  (macOS) and the systemd `OnUnitActiveSec=15min` timer in (c) below (Linux)
+  — 96 missed ticks at that cadence, comfortably past an overnight or
+  weekend offline window while still bounding a genuinely broken remote's
+  silence to about a day. See README.md's "Queue escalation" section under
+  `watch` for the full rationale.
 - **`watch` is edge-triggered and does not pull — this is why the periodic
   sync job is required, not optional.** `watch` only commits+pushes when
   *this* machine's local files change; it never reads from the remote. Its
@@ -283,8 +312,19 @@ empty/stale local workspace as if it were authoritative.
    `Persistent=true` catches up a missed tick (e.g. the machine was off)
    shortly after boot instead of waiting a full interval. A tick that fires
    while the mini is unreachable is a fast, clean no-op — same reachability
-   precheck `run` always uses — so a short 15-minute interval is safe; it
-   will not pile up overlapping ssh attempts or spam logs. Install with:
+   precheck `run` always uses — so a short 15-minute interval is safe below
+   the queue escalation threshold (see the "Queue escalation" bullet above):
+   it will not pile up overlapping ssh attempts or spam logs. That stops
+   being true once the OLDEST queued snapshot crosses
+   `queueEscalationThresholdMs` (default 24h, ~96 missed ticks at this
+   interval) — from then on, each 15-minute tick exits non-zero (`6`) with a
+   clear stderr message instead of a silent no-op. Because this is a
+   `Type=oneshot` service fired by a `.timer`, not a `Restart=on-failure`
+   daemon, the timer keeps firing it again every 15 minutes regardless of
+   that failure, so expect one failure line in `journalctl -u
+   agent-memory-sync-sync.service` per tick until the remote is fixed — that
+   repeated visibility is the intended outcome of escalation, not log spam
+   to suppress. Install with:
    `systemctl daemon-reload && systemctl enable --now agent-memory-sync-sync.timer`.
 
 ## d) Restore / rollback

packages/agent-memory-sync/src/commands/run.ts

@@ -3,7 +3,7 @@ const {
   requireRemoteUrl,
   resolveRunConfig
 } = require("../config/loader");
-const { CliError, isCliError } = require("../errors");
+const { CliError, RemoteQueueEscalationError, formatErrorMessage, isCliError } = require("../errors");
 const { performPull } = require("../memory-sync/pull");
 const { performPush } = require("../memory-sync/push");
 const { summarizeOperation } = require("../memory-sync/preview");
@@ -96,13 +96,56 @@ function registerRunCommand(program: import("commander").Command): void {
 
       const runs: Array<Record<string, unknown>> = [];
       let remainingRuns = maxRuns || (runConfig.schedule ? Number.POSITIVE_INFINITY : 1);
+      // Set once a scheduled tick escalates (RemoteQueueEscalationError) so
+      // the whole invocation still exits 6 after the loop below finishes —
+      // see the try/catch inside the loop for why a scheduled run does not
+      // stop ticking the moment that happens.
+      let queueEscalationError: unknown = null;
 
       while (remainingRuns > 0) {
         if (options.dryRun) {
           writeDryRun(`executing ${runConfig.mode} for profile '${runConfig.profile}'`, outputOptions);
         }
 
-        const execution = await executeMode(runConfig, { dryRun: options.dryRun }, outputOptions);
+        let execution: Record<string, unknown>;
+        try {
+          execution = await executeMode(runConfig, { dryRun: options.dryRun }, outputOptions);
+        } catch (error) {
+          // Single run (no --schedule): preserve the pre-fix behavior
+          // exactly — RemoteQueueEscalationError (and everything else)
+          // propagates immediately, uncaught, straight out of this action
+          // handler to main.ts's top-level catch, with writeResult() below
+          // never reached. Immediate exit 6, same as before this fix.
+          //
+          // Scheduled run: an escalating tick must not kill the scheduler
+          // outright. run --schedule IS its own supervisor/replay loop — a
+          // tick that escalated failed to drain the queue on THIS tick, but
+          // the queue is still safely persisted and a later tick, once the
+          // remote recovers, is exactly what replays it. Dying on the first
+          // escalation stranded every remaining tick from ever getting that
+          // chance (measured: exit 6 after tick 1 of 3, zero stdout, no
+          // further ticks ran at all). So: record it, keep ticking, and
+          // still exit 6 once the loop ends — just after every remaining
+          // tick had its shot, and after writeResult() below has run so a
+          // --output json consumer still sees every tick that did complete.
+          if (!runConfig.schedule || !(error instanceof RemoteQueueEscalationError)) {
+            throw error;
+          }
+
+          queueEscalationError = error;
+          execution = {
+            kind: runConfig.mode,
+            status: "escalated",
+            remoteHeadBefore: null,
+            remoteHeadAfter: null,
+            appliedFiles: [],
+            mergedFiles: [],
+            conflictFiles: [],
+            queuedSnapshotId: null,
+            notes: [formatErrorMessage(error)]
+          };
+        }
+
         runs.push(execution);
         remainingRuns -= 1;
 
@@ -128,6 +171,10 @@ function registerRunCommand(program: import("commander").Command): void {
       };
 
       writeResult(payload, runConfig.outputFormat, () => runs.map(summarizeOperation).join("\n"));
+
+      if (queueEscalationError) {
+        throw queueEscalationError;
+      }
     });
 }