[Bug]: worker last_seen frozen for ~12 active workers — heartbeat-ingest appears to stop while workers stay functional

[tjluyao]

[Bug]: worker last_seen frozen for ~12 active workers — heartbeat-ingest appears to stop while workers stay functional

Component: Host (orchestrator)

Describe the bug

On the live kv.run:8000/flowmesh host (cluster lumid0, namespace lumid), every worker except two long-stale rows reports an last_seen clustered tightly between 2026-05-27T18:18:31 and 2026-05-27T18:19:01 UTC — a ~30-second window 12 workers fell into and none have left. As of 2026-05-28T10:52Z that's a 16.6h freeze across 12 disparate workers on 8 different nodes (incl. CPU + RTX 5080 + RTX 6000 Ada + GB10 + RTX PRO 4000 boxes). The workers themselves are clearly still functional — three of them (wkr-24, wkr-21, wkr-12) report status: BUSY against the same stale last_seen, so they're actively executing tasks; only the host's heartbeat-ingest side seems to have stopped writing through that point.

Expected: last_seen should advance per worker on each heartbeat regardless of execution state.
Actual: all workers stalled at near-identical timestamps in a ~30s window, despite continuing to pick up jobs.

Reproduction (read-only probe):

curl -sS -H "Authorization: Bearer $PAT" \
  https://kv.run:8000/flowmesh/api/v1/workers \
  | jq '.[] | {id, node_id, status, last_seen}' | head -40

Output (representative subset, taken at 2026-05-28T10:52Z):

{"id":"wkr-15","node_id":"nde-10","status":"IDLE","last_seen":"2026-05-27T18:18:54.…Z"}
{"id":"wkr-10","node_id":"nde-13","status":"IDLE","last_seen":"2026-05-27T18:18:34.…Z"}
{"id":"wkr-16","node_id":"nde-14","status":"IDLE","last_seen":"2026-05-27T18:18:35.…Z"}
{"id":"wkr-24","node_id":"nde-20","status":"BUSY","last_seen":"2026-05-27T18:18:34.…Z"}
{"id":"wkr-21","node_id":"nde-6", "status":"BUSY","last_seen":"2026-05-27T18:18:37.…Z"}
{"id":"wkr-12","node_id":"nde-7", "status":"BUSY","last_seen":"2026-05-27T18:18:58.…Z"}
... (12 rows total, all between 18:18:31 and 18:19:01)
{"id":"wkr-23","node_id":"nde-18","status":"IDLE","last_seen":"2026-05-26T00:21:19.…Z"}  // 58h, predates the freeze
{"id":"wkr-2", "node_id":"nde-2", "status":"IDLE","last_seen":"2026-05-19T08:21:11.…Z"}  // genuine zombie (nde-2 offline)
{"id":"wkr-3", "node_id":"nde-2", "status":"IDLE","last_seen":"2026-05-19T08:21:11.…Z"}  // genuine zombie

That tight clustering across 8 nodes + the BUSY-with-stale-heartbeat combination point to a host-side ingest issue (Redis stream consumer / worker-state writer stalled, channel desync, etc.) rather than 12 simultaneous worker failures.

Impact:

• Registry can no longer distinguish live workers from dead ones (every monitoring threshold built on last_seen flags healthy workers as stale).
• Pre-existing zombies (wkr-2, wkr-3 on offline node nde-2/luyao1) become indistinguishable from the rest of the fleet.
• Downstream dashboards / alerting (e.g. lum.id cluster registry) lose ground truth.

Suggested investigation:

1. Grep the host logs around 2026-05-27T18:18:31Z ± 60s — look for an exception, a Redis reconnect, or a deploy/restart.
2. Check whether the worker-heartbeat consumer group is stuck on a pending entry (Redis XPENDING).
3. Confirm whether last_seen is updated only on heartbeat ingest, or also on result-submission / task-claim paths (status: BUSY updates without last_seen updates suggests the two paths diverged).

Environment

• FlowMesh version: deployed image as of 2026-05-13T18:06:47Z (worker started_at); pre-v0.1.2 redeploy.
• Cluster: lumid0, namespace lumid, multi-node (8 nodes, 15 workers — 12 GPU + 3 CPU).
• Observation captured at: 2026-05-28T10:52Z.

Additional context

This was surfaced while validating the upcoming lumid-flowmesh-plugin v0.2.0 rollout. The plugin work is unrelated; the heartbeat-ingest behavior reproduces against the current v0.1.1 stack. A GET /api/v1/admin/workers/prune (or equivalent) endpoint that drops workers with last_seen older than a configurable threshold would also help — currently the host exposes only GET on /api/v1/workers/{id} and /api/v1/nodes/{id} (confirmed via OPTIONS), so stale rows can't be cleaned out via the API.

[tjluyao]

Self-correction after digging into the source. Two write paths I conflated:

• update_worker_status writes {status, last_seen} into the worker hash. Called from host-side task-lifecycle (BUSY/IDLE).
• update_worker_hb writes a separate Redis key (worker_hb_key) with TTL. Called from monitoring.py on HEARTBEAT events (default ttl_sec = 120).

is_worker_stale checks the TTL on worker_hb_key, not the hash. The watchdog (WORKER_DEATH_CHECK_INTERVAL=30, WORKER_DEATH_GRACE_SEC=60, enabled by default) would have noticed staleness within ~90s and emitted synthetic TASK_FAILED events. It just doesn't clean up the hash row — that's why the API still returns frozen last_seen + last-known status.

status: BUSY on these workers therefore means "BUSY at the moment heartbeats stopped", not "actively executing now". That removes the smoking gun.

The corrected story is simpler and probably not a FlowMesh code bug: ~12 workers across 8 nodes stopped publishing heartbeats at the same ~30s window at 2026-05-27T18:18Z. Watchdog handled it correctly (synthetic failures + dispatch refusal — confirmed by post-freeze tasks all being CANCELLED with worker=None). The hash just isn't TTL'd, so the rows look alive forever.

That leaves a real cluster-side event at 18:18Z to find — almost certainly:

• The Redis instance the workers publish to (not the host's; the broker side) had a blip.
• Or a network partition between the cluster network and that Redis.
• Or a coordinated maintenance action that nobody logged.

What this issue could still actionably ask for (independent of the root-cause investigation):

1. Don't leave stale rows looking alive. When the watchdog marks a worker dead, also set status = LOST (or similar terminal-state enum) in the hash, or emit an event that the host's own monitor consumes to update the hash. Today the watchdog's dead-mark is purely in-memory; restart loses it.
2. A GET /api/v1/workers?stale=false query param (or making stale an actual field in the response) so consumers can filter without comparing timestamps client-side.
3. An admin DELETE /api/v1/workers/{id} / POST /api/v1/admin/workers/prune endpoint so operators can drop rows manually when a node is decommissioned (currently the resource is GET-only — confirmed via OPTIONS).

Original "host-side heartbeat-ingest bug" framing was wrong — leaving this open as a UX/observability ask only, with apology for the noise. Happy to retitle / close + re-open as a smaller observability issue if maintainers prefer.

[tjluyao]

Third-time-lucky correction — apologies for the noise; here's the real story, with measurements.

The "freeze" was an illusion. Worker clocks are ~16h 50m behind UTC. Heartbeats arrive normally; the host correctly writes the worker-published event.ts; that timestamp is just wrong.

Evidence

1. The cumulative counter worker_heartbeats is advancing live: sampled twice 73s apart, +24 → +40 over ~80s. Workers are publishing.
2. Sampled /api/v1/workers 90s apart — per-worker last_seen advanced by +60s for 10 of 15 workers (heartbeat interval), +30s for 3 (just-after-heartbeat capture). Three workers — wkr-2, wkr-3 (nde-2 / luyao1), and wkr-23 (nde-18) — didn't move at all over 90s and are real zombies.
3. Watchdog (is_worker_stale) checks the TTL on worker_hb_key, which is wall-clock-relative — not derived from event.ts. So a 17h-stale-looking last_seen doesn't cause the watchdog to fire, because the actual heartbeat-TTL keeps getting renewed. Dispatch and scheduling are healthy on real workers; the post-"freeze" CANCELLEDs I cited earlier are about the genuine zombies (3 of them) tying up some slot or routing decision, not about the apparent 12-worker freeze.

What's actually wrong

• Worker-side clock drift / mis-set TZ on the cluster nodes. Most likely cause: their container images / systemd-timesyncd / TZ env are misconfigured and the system clock is ~17h behind UTC. The "tight 18:18–18:19 clustering" I called a smoking gun was just heartbeat-interval clustering after a 17h offset.
• luyao1 (nde-2) is genuinely off — wkr-2/wkr-3 last advanced 2026-05-19. Real zombie, operator-side fix.
• nde-18 — wkr-23 last advanced 2026-05-26T00:21. Either real zombie or its clock drifted to a different past offset. Worth checking.

Reduced-scope asks (still useful, independent of the clock story)

1. Validate or normalize event.ts on ingest. If the worker's published timestamp is more than N minutes off the host clock, log a warning + store the host's recv_ts alongside (or in place of) the worker's event.ts for the last_seen field. As-is, a worker with a wrong TZ produces a last_seen that misleads every operator + dashboard.
2. Surface stale: true based on the watchdog's actual judgment, not derived from last_seen vs now. The hash already carries a stale field (WorkerInfo); make sure the API serializes that authoritatively so consumers don't have to recompute from last_seen (which is exactly the field that's wrong here).
3. Watchdog should mark the hash too. When _mark_dead fires, also set status = LOST in the hash row, so a maintainer scanning /api/v1/workers sees the watchdog's verdict instead of last-known status.
4. Admin prune endpoint still useful — the 3 real zombies have no API path to clean up.

Net: not a FlowMesh code bug beyond #1–#4 (UX + ingest robustness). Sorry for the wild goose chase. Going to retitle the issue if you'd prefer something narrower, or close + reopen as "validate event.ts on ingest" — let me know your preference.

— investigation log on our side: probed /system/metrics counter advance (60 MB payload, slow but counter ticks), /workers per-id last_seen movement over 90s, update_worker_hb source (host writes worker-published ts verbatim, TTL is wall-clock-relative).

[kaiitunnz]

@tjluyao This is because of the incorrect deployment and teardown procedures, leaving stale worker records in the root node's Redis. It can be resolved by completely restarting the entire cluster, including all the root and worker nodes.

Nonetheless, we will improve the robustness of worker staleness checks and pruning in a future version.