gw#601: generic worker-activity progress — mint/warmup report liveness, typed activity_failed

[PaulFidika]

Companion: tensorhub th#929 (hub side: generic stall enforcement — PR cozy-creator/tensorhub#504). Wire names are byte-identical across repos; the canonical proto lives in tensorhub and is mirrored here (proto/worker_scheduler.proto, pb regenerated).

Paul's rulings (2026-07-20): NOT mint-specific — one generic worker-activity contract, self-mint is just one activity kind; detect STALL, not slowness — the watchdog proves liveness, the hub owns termination; a silent death is a bug by contract.

Wire (additive)

• WorkerMessage.activity_update = 9: {kind, phase, step/total, monotonic seq, state, error, detail}. Kinds at launch: self_mint_compile, warmup. Model downloads keep their richer ModelEvent byte shape (hub folds them into the same store).

Worker

• gen_worker/activity.py: process-monotonic seq; begin/running bracket (COMPLETED on clean exit, FAILED carrying the exception on raise — the typed activity_failed); watchdog bracket for long wire-silent calls (inductor etc.): background thread heartbeats ~60s ONLY while an evidence counter (default process-CPU seconds) advances — an induced hang stops the beat within one interval. Without a bound transport sink, reports land on the logger (the cozy-local progress UI).
• Executor: setup+warmup is one activity (self_mint_compile when compile= is declared, else warmup), begun around _setup_locked under the watchdog; phases load → inductor_compile (pending self-mint) / warmup_forward n/N (per synthesized warmup job) → seal_publish (finalize_self_mint). Sink bound to the transport loop at ensure_setup.
• cozy-local (gw#555): local_cells._mint runs under the same activity+watchdog bracket — same events, local sink.

Tests (tests/test_activity_gw601.py, real executor code path — no executor mocks)

• scripted setup+warmup emits monotonic phases (load → warmup_forward 1/2, 2/2 → COMPLETED; strictly increasing seq) through the production ensure_setup.
• induced setup crash → terminal FAILED envelope carrying RuntimeError: induced setup crash.
• watchdog: heartbeats while evidence advances; an induced hang stops the beat within one interval; seq stays monotonic.
• running bracket reports FAILED with the exception and clears the current activity.
• Full suite: 992 passed / 60 skipped locally; the 61 failing/error tests are pre-existing heavy-dep gaps in this venv (identical on clean master, verified by stash-rerun) — CI has the full dev deps.

🤖 Generated with Claude Code