fix: respawn panicked solver workers instead of losing them

[zizou0x]

Problem

Prod incident 2026-07-23. A quote request caused a divide-by-zero panic in pool math. Each occurrence killed
one solver worker thread. The panic was logged only at shutdown join(), about 100 minutes later. After all
workers on both ethereum pods died, every quote returned no_route_found for about 1.5 hours, until the pods
were recreated.

Root cause

spawn_workers_generic ran SolverWorker::run directly in the thread body. A panic unwound the thread and
ended it. The pool kept the JoinHandle, so it reported the full worker count and stayed silent. Nothing
logged the loss until shutdown.

Fix

Each worker thread now runs worker sessions in a loop, in WorkerContext::run_sessions. A session is wrapped
in catch_unwind. spawn_workers_generic only spawns threads.

• A panic ends the session, logs at ERROR (pool, algorithm, worker_id, message), increments
  worker_pool_worker_panics_total{pool}, and respawns the worker after a backoff.
• Backoff starts at 100 ms and doubles to a 2 s cap. The worker gives up on the 10th consecutive failure. A
  session that lives 600 s or more resets the budget.
• On give-up the process logs and exits 1, so the orchestrator restarts it. Workers are homogeneous, so a
  budget-exhausting failure is almost certainly shared. Exactly one thread runs the exit.
• A fresh worker seeds its allow_stale readiness from the shared derived-data store, so a respawned worker
  does not fail tasks NotReady until the next broadcast event.
• Session shutdown receivers come from resubscribe(), so workers exit when the pool drops its sender. A
  pre-check on the long-lived receiver catches a shutdown sent while no session was listening.

Complements #347, which stops that specific panic escaping the simulation call.

Notes for reviewers

Read supervisor.rs first, then the exit policy in pool.rs — that is the one product decision here. Read
the last commit separately: it is a pure test refactor that folds three mock algorithms into one stub, and it
accounts for most of the diff size.

Two accepted trade-offs:

• Graceful shutdown can take up to 2 s longer. A worker sleeping in backoff cannot observe the signal.
• The 600 s stability window resets the budget rather than decaying it, so failures never expire on their
  own. A worker that panics about every 9 minutes still exits the process after about 90 minutes. Deliberate:
  the give-up path targets deterministic panics, which burn the budget in about 11 seconds.

worker_pool_worker_panics_total under-reports the give-up case, because ~11 s of crash loop rarely spans a
scrape. Alert on restart count and the give-up log line.

Follow-ups

• Per-task catch_unwind inside SolverWorker::run, so a panicking task answers its requester instead of
  dropping the oneshot.
• Gate the task select arm on readiness.
• Share one panic-payload downcast helper between sim_guard.rs and supervisor.rs.
• Give the pool a health signal: num_workers() reports the static count and would over-report if a worker
  ever gives up without exiting the process.

🤖 Generated with Claude Code

[brunoguerios]

The respawn direction is good and the shutdown buffering is race-free. Three things to address before merge:

1. Worker threads now hold a shutdown sender clone, so dropping the pool without an explicit shutdown no longer stops the workers. Use resubscribe instead.
2. The retry loop respawns forever. Add exponential backoff with a bounded attempt budget that resets on stable sessions, and kill the process when any worker gives up (workers are homogeneous, so a budget-exhausting failure is almost certainly shared; exiting at the first give-up avoids a permanently degraded pool).
3. A respawned worker starts not-ready behind a green health check. Seed its readiness tracker from the shared derived data.

Inline comments carry the details, plus test and doc fixes. Note the branch now conflicts with main: spawn_workers_generic gained liquidity_scope, exclude_protocols and fallback_fee_tiers, which must move into the session loop on rebase.

15 comments — 0 Blocker · 4 Important · 3 Suggestion · 6 Nit · 2 Context.

Important

• IMPL #1 Worker threads hold a shutdown sender clone, so the shutdown channel can never close — registry.rs:170
• TESTS #1 Poison-task result is discarded, so the test can pass without a panic — registry.rs:512
• TESTS #2 No test for the buffered-shutdown path (shutdown sent mid-respawn) — registry.rs:190
• DOCS #1 spawn_workers_generic doc contradicts the new respawn behavior — registry.rs:139

Suggestions

• DESIGN #2 Session/supervision loop hidden in a 75-line inline closure — registry.rs:178
• DESIGN #3 Fixed 100ms backoff allows a permanent 10 Hz error-log/rebuild loop — registry.rs:42
• DOCS #2 "(or poisoned)" invents a tokio semantics that does not exist — registry.rs:184

Nits

• DESIGN #4 Third bespoke mock Algorithm impl in tests — registry.rs:449
• TESTS #4 POISON_AMOUNT as u128 cast — registry.rs:514
• DOCS #3 Loop-top comment narrates the fixed bug and is verbose — registry.rs:179
• DOCS #4 session binding holds the session's result, not the session — registry.rs:198
• DOCS #6 Test doc says the panic "does not kill the worker" — registry.rs:449
• DOCS #7 Non-parallel names for the two test tasks — registry.rs:513

Context

• BIZ #2 Respawned worker answers NotReady (after waiting the full algorithm timeout) until the next derived-data event — registry.rs:198
• BIZ #3 Deterministic startup panics now retry forever instead of dying once — registry.rs:183

---

[DOCS #1 · Important]

Description: The doc says "The factory closure is called once per worker to create the algorithm instance" (line 146), but the loop now calls factory once per session — again after every panic respawn (registry.rs:205). Factory implementors must now tolerate repeated calls; the doc states the opposite contract, and the bullet list omits the panic-containment behavior that is now the function's most important guarantee.

Suggestion: Update the doc: state that each worker thread runs sessions in a loop, a panic respawns the worker after a backoff, and the factory is called at every (re)start. Remove or correct "called once per worker".