Skip to content

fix(captcha): cancel timed-out worker races - #81

Merged
TriDefender merged 6 commits into
TriDefender:masterfrom
AlexDeveloperUwU:captcha-worker-cancellation
Nov 16, 2026
Merged

TriDefender merged 6 commits into
TriDefender:masterfrom
AlexDeveloperUwU:captcha-worker-cancellation

Conversation

@AlexDeveloperUwU

@AlexDeveloperUwU AlexDeveloperUwU commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Problem

CAPTCHA solves were failing in a cascade: a worker hung and hit the 30000ms worker timeout, the pool retried (four attempts), and parallel solves that had already lost the client-facing race kept running. Orphaned workers piled up, so later solves also failed and requests surfaced upstream 503s.

Fix

  • Thread an AbortSignal through captcha pool → solver → worker dispatch.
  • When the race deadline expires, abort and terminate the parallel worker solves instead of leaving them running.
  • Cancellation works for both warm workers and worker-per-solve workers.
  • Add a hung-worker regression test in captcha-worker-dispatch.test.ts.

Scope

Only src/proxy/captcha-pool.ts, captcha-solver.ts, captcha-worker-dispatch.ts and its test. No config or API changes.

Validation

bun test passes and the Docker image builds from this change (run on my side before opening the PR).

Fixes #82

Propagate an AbortSignal from the CAPTCHA pool through the solver to worker dispatch, and abort/terminate parallel worker solves when the race deadline expires. Covers warm workers and worker-per-solve workers, with a hung-worker regression test.
@AlexDeveloperUwU

Copy link
Copy Markdown
Contributor Author

Btw, after doing this change, I'm not seeing any more captcha failures.

@TriDefender TriDefender left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified on a local checkout of ea92cea: bun x tsc --noEmit clean, all 74 captcha tests green. I also ran ad-hoc probes against real worker fixtures to exercise the abort wiring end to end.

The core direction is right and mostly solid — the retry-ladder stop in solveFresh, not spreading signal into the worker postMessage, no in-process degrade on abort, no noteWarmDeath on abort-kill, and clearing the deadline timer on success are all handled correctly. Worker-per-solve mid-flight abort rejects in ~124ms (vs the 30s worker timeout) and warm mid-flight abort in ~4ms in my probes.

Two should-fix items before merge:

1. [P2] Stale abort listener in dispatchToWarm can kill an unrelated take's in-flight solve

The abort listener is registered {once: true} but is never removed when the dispatch ends through a normal (non-abort) settlement — the ok:false reply path, the 30s timeout path, or the message handler. solveInWorker removes the listener in its settle helper; dispatchToWarm has no equivalent.

Repro (real worker fixtures, verified end to end):

  1. Take A's warm dispatch fails with ok:false (e.g. F008 velocity) → the handle returns to the pool idle, but dispatch #1's listener stays attached to A's signal.
  2. Take B acquires the same idle handle and hangs on its solve.
  3. Take A's race deadline fires → A's stale listener runs killWarm(h) unconditionally → B's in-flight worker is terminated and B rejects with A's deadline reason, while b.signal.aborted is still false (asserted in the probe).

Impact is bounded but real on Android (warm pool of 3; desktop default ZCODE_CAPTCHA_WARM_WORKERS=0 is immune): under exactly the hung-worker storm conditions of #82, handles churn fast and takes overlap, so this is reachable — and the killed solve surfaces in solveFresh as noteMintFailure("captcha take deadline"), polluting the mint-storm counters (8 such failures in 5min with no successes → false storm → telegram IP reset).

One-line guard (verified: eliminates the repro without affecting legitimate mid-flight aborts):

const onAbort = () => {
  // Stale-listener guard: if this dispatch already ended (ok:false reply,
  // timeout), the handle may have been re-acquired by ANOTHER take — a late
  // abort from the original signal must not kill that solve.
  if (h.pending?.id !== id) return;
  killWarm(h, req.signal?.reason instanceof Error ? req.signal.reason.message : "captcha solve aborted");
  dropFromPool(h);
  reject(req.signal?.reason instanceof Error ? req.signal.reason : new Error("captcha solve aborted"));
};

Alternatively mirror solveInWorker's removeEventListener in a settle-style completion path.

2. Regression test does not pin the mid-flight termination

"aborting a solve terminates a hung worker immediately" aborts synchronously right after calling solveViaWorkerOrInProcess, before the async entry resolution resumes — so it only exercises the throwIfAborted pre-check. Evidence: with both addEventListener("abort", ...) registrations deleted, this test still passes in 0.25ms (no worker is ever spawned), while a probe that spawns the worker, waits ~120ms, then aborts hangs without the wiring.

That means a future refactor could silently drop the onAbort wiring and this test stays green — issue #82 would regress without a signal. Suggested shape:

const pending = solveViaWorkerOrInProcess({ ..., signal: controller.signal });
await new Promise((r) => setTimeout(r, 100)); // worker spawned, job dispatched
const t0 = Date.now();
controller.abort(new Error("request deadline"));
await expect(pending).rejects.toThrow("request deadline");
expect(Date.now() - t0).toBeLessThan(5_000); // must not ride the 30s worker timeout

plus a warm-pool variant (ZCODE_CAPTCHA_WARM_WORKERS=1) asserting the killed handle is dropped from the pool and the next solve respawns. Also note the test lives in the "warm worker pool" describe but deletes ZCODE_CAPTCHA_WARM_WORKERS and exercises worker-per-solve — worth relocating or commenting while touching it.

Minor (non-blocking)

  • The in-process fallback (solveTraceless) ignores the signal. That's acceptable — a main-thread solve cannot be terminated and is bounded by its own timeouts — but worth a one-line comment stating cancellation covers the worker faces only, so nobody assumes full coverage.
  • Pre-existing, out of scope: a racer that completes after the deadline has its first token dropped rather than banked (solveRaced's served/resolve path).

Fix #1 (one line) and reshape the test per #2, and this is approve-worthy — the cancellation mechanism itself works exactly as described in the PR body.

…tests

Skip the abort handler in dispatchToWarm when the dispatch already ended, so a late abort cannot kill an unrelated take that re-acquired the handle. Reshape the regression test to abort after the worker is dispatched, add a warm-pool variant, and note that in-process solves are not cancellable.
@AlexDeveloperUwU

Copy link
Copy Markdown
Contributor Author

Thanks for the thorough review, and for the repro. Both items are addressed in the three new commits.

1. Stale abort listener in dispatchToWarm (eaeb5c3)
I used your guard: onAbort now returns early when h.pending?.id !== id. A late abort from a finished dispatch can no longer kill an unrelated take that re-acquired the handle.

2. Regression test (eaeb5c3, 1ecb6b3)

  • The worker-per-solve test now waits 100ms so the worker is spawned and the job dispatched. It then aborts and asserts the rejection reason and that it finishes in under 5s.
  • I added a warm-pool variant with ZCODE_CAPTCHA_WARM_WORKERS=1. It aborts mid-flight on two consecutive solves and asserts the second one respawns, which shows the killed handle was dropped.
  • The hanging fixture never actually hung, because it had no message listener and exited immediately. It now has one, and it also records its spawns so the warm test can count them.
  • I added a one-line comment at the in-process fallback saying cancellation only covers the worker paths. bff1dfa removes the code comments again, to follow the repo's no-comments style.

Validation: I ran the repo's ci.yml (tsc --noEmit + bun test) on my fork against the same branch. It passes at 1ecb6b3 with 1223 pass and 0 fail, including both new abort tests: https://github.com/AlexDeveloperUwU/zcode-api/actions/runs/43888915365

@TriDefender TriDefender left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-verified at 1ecb6b3 on a fresh checkout: tsc --noEmit clean, full bun test 1223 pass / 0 fail, matching your fork CI.

Both items are properly resolved:

1. Stale listener guard — if (h.pending?.id !== id) return; in onAbort is exactly right. I re-ran my original repro end to end against your head (take A settles ok:false → handle re-acquired by take B → A's signal aborts late): take B stays pending and unaffected, then rejects cleanly when the pool is stopped. The guard also passes through legitimate mid-flight aborts unchanged (~4ms rejection on the warm path in my probes).

2. Regression tests now pin the wiring — I deleted both addEventListener("abort", ...) registrations locally and re-ran the abort-test scenario: without the wiring it cannot reject quickly (rides the 30s worker timeout; the test hits bun's 5s timeout), so a future refactor that drops the wiring now fails CI instead of passing in 0.25ms. Good catch on the fixture, too — the original hanging-worker.mjs had no message listener, so the worker's event loop emptied and it exited immediately; with parentPort.on('message', () => {}) it genuinely hangs and the test exercises a real mid-flight termination. The warm variant's spawn counting (delta = 2 across two aborted solves) correctly pins both the mid-flight kill and that the dropped handle forces a respawn.

Two non-blocking notes, no action needed:

  • There's no direct regression test for the stale-listener scenario itself (ok:false settle → re-acquire → late abort from the first signal). The guard is one line and I've verified it live, so I'm not asking for one — just noting it if you ever touch this code path again.
  • The in-process fallback comment from eaeb5c3 ended up removed by bff1dfa. Fine either way.

Nice work — this closes both should-fix items cleanly. Approving.

@AlexDeveloperUwU

Copy link
Copy Markdown
Contributor Author

Follow-up on your non-blocking note:

  • Merged the latest master into the branch (c74322a, docs-only), so it is no longer behind.
  • Added a regression test for the stale-listener scenario (4079a45): an ok:false reply settles take A, take B reuses the handle, then A's late abort fires. It asserts B is not killed and only aborts on its own deadline.

Checked on my fork's CI: all 1224 tests pass at 4079a45. With the guard removed, only the new test fails, so it pins the fix.

@TriDefender
TriDefender merged commit 17efdcf into TriDefender:master Nov 16, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] captcha: hung worker triggers timeout cascade (30000ms timeout, 4 retries, upstream 503s)

2 participants