Skip to content

Keep workers reconnecting across hub outages#258

Merged
PaulFidika merged 2 commits into
masterfrom
agent/worker-stream-reconnect
Jul 15, 2026
Merged

Keep workers reconnecting across hub outages#258
PaulFidika merged 2 commits into
masterfrom
agent/worker-stream-reconnect

Conversation

@PaulFidika

Copy link
Copy Markdown
Contributor

What changed

  • remove the 600-second disconnected worker self-exit and its WORKER_DISCONNECTED_TIMEOUT_S configuration
  • keep the existing jittered reconnect loop active until explicit Drain or process stop
  • delete the drain-only timeout bypass that became dead code
  • add a real gRPC regression that stops the hub, observes repeated reconnect attempts, rebinds the same port, and verifies a fresh Hello without worker exit

Root cause

The worker owned a fixed disconnected lifetime even though the protocol contract requires reconnecting after every non-Drain stream loss. Once that lifetime elapsed, PID 1 exited while the provider pod could remain allocated and appear RUNNING but idle. The live four-minute incident did not reach the old 600-second default, so this removes a proven lifecycle defect without claiming it was the only cause of that event.

Validation

  • pytest tests/test_drain_flush.py tests/test_token_refresh.py tests/test_worker_grpc_e2e.py: 24 passed
  • mypy src: clean across 114 files
  • ruff check: clean
  • full suite: 1,078 passed, 13 skipped, one unrelated CUDA residency assertion; the same assertion fails on unchanged master in this host environment
  • git diff --check: clean

@PaulFidika PaulFidika merged commit be53038 into master Jul 15, 2026
1 check passed
@PaulFidika PaulFidika deleted the agent/worker-stream-reconnect branch July 15, 2026 01:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant