Skip to content

gateway: blue-green successor stands by before the single-owner guards - #10

Merged
IEatCodeDaily merged 2 commits into
mainfrom
fix/gateway-handover-standby
Nov 18, 2026
Merged

IEatCodeDaily merged 2 commits into
mainfrom
fix/gateway-handover-standby

Conversation

@IEatCodeDaily

Copy link
Copy Markdown
Owner

Why

Two hermes gateway restart runs on Eternus today (18:39 and earlier) fell back to the classic drain. Journal:

hermes[4082114]: ✓ The host gateway already serves profile 'default' — nothing to start.
gateway.handover: Blue-green handover: successor (PID 4082114) exited during startup

The successor's run_gateway → _attach_to_host_gateway_or_guard saw blue alive and exited before any handover code ran. Merged v1 (#6) never worked live. The v2 standby (3996e635, pushed after #6 merged, carried here) waited inside start_gateway, which is still after that guard, so it would have failed the same way.

During the fallback drain, every editor reconnect also failed with runtime_draining, because ACP load_session calls config.get/session.stats and neither was on the restart-wait allowlist. That is the "sessions failed" the user saw.

What

File Change
hermes_cli/gateway.py run_gateway calls handover.stand_by() before the attach, supervised and duplicate-process guards.
gateway/handover.py stand_by() is synchronous and does four things: writes the standby marker; disarms the 300 s startup watchdog (re-armed after the guards); feeds WATCHDOG=1 from a thread, because green is MainPID from standby-confirm and its own watchdog starts only at READY; then blocks until blue exits. It has no timeout, because blue's restart-handoff watchdog already bounds blue's exit. The PR also fixes wait_successor_standby, which polled the marker by the child pid while green keys it by the parent pid, so blue never saw a real standby. The dead arm_after_predecessor_exit and feed_watchdog_until_predecessor_exit are removed.
gateway/run_bootstrap.py The in-loop standby is removed (now an ordinary PID claim).
gateway/drain_control.py config.get and session.stats are allowed during a restart's after-turn wait.
gateway/run_shutdown.py The handover gate attrs are read with getattr, so 10 test_restart_drain tests on fork main no longer crash on runner doubles.

Every guard then sees a dead owner, so nothing is bypassed and the stores are never co-held. Serve-during-drain (new sessions landing on green while blue drains) is still out of scope.

Gate (MacBook arm64, uv sync --frozen, py3.14)

  • ruff clean on all 6 files
  • tests/gateway/test_handover.py: 13 passed (new: stand_by lifecycle, stand_by-before-attach-guard regression, marker keyed by blue pid, drain allowlist)
  • restart/drain suites: 69 passed, 1 failed
    • The failure is test_restart_wait_ws_serves_the_parked_turn…. It fails identically on unmodified fork main, because CanonicalSnapshot.handoff is undeclared on fork main (the live tree declares it).
    • Fork main baseline is 11 failed. The other 10 are the getattr crash fixed here.
  • tests/hermes_cli/test_gateway*.py (4 files): 144 passed

Activation

Install on Eternus, then the operator runs hermes gateway restart (SIGUSR1). Then check the journal for:

  • Blue-green handover: standing by until gateway PID …
  • MAINPID switch with NRestarts unchanged
  • no already serves profile

…uard bypass

Finding #2 resolution that sidesteps the custody question entirely: the
successor boots to STANDBY (interpreter + config pre-warmed, watchdog fed,
standby marker written), waits for the predecessor's exit, then runs the
normal ownership bootstrap — attach decision, PID record, profile
reservation and stores all see a dead owner, so no guard is bypassed and
two processes never hold the stores.

Blue promotes green at standby-confirm (sd_notify MAINPID from blue; green
feeds the watchdog from standby so the early switch cannot starve it), then
drains and exits; green boots fully into a freed home with no cold-start
gap. The serve-during-drain protocol (finding #9/#10 custody, approval
affinity, fixed ports) stays with the design session — this changes the
takeover, not the drain semantics.

Tests: standby marker round-trip/isolation, probe vs marker/dead-child,
fallback termination, predecessor-exit arming, PID-reuse guard. The claim
and identify-probe machinery was removed with the standby design.
…guards

The successor `gateway run` exited at once with "The host gateway already serves
profile 'default'": run_gateway's host-attach guard saw blue alive and refused,
before any handover code ran. The v2 standby (cherry-picked) waited inside
start_gateway — still after that guard — so it could never have worked live.

- handover.stand_by(): runs at the top of run_gateway, BEFORE every single-owner
  guard. Writes the standby marker, disarms the 300s startup watchdog (a long
  drain is not a wedged boot; run_gateway re-arms it), feeds systemd WATCHDOG=1
  from a thread (green is MainPID from standby-confirm) through BOOT_FEED_S of
  post-takeover boot, and blocks until blue exits. All guards then see a dead
  owner; nothing is bypassed.
- wait_successor_standby polled the marker by the CHILD's pid; green keys it by
  the PARENT's pid, so blue never saw a real standby. Poll by os.getpid().
- drain_control: allow config.get + session.stats during a restart's after-turn
  wait. ACP load_session calls both; refusing them made every editor reconnect
  fail with runtime_draining for the whole drain ("sessions failed").
- run_shutdown: getattr for the handover gate attrs (10 restart_drain tests on
  fork main crashed on runner doubles without them).

Tests: stand_by blocks until predecessor exit + marker lifecycle; run_gateway
stands by before the attach guard (the live regression); marker keyed by blue's
pid; drain allowlist for ACP reattach reads.
@IEatCodeDaily
IEatCodeDaily merged commit f092952 into main Nov 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant