Repository navigation
gateway: blue-green successor stands by before the single-owner guards - #10
Merged
Merged
Conversation
…uard bypass Finding #2 resolution that sidesteps the custody question entirely: the successor boots to STANDBY (interpreter + config pre-warmed, watchdog fed, standby marker written), waits for the predecessor's exit, then runs the normal ownership bootstrap — attach decision, PID record, profile reservation and stores all see a dead owner, so no guard is bypassed and two processes never hold the stores. Blue promotes green at standby-confirm (sd_notify MAINPID from blue; green feeds the watchdog from standby so the early switch cannot starve it), then drains and exits; green boots fully into a freed home with no cold-start gap. The serve-during-drain protocol (finding #9/#10 custody, approval affinity, fixed ports) stays with the design session — this changes the takeover, not the drain semantics. Tests: standby marker round-trip/isolation, probe vs marker/dead-child, fallback termination, predecessor-exit arming, PID-reuse guard. The claim and identify-probe machinery was removed with the standby design.
…guards
The successor `gateway run` exited at once with "The host gateway already serves
profile 'default'": run_gateway's host-attach guard saw blue alive and refused,
before any handover code ran. The v2 standby (cherry-picked) waited inside
start_gateway — still after that guard — so it could never have worked live.
- handover.stand_by(): runs at the top of run_gateway, BEFORE every single-owner
guard. Writes the standby marker, disarms the 300s startup watchdog (a long
drain is not a wedged boot; run_gateway re-arms it), feeds systemd WATCHDOG=1
from a thread (green is MainPID from standby-confirm) through BOOT_FEED_S of
post-takeover boot, and blocks until blue exits. All guards then see a dead
owner; nothing is bypassed.
- wait_successor_standby polled the marker by the CHILD's pid; green keys it by
the PARENT's pid, so blue never saw a real standby. Poll by os.getpid().
- drain_control: allow config.get + session.stats during a restart's after-turn
wait. ACP load_session calls both; refusing them made every editor reconnect
fail with runtime_draining for the whole drain ("sessions failed").
- run_shutdown: getattr for the handover gate attrs (10 restart_drain tests on
fork main crashed on runner doubles without them).
Tests: stand_by blocks until predecessor exit + marker lifecycle; run_gateway
stands by before the attach guard (the live regression); marker keyed by blue's
pid; drain allowlist for ACP reattach reads.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Two
hermes gateway restartruns on Eternus today (18:39 and earlier) fell back to the classic drain. Journal:The successor's
run_gateway→_attach_to_host_gateway_or_guardsaw blue alive and exited before any handover code ran. Merged v1 (#6) never worked live. The v2 standby (3996e635, pushed after #6 merged, carried here) waited insidestart_gateway, which is still after that guard, so it would have failed the same way.During the fallback drain, every editor reconnect also failed with
runtime_draining, because ACPload_sessioncallsconfig.get/session.statsand neither was on the restart-wait allowlist. That is the "sessions failed" the user saw.What
hermes_cli/gateway.pyrun_gatewaycallshandover.stand_by()before the attach, supervised and duplicate-process guards.gateway/handover.pystand_by()is synchronous and does four things: writes the standby marker; disarms the 300 s startup watchdog (re-armed after the guards); feedsWATCHDOG=1from a thread, because green is MainPID from standby-confirm and its own watchdog starts only at READY; then blocks until blue exits. It has no timeout, because blue's restart-handoff watchdog already bounds blue's exit. The PR also fixeswait_successor_standby, which polled the marker by the child pid while green keys it by the parent pid, so blue never saw a real standby. The deadarm_after_predecessor_exitandfeed_watchdog_until_predecessor_exitare removed.gateway/run_bootstrap.pygateway/drain_control.pyconfig.getandsession.statsare allowed during a restart's after-turn wait.gateway/run_shutdown.pygetattr, so 10test_restart_draintests on fork main no longer crash on runner doubles.Every guard then sees a dead owner, so nothing is bypassed and the stores are never co-held. Serve-during-drain (new sessions landing on green while blue drains) is still out of scope.
Gate (MacBook arm64,
uv sync --frozen, py3.14)tests/gateway/test_handover.py: 13 passed (new: stand_by lifecycle, stand_by-before-attach-guard regression, marker keyed by blue pid, drain allowlist)test_restart_wait_ws_serves_the_parked_turn…. It fails identically on unmodified fork main, becauseCanonicalSnapshot.handoffis undeclared on fork main (the live tree declares it).getattrcrash fixed here.tests/hermes_cli/test_gateway*.py(4 files): 144 passedActivation
Install on Eternus, then the operator runs
hermes gateway restart(SIGUSR1). Then check the journal for:Blue-green handover: standing by until gateway PID …MAINPIDswitch withNRestartsunchangedalready serves profile