Skip to content

fix(stack): log target failures and handle disconnections quietly - #6691

Merged
avallete merged 5 commits into
developfrom
avallete/magical-knuth-y0ezec
Sep 21, 2026
Merged

avallete merged 5 commits into
developfrom
avallete/magical-knuth-y0ezec

Conversation

@avallete

@avallete avallete commented Sep 21, 2026

Copy link
Copy Markdown
Member

TL;DR

  • A failed lazy wake was invisible. It surfaced only as a 502 or a dead socket, with the cause discarded and error=undefined in stack status. The proxy now logs the route and cause, and the failure reaches the service observation.
  • Ordinary client disconnects stay quiet. A client that goes away is its own typed error, not a proxy failure, so normal traffic adds no error logs.
  • stack port tests (windows-latest) could not allocate a port. The auto scan walked contiguously and died inside a Windows reserved block. It now walks 20000..32767 with stride 257 per ADR 0017, so a reserved block up to 12511 of the 12768 ports is escaped within the bounded attempts.
  • Port allocation is reproducible. The scan start derives from the stack rather than a random draw, so a stack reassigns the same port and a future failure repeats instead of depending on the draw.

Details

Two independent stack failures, both surfaced while investigating the 2026-09-21 Test breakage.

A failed lazy wake was unobservable

A wake that failed reached clients only as a dead connection. HttpProxy discarded the cause in both arms — the request path answered 502 and the upgrade path destroyed the socket — and Service.startAt runs definition.prepare (where native artifact preparation happens) before it touches the observation, so a preparation failure left error undefined. Every wake failure was therefore indistinguishable from a flake.

Both proxy arms now log the route id and the underlying cause before responding, and a failed wake preparation reaches the service observation, so stack status and the reopened-stack diagnostics name the cause instead of printing error=undefined.

The two halves cover different topologies. A detached owner is spawned with stdout/stderr ignored, so the proxy log lines land only where the runtime shares the caller's process; the recorded observation is what names the cause through stack status and the e2e diagnostics in the detached topology. Retaining a detached owner's output is a separate pre-existing gap.

Client exits stay quiet, and are told apart from upstream faults by type rather than by message: a client that disconnects while the target is waking, goes away mid-response, or resets an established upgrade fails with HttpProxyDisconnected, while upstream faults keep failing with HttpProxyError and log. Both paths settle before destroying their sockets, because destroying a partially received upstream response emits aborted synchronously and would otherwise resettle a disconnect as a proxy failure. A preparation failure is recorded only while the service is still the stopped, registered attempt that prepared it, so a concurrent start that already relaunched keeps its own state.

stack port tests (windows-latest) could not allocate an auto port

The auto scan walked contiguously from a random start over 20000..49999 and gave up after 64 consecutive bind failures. Windows publishes reserved TCP ranges as contiguous blocks commonly wider than 64 ports, so a start landing inside one failed with No public port is available while tens of thousands of ports were free. That range also overlapped the Linux default ephemeral range.

The scan follows ADR 0017 again: 20000..32767 with stride 257, which is co-prime with the span and so traverses every candidate once across a full scan. Within the ADR's 64-attempt bound, a contiguous reserved block up to 12511 of the 12768 ports is always escaped, worst case 49 failures, verified over every start and block offset. Wider blocks can still exhaust the bound — that limit is the ADR's, and the stride does not remove it — but they are far beyond the width of a published excluded range.

The one deliberate divergence from the ADR is the scan start, which derives from the stack's project root, id, and listener key rather than a random draw. Allocation is reproducible: a stack reassigns the same port across runs, separate checkouts stay apart, and a future failure repeats instead of depending on the draw. The ADR records this.

The exhausted error also carries the last bind failure, so a reserved range (EACCES) reads differently from an occupied port (EADDRINUSE) from the log alone.

Changes

  • HttpProxy.ts: log route request and upgrade failures with the route id and cause; give client-originated disconnects a distinct tagged error; settle before destroying sockets
  • Ports.ts: walk the documented range with a co-prime stride from a derived start; attach the last bind failure to the exhausted error. No longer needs the platform crypto service, so HostProcess stops carrying that requirement
  • Service.ts: record a failed wake preparation on the observation, guarded against a concurrent relaunch
  • docs/adr/0017: record the derived scan start

Service.ts here is the wake path only; restart is unchanged because its caller already receives the error.

🤖 Generated with Claude Code

https://claude.ai/code/session_01LcUaThuQfDKPwi9mCE9XeD

avallete and others added 2 commits September 21, 2026 16:31
A wake failure reached clients only as a dead connection: the request path
answered 502 and the upgrade path destroyed the socket, both discarding the
cause, while preparation failures never reached the service observation. Any
failed wake was therefore indistinguishable from a flake.

Both proxy arms now log the route and the underlying cause, with a client that
disconnects first carrying its own tagged error so ordinary aborts stay quiet,
and a failed wake preparation is recorded on the observation so status names it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LcUaThuQfDKPwi9mCE9XeD
… them

The auto scan walked contiguously from a random start and gave up after 64
consecutive bind failures, so a start inside a reserved block wider than 64
ports failed with "No public port is available" while tens of thousands of
ports were free. Windows publishes such excluded ranges, which is where this
surfaced.

Each probe is now redrawn, and the exhausted error carries the last bind
failure so a reserved range reads differently from an occupied port.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LcUaThuQfDKPwi9mCE9XeD
@avallete
avallete requested a review from a team as a code owner September 21, 2026 16:33

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI Review

Three substantive findings are confirmed after merging the overlapping reviews: randomized port probing can falsely exhaust candidates, stale preparation failures can overwrite current observations, and post-acquisition client disconnects are logged as proxy errors. Two Claude-only consistency/documentation findings are refuted with concrete counter-evidence. No uncertain findings remain.

Findings

Severity Location Category Sources Claim
🟡 MINOR packages/stack/src/HttpProxy.ts:292 observability claude+codex Client disconnects after target acquisition are classified as proxy failures and logged at error level.
🟡 MINOR packages/stack/src/Ports.ts:66 correctness claude+codex Automatic port probing samples with replacement and omits port 49999, allowing false exhaustion while usable candidates remain.
🟡 MINOR packages/stack/src/Service.ts:433 concurrency claude+codex A stale preparation failure can overwrite the error field of a newer running service observation.
Refuted findings (kept for transparency, not posted as review comments)
  • packages/stack/src/Service.ts:431 (consistency): Recording preparation failures for every startAt call but not restart is an inconsistent loss of restart failure observability.
    Refuted: A restart preparation failure is not dropped: it is returned to the restart caller, and the currently running service remains valid. The comment identifies the distinct wake case, which lacks such a caller; it does not require failed restart attempts to mark a healthy running observation as errored.
  • packages/stack/src/Ports.ts:66 (documentation): This PR newly makes automatic port probing inconsistent with ADR 0017.
    Refuted: The ADR/code divergence predates this PR, so the change did not introduce the claimed documentation regression. The newly introduced exclusive-upper-bound problem is preserved in the merged port-correctness finding.

Stats

Claude findings: 5 · Codex findings: 3 · Confirmed: 3 · Refuted: 2 · Uncertain: 0


Models: claude-opus-5 + gpt-5.6-sol · Trigger: auto · Workflow run

This review runs once per PR. A maintainer can request another with a /ai-review comment.

Comment thread packages/stack/src/HttpProxy.ts
Comment thread packages/stack/src/Ports.ts Outdated
Comment thread packages/stack/src/Service.ts Outdated
avallete and others added 2 commits September 21, 2026 16:48
Drawing a random port per probe cleared reserved ranges only by chance. A
stride co-prime with the span visits every port once and keeps consecutive
probes far apart instead, so a reserved range narrower than the stride costs
at most one probe and can never exhaust the failure budget.

The scan starts from an offset derived from the stack's checkout, id, and key,
so separate checkouts and keys stay apart while one stack reassigns the same
port across runs. Ports no longer needs the platform crypto service, and its
callers stop carrying that requirement.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LcUaThuQfDKPwi9mCE9XeD
The auto scan now follows the architecture ADR again: it walks `20000..32767`
with stride 257, staying below the Linux ephemeral range and visiting every
candidate once, so repeated probes can neither skip a port nor spend the bind
allowance twice. Only the start differs from the ADR, deriving from the stack
instead of a random draw; the ADR records that.

A client that goes away mid-response or resets an established upgrade now fails
with the disconnect error rather than a proxy error, so ordinary client exits
stay out of the log, and a preparation failure reaches the observation only
while the service is still the stopped, registered attempt that prepared it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LcUaThuQfDKPwi9mCE9XeD
@avallete
avallete enabled auto-merge September 21, 2026 17:55

@jgoux jgoux left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes for the reproduced mid-response disconnect regression below. The existing 42 targeted integration tests pass, but a client that closes after receiving the first response chunk produces a spurious error log; the same reproduction passes against the base.

The port scan is a separate, reasonable fix; its description overstates the guarantee, as noted inline. A separate pre-existing limitation is that detached hosts discard stdout/stderr, so the new proxy logs are not retained in that topology.

Comment thread packages/stack/src/HttpProxy.ts Outdated
Comment thread packages/stack/src/Ports.ts
Destroying a partially received upstream response emits `aborted` synchronously,
and the abort listener re-entered the abandon path while the operation was still
unsettled. A client that closed after receiving a body chunk therefore resettled
as `HttpProxyError: undefined` and logged, defeating the quiet classification.

Both proxy paths now settle and drop their listeners before destroying, and the
mid-response disconnect is covered: the previous coverage closed the client
before any upstream response arrived.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LcUaThuQfDKPwi9mCE9XeD
@avallete
avallete requested a review from jgoux September 21, 2026 18:41
@avallete
avallete added this pull request to the merge queue Sep 21, 2026
Merged via the queue into develop with commit 07438e2 Sep 21, 2026
48 checks passed
@avallete
avallete deleted the avallete/magical-knuth-y0ezec branch September 21, 2026 19:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants