Skip to content

Add failure-only production 503 diagnostics - #40

Merged
sdcarlson merged 11 commits into
mainfrom
cursor/issue22-production-diagnostics-41a2
Sep 6, 2026
Merged

sdcarlson merged 11 commits into
mainfrom
cursor/issue22-production-diagnostics-41a2

Conversation

@sdcarlson

@sdcarlson sdcarlson commented Sep 6, 2026 •

Copy link
Copy Markdown
Contributor

Problem and result

Related to #22. Does not close #22. Issue 22 remains unresolved.

The compiled production integration test can return 503 on the valid owner-A save immediately after the intentional cross-origin 403. This PR only adds failure-only diagnostics and reliable process-group teardown so the next failure can be distinguished from an application response. It does not change production runtime, retries, health polls, CI workflows, or dependencies.

CI has already captured a Wrangler ProxyWorker synthesized body (Your worker restarted mid-request..., text/plain; charset=UTF-8, retry-after: 0, parent exit/signal null). That is not evidence of an actual restart. No runtime fix is included.

The process-group fixture now installs the SIGTERM handler before publishing grandchild=, and flushes after-term in the write callback before exit, so Quality cannot default-kill the leader and drop that line.

Quality on head dda4090 (run 34003902709) failed pnpm lint with 12 oxlint errors in tests/production-diagnostics.test.mjs: 10 un-awaited test() registrations and 2 unary minuses on ChildProcess.pid (number | undefined). The aggregator CI job failed because Quality failed. This revision matches the existing void test() style and narrows pids before group SIGKILL. Helper 3d65c2d5 is unchanged.

This branch includes current main (7cfec26, including #36) merged into the diagnostic history. Reviewed diagnostic commit 3d65c2d5 is an ancestor.

Evidence

  • Exact head: 7b1f4cc3a7d9b82f20cc5e240e544acfb90d0d9f
  • pnpm lint: oxlint exit 0
  • pnpm typecheck: tsc --noEmit exit 0
  • pnpm test: 108 passed, 0 failed
  • pnpm build: vinext build complete
  • pnpm test:production (once): PASS (built app rendering/assets, signed identity, persistence, tenant isolation, forged headers, request origin, exact acceptance and stale-write integrity)
  • GitHub run 34004136177 on this head: Quality, CI aggregator, and remaining required jobs succeeded
  • User-visible proof: none; CI diagnostic only, fictional records and locally generated tokens
  • Security, data migration, and rollback impact: no schema or runtime auth change from the diagnostic work. Request headers and authentication JWTs are not printed. CodeQL alert 4 was reviewed/dismissed as used in tests; query/gates remain enabled.

Review

  • One issue owns this scope; unrelated changes are separate.
  • Tests exercise the changed behavior, including failure paths.
  • No credentials, personal applications, or private research are included.
  • Agent findings are fixed or answered with evidence.
  • A maintainer other than the author will approve.

Diagnostics alone must not close #22. Independent review is separate. No merge or approval from this agent.

Open in Web Open in Cursor 

Failure-only diagnostics capture status, selected headers, a
deadline-bounded body, and process exit/signal without printing request
headers or JWTs. POSIX teardown still signals the detached process
group; Windows keeps taskkill /T. Related to #22.

Co-authored-by: Seth Carlson <sdcarlson@users.noreply.github.com>
Comment thread scripts/ci/run-production.mjs Dismissed
Pending reader.cancel() no longer outlives the timeout, and chunked
reads no longer leave a delay timer per chunk. Redaction and success-path
body handling are unchanged.

Co-authored-by: Seth Carlson <sdcarlson@users.noreply.github.com>

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale comment

No actionable defects at 7acccc79e3445e3a06d1331871586cc12c5a8f17.

This synchronize only changes readBoundedBody to one cancellable deadline and stops awaiting reader.cancel() past that deadline. Success-path request semantics, JWT redaction, and selected response headers (content-type, retry-after) are unchanged.

Ran

  • node --test tests/production-diagnostics.test.mjs: 8/8 passed on this checkout.
  • GitHub Actions 34002889572 for this head: Quality passed. Integration (production) failed on the existing owner-a save 503 (Your worker restarted mid-request...). expectStatus recorded operation, status, selected headers, a 125-byte body, and exit=null. That is issue #22, not a helper regression. Artifact integration-production-1 contains the unexpected-response block after the 403.

Inspected (not re-executed here)

  • Diff e0f69c97...7acccc79 (3 files) plus adjacent expectStatus / teardown wiring.
  • AGENTS.md + docs/threat-model.md: no auth, user-isolation, exact-acceptance, stale-write, migration, or workflow-permission changes. Request headers and JWTs are not printed; JWT-shaped tokens in a failure body are redacted. Failure bodies go only to the existing outputs/ci artifact.
  • Prior automation threads: none. CodeQL “network data written to file” is the intended bounded failure write.

The disclosed teardown residual (exitCode/signalCode ≠ stdio drained / detached grandchild) is not claimed fixed and did not hide this run’s 503 body. Human approval remains mandatory.

Open in Web View Automation 

Sent by Cursor Automation: RELAY pull request review

@sdcarlson

Copy link
Copy Markdown
Contributor Author

Independent review is still in progress; current head 7acccc79e3445e3a06d1331871586cc12c5a8f17 is not ready to merge.

The body-read deadline and leftover timer defects are fixed and independently verified. One real Linux teardown reproduction remains: when the detached group leader exits first, finishProductionServer returns in about 3 ms while a grandchild is still alive, closes the log, and misses final stdout/stderr. The reproduction's final cleanup killed the surviving process group. Grok is fixing this in the same PR and adding the regression.

Issue #22 remains open: these changes improve failure evidence and cleanup; they do not establish or fix the original intermittent 503 cause.

exitCode and signalCode no longer skip SIGTERM/SIGKILL of remaining
group members or waiting for both stdio streams to close. Related to #22.

Co-authored-by: Seth Carlson <sdcarlson@users.noreply.github.com>

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale comment

No actionable defects at 4aa68e563970916ba833350c0f63c2afe48d9198.

This synchronize stops treating a dead group leader as finished teardown. waitForClose now waits until stdout/stderr are drained and exitCode/signalCode is set; signalGroup still sends SIGTERM/SIGKILL to -pid after the leader exits; SIGKILL still runs when those pipes remain open. The new Linux regression covers a TERM-ignoring descendant that inherits the leader’s pipes.

Prior body-read deadline/timer findings remain addressed. No prior automation defect threads to resolve.

Ran

  • node --test tests/production-diagnostics.test.mjs: 9/9 passed on this checkout.
  • Re-read remote head immediately before posting: still 4aa68e5.
  • GitHub Actions 34003084962 for this head: Quality passed. Integration (production) failed on the existing owner-a save 503 (Your worker restarted mid-request...). expectStatus recorded operation, status, selected headers, a 125-byte body, and exit=null. That is issue #22, not a helper or teardown regression.

Inspected (not re-executed here)

  • Delta 7acccc79...4aa68e5 (2 files) plus adjacent finishProductionServer / run-production.mjs spawn (stdio: pipe, POSIX detached).
  • AGENTS.md + docs/threat-model.md: no auth, user-isolation, exact-acceptance, stale-write, migration, or workflow-permission changes. Request headers and JWTs are not printed; JWT-shaped tokens in a failure body are redacted. Failure bodies go only to the existing outputs/ci artifact.
  • CodeQL “network data written to file” remains the intended bounded failure write.

These changes improve failure evidence and cleanup. They do not establish or fix the original intermittent 503. Human approval remains mandatory.

Open in Web View Automation 

Sent by Cursor Automation: RELAY pull request review

A TERM-ignoring process can close stdout and stderr and still be alive.
SIGKILL now follows process-group existence, not stream state. Related
to #22.

Co-authored-by: Seth Carlson <sdcarlson@users.noreply.github.com>

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale comment

No actionable defects at 3d65c2d5fd8ac725e4b00b1dcc9642d2ce6c072c.

This synchronize only changes POSIX SIGKILL to follow process-group liveness (kill(-pid, 0)), not open stdio. That covers the residual from 4aa68e5: a TERM-ignoring process can end both streams and stay alive. Windows still uses leader exitCode/signalCode after taskkill /T /F. Prior body-read deadline/timer and leader-exit TERM/KILL findings remain addressed. No prior automation defect threads to resolve.

Ran

  • node --test tests/production-diagnostics.test.mjs: 10/10 passed on this checkout (includes the new closed-stdio KILL case).
  • Re-read remote head immediately before posting: still 3d65c2d.
  • GitHub Actions on this head: Quality, Integration (api/browser/production), Dependency security, Bundled agent contracts, CodeQL, and the required CI check all completed successfully. A green production job here does not close issue #22; that 503 remains intermittent and this PR does not change runtime behavior.

Inspected (not re-executed here)

  • Delta 4aa68e5...3d65c2d (2 files) plus adjacent finishProductionServer / waitForClose / run-production.mjs spawn (stdio: pipe, POSIX detached).
  • AGENTS.md + docs/threat-model.md: no auth, user-isolation, exact-acceptance, stale-write, migration, or workflow-permission changes. Request headers and JWTs are not printed; JWT-shaped tokens in a failure body are redacted. Failure bodies go only to the existing outputs/ci artifact.
  • CodeQL “network data written to file” remains the intended bounded failure write; maintainer already triaged that thread.

These changes improve failure evidence and cleanup. They do not establish or fix the original intermittent 503. Human approval remains mandatory.

Open in Web View Automation 

Sent by Cursor Automation: RELAY pull request review

Bring in the #15 session-refresh fixes while keeping the reviewed
issue 22 diagnostic commits.

Co-authored-by: Seth Carlson <sdcarlson@users.noreply.github.com>

@sykosyber sykosyber left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Partner Extra High review of 3d65c2d5fd8ac725e4b00b1dcc9642d2ce6c072c.

Independent of the High Fast automation comment on 4aa68e5. This is still a draft, still behind current main (431b207), and it does not close #22. Extra High is not approving.

Traced scripts/ci/production-diagnostics.mjs and scripts/ci/run-production.mjs:

  • Failure-only diagnostics: operation, status, content-type/retry-after, 2s/2048-byte body, pid/exit/signal. Request headers and JWTs are not printed; JWT-shaped tokens in a failure body are redacted.
  • waitForClose waits until stdout/stderr are drained and exitCode/signalCode is set.
  • signalGroup still sends SIGTERM/SIGKILL to -pid after the leader exits.
  • 3d65c2d adds groupAlive (process.kill(-pid, 0) on POSIX) so SIGKILL still runs when the group is alive even after those pipes have ended. That matches the closed-stdio leftover described on 4aa68e5.
  • No retries, app/auth, workflow-permission, or schema changes. Assertions stay strict.

Required checks on this head are green, including Integration (production): run 34003374202. That is evidence the helper did not break the success path on this run; it is not a #22 root-cause fix and does not make this PR merge-ready.

Mark ready and rebase onto current main before Extra High will consider approval. Keep #22 open.

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale comment

No actionable defects at 20c95b4d32176f88145e2f5d8fa39e2b8590a051.

This synchronize is a merge of main (431b207) onto the already-reviewed diagnostics head 3d65c2d. The unique PR files (scripts/ci/production-diagnostics.mjs, scripts/ci/run-production.mjs, tests/production-diagnostics.test.mjs) are unchanged. Prior body-read deadline/timer, leader-exit TERM/KILL, and closed-stdio SIGKILL findings remain addressed. No prior automation defect threads to resolve.

Ran

  • node --test tests/production-diagnostics.test.mjs: 10/10 passed on this checkout.
  • Re-read remote head immediately before posting: still 20c95b4.
  • GitHub Actions 34003520817 for this head: Quality, Integration (production), Integration (browser), Dependency security, Bundled agent contracts, and CodeQL succeeded. Required CI failed because Integration (api) exited: run-integration.mjs reported the Sites/Vite worker dead after workerd RPC traversalLimitInWords errors and Port … is already in use. That path is not in this PR’s unique diff and is not a production-helper regression. Integration (production) stayed green. A green production job does not close issue #22.

Inspected (not re-executed here)

  • Diff vs main: only the three diagnostics files. Merge delta vs 3d65c2d is the already-landed #15 session-refresh files.
  • Adjacent finishProductionServer / groupAlive / run-production.mjs spawn (stdio: pipe, POSIX detached). Workflow permissions: contents: read unchanged; failure bodies still go only to the existing outputs/ci artifact.
  • AGENTS.md + docs/threat-model.md: no auth, user-isolation, exact-acceptance, stale-write, migration, or workflow-permission changes. Request headers and JWTs are not printed; JWT-shaped tokens in a failure body are redacted.
  • CodeQL “network data written to file” remains the intended bounded failure write; maintainer already triaged that thread.

These changes improve failure evidence and cleanup. They do not establish or fix the original intermittent 503. Human approval remains mandatory.

Open in Web View Automation 

Sent by Cursor Automation: RELAY pull request review

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale comment

Quality failed at 85dfe36. Unique diagnostics files are unchanged from 3d65c2d; this synchronize only merges already-landed main (#13, #39). Prior body-read deadline, leftover-group TERM/KILL, and closed-stdio SIGKILL findings remain addressed. No prior automation defect threads to resolve.

High — teardown test races the SIGTERM handler (tests/production-diagnostics.test.mjs:157-161)

Trigger: Actions 34003642709 Quality pnpm test on this head. detached process-group teardown captures post-SIGTERM output and reaps nested children failed: log was grandchild=2693 with no after-term.

The child prints grandchild= before process.on('SIGTERM'). The parent treats that line as ready and immediately signals the group. If the handler is not installed yet, default SIGTERM kills the leader and the assertion cannot pass. Immediate process.exit(0) after stdout.write can also drop the line.

Impact: required Quality is red; the test does not reliably prove post-SIGTERM log capture.

Smallest fix: register the SIGTERM handler (and spawn the grandchild) before printing grandchild=. Flush stdout in the write callback before exit.

Ran: node --test tests/production-diagnostics.test.mjs 10/10 locally; 30/30 on that case. Re-read remote head immediately before posting: still 85dfe36. Inspected Actions 34003642709: Quality failed on that test; lint/typecheck passed; Integration (production) and Integration (api) succeeded.

Inspected (not re-executed here): unique diff vs main (3 files); merge delta vs 20c95b4 is #13/#39 only; adjacent finishProductionServer / groupAlive / spawn (stdio: pipe, POSIX detached); AGENTS.md + docs/threat-model.md (no auth, isolation, exact-acceptance, stale-write, migration, or workflow-permission changes); permissions: contents: read unchanged. Request headers and JWTs are not printed.

These helpers still do not fix issue #22. Human approval remains mandatory.

Open in Web View Automation 

Sent by Cursor Automation: RELAY pull request review

Comment thread tests/production-diagnostics.test.mjs Resolved
The process-group fixture published grandchild= before the leader
handled SIGTERM, so teardown could default-kill and drop after-term.
Flush that line in the write callback before exit. Related to #22.

Co-authored-by: Seth Carlson <sdcarlson@users.noreply.github.com>
sdcarlson pushed a commit that referenced this pull request Sep 6, 2026
<!-- CURSOR_AGENT_PR_BODY_BEGIN -->
## Problem and result

Closes #8
Closes #9
Closes #19

`main` already includes protected CI/CD (#14), context playbooks (#18),
session-expiry (#15 / #7), ChatGPT/Codex public-copy (#13), Apache 2.0
(#39 / #38), and tracker CSV (#11 / #17). This branch merges that tip
and keeps the remaining unique trees:

- Local-dev Node/port/sign-in, with a Windows-safe Node guard instead of
Bash (#12 / #19)
- Copy/paste draft handoffs, exact-acceptance actions, and
dirty-navigation guards (#24)
- Selected-job review history beyond the GET 200-event window (#8)
- Assistant packets for valid jobs with `url: null` (#9)
- Canonical public-copy listing for Tracker CSV (the #11 squash kept
that note outside the maintained block)

Head `4d8f007` merges `origin/main` (`256854b`, #11) into the previous
landing. Conflict resolution kept `void test` wrappers, history/null-URL
API coverage, and one Tracker CSV setup section.

The changes-requested review is on superseded `9e307b9`. Independent
preservation review of `8d6ba1a` retired the five original code
findings. These merges do not restore them.

Deferred, matching maintainer comments, and **not** in this PR:

- Profile calibration loop (#26)
- `shadcn` 4.20.1 (#32)
- Production 503 diagnostics (#22 / draft #40)
- Assistant research/next-task handoff (#37 / #41) — stacked on landed
#13, not this consolidation
- Unused UI-kit cut / competing review-loop draft (#42)

Keep #16 open: remaining development-tool advisories. Keep #21 open
until post-merge Pages verification. Keep #22 open. Cloudflare hosting
(#3), two-account isolation (#4), recruiting (#5), backup rehearsal
(#6), and Grok Bot VM command review (#20) still need human/environment
work.

## Evidence

- Local on `4d8f007`: `pnpm lint`, `pnpm typecheck`, `pnpm test` (98
passed), `pnpm public-copy:check`, `pnpm build`, `pnpm test:api`, `pnpm
test:e2e` (6 passed), `pnpm test:production` passed.
- GitHub CI on `4d8f007` ([run
34003847782](https://github.com/SyberLabs/relay/actions/runs/34003847782)):
quality, API, browser, production, CodeQL, dependency-security, and
bundled agent contracts were running or already green after this merge;
required `CI` still needs the remaining jobs.
- User-visible proof: copy/paste packet, Windows-safe
`scripts/ensure-node.mjs`, Load earlier review history, null-URL
packets. CSV import and ChatGPT/Codex prepare now come from `main`.
- Security, data migration, and rollback impact: no new D1 migration.
React trio stays at 19.2.8. CI Vinext still uses ephemeral ports when
`RELAY_CI_STATE` is set. Production integration keeps the original
request sequence.

## Review

- [ ] One issue owns this scope; unrelated changes are separate. This is
an explicit integration PR covering the remaining open merge train.
- [x] Tests exercise the changed behavior, including failure paths
(including compiled Workspace continuation and history-after-expiry
regressions).
- [x] No credentials, personal applications, or private research are
included.
- [x] Agent findings from the `9e307b9` review remain addressed.
- [ ] A maintainer other than the author will approve. **@sdcarlson must
approve the latest push** (`4d8f007`); prior reviews do not cover this
merge commit.
- [x] Public capabilities, integration limits and lead-engineer credit
match `docs/public-copy.json`; run `pnpm public-copy:check` and follow
`docs/PUBLIC-COPY.md` when these change.

After merge, close the superseded feature PRs (#12, #24) as landed here
rather than squash-merging them separately. #11, #13, #15, and #39 are
already merged. #26 and #32 stay closed as deferred. #40, #41, and #42
stay on their own tracks.

<!-- CURSOR_AGENT_PR_BODY_END -->

<div><a
href="https://cursor.com/agents/bc-3877d527-b5e4-46d7-8c5f-573418309ece?cursor_ref=pr_footer&cursor_cta=open_in_web"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://cursor.com/assets/images/open-in-web-dark.png"><source
media="(prefers-color-scheme: light)"
srcset="https://cursor.com/assets/images/open-in-web-light.png"><img
alt="Open in Web" width="114" height="28"
src="https://cursor.com/assets/images/open-in-web-dark.png"></picture></a>&nbsp;<a
href="https://cursor.com/background-agent?bcId=bc-3877d527-b5e4-46d7-8c5f-573418309ece&cursor_ref=pr_footer&cursor_cta=open_in_cursor"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://cursor.com/assets/images/open-in-cursor-dark.png"><source
media="(prefers-color-scheme: light)"
srcset="https://cursor.com/assets/images/open-in-cursor-light.png"><img
alt="Open in Cursor" width="131" height="28"
src="https://cursor.com/assets/images/open-in-cursor-dark.png"></picture></a>&nbsp;</div>

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: sykosyber <sykosyber@users.noreply.github.com>
@sdcarlson
sdcarlson marked this pull request as ready for review September 6, 2026 01:28

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale comment

Prior SIGTERM-handler race is fixed. Required Quality is red on this head for lint in the new test file.

High — new diagnostics tests fail required oxlint (tests/production-diagnostics.test.mjs)

Trigger: Actions 34004001791 Quality pnpm lint at 0416146. 12 errors, 0 warnings. Typecheck, unit, and build did not run.

  1. Every test(...) is a floating promise. Sibling unit files use void test(...). typescript(no-floating-promises) fires on all 10 cases (first at :32, last at :325).
  2. process.kill(-child.pid, 'SIGKILL') at :315 and :362: ChildProcess.pid is number | undefined. typescript(no-unsafe-unary-minus).

Impact: required Quality stays red. Integration (production) and Integration (api) succeeded on this head; that does not close #22.

Smallest fix: prefix each test( with void ; negate child.pid only after a truthy guard (as :189).

The earlier High teardown race is addressed in dda4090: SIGTERM is registered before grandchild=, and after-term is flushed in the write callback. Unique diagnostics files are unchanged since that commit; this synchronize only merges main (#36).

No auth, user-isolation, exact-acceptance, stale-write, migration, or workflow-permission defects in the unique diff. Request headers and JWTs are not printed; JWT-shaped tokens in a failure body are redacted.

Ran: node --test tests/production-diagnostics.test.mjs — 10/10 passed. Re-read remote head immediately before posting: still 0416146.

Inspected (not re-executed here): unique 3-file diff vs main; dda4090 fixture delta; finishProductionServer / groupAlive / spawn (stdio: pipe, POSIX detached); Actions 34004001791 logs/annotations; AGENTS.md + docs/threat-model.md; permissions: contents: read unchanged.

These helpers still do not fix issue #22. Human approval remains mandatory.

Open in Web View Automation 

Sent by Cursor Automation: RELAY pull request review

Comment thread tests/production-diagnostics.test.mjs Resolved
Comment thread tests/production-diagnostics.test.mjs Resolved

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale comment

Quality is red at 0416146. Unique diagnostic files are unchanged from dda4090. The prior SIGTERM-handler race remains fixed (handler is registered before grandchild=; after-term flushes in the write callback). Required Quality never reaches pnpm test on this head.

High — oxlint fails on the new diagnostics tests (tests/production-diagnostics.test.mjs)

Trigger: Actions 34004001791 Quality pnpm lint: 12 errors, 0 warnings, all in this file (same failures on dda4090 run 34003902709).

  • All 10 test( calls are floating promises. Type-aware oxlint requires void test( (see tests/workspace-refresh.test.mjs).
  • process.kill(-child.pid) at lines 315 and 362: ChildProcess.pid is number | undefined.

Impact: required Quality/CI fail. The fixture correction is not Quality-verified here. Integration (production) stayed green.

Smallest fix: prefix every test( with void; guard if (child.pid) before -child.pid, matching the cleanup at line 189.

Ran: none locally. Inspected GitHub annotations and Quality logs for 34004001791 and 34003902709.

Inspected (not re-executed): unique 3-file diagnostic diff vs this PR’s unique scope; merge 0416146 does not touch those files; finishProductionServer / groupAlive / spawn (stdio: pipe, POSIX detached); AGENTS.md + docs/threat-model.md (no auth, isolation, exact-acceptance, stale-write, migration, or workflow-permission changes); permissions: contents: read unchanged. Request headers and JWTs are not printed; JWT-shaped failure-body tokens are redacted.

These helpers still do not fix issue #22. Human approval remains mandatory.

Open in Web View Automation 

Sent by Cursor Automation: RELAY pull request review

Comment thread tests/production-diagnostics.test.mjs Resolved
Comment thread tests/production-diagnostics.test.mjs Resolved
Quality failed because node:test registrations were un-awaited and
unary minus was applied to ChildProcess.pid (number | undefined). Match
the existing void test() style and narrow pids before SIGKILL. No helper
or runtime change.

Co-authored-by: Seth Carlson <sdcarlson@users.noreply.github.com>

@sykosyber sykosyber left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Partner Extra High review of 0416146ccc8618ded01dd29cf069ad5b23c9c3cb. Not approving.

Independent of the High Fast comment on superseded 85dfe36. This head is marked ready and includes current main (7cfec262).

Traced the unique diagnostics files again:

  • Failure-only expectStatus: operation, status, content-type/retry-after, 2s/2048-byte body, pid/exit/signal. Request headers and JWTs are not printed; JWT-shaped tokens in a failure body are redacted.
  • waitForClose waits until stdout/stderr are drained and exitCode/signalCode is set. signalGroup still sends SIGTERM/SIGKILL to -pid. groupAlive (process.kill(-pid, 0) on POSIX) still KILLs a live group after those pipes end.
  • The process-group fixture now registers process.on('SIGTERM') (flushing after-term in the write callback before process.exit(0)) before spawning the grandchild and before printing grandchild=. That addresses the race Extra High independently confirmed on 85dfe36: Actions 34003642709 Quality failed because the log was only grandchild=2693\n with no after-term.
  • No retries, app/auth, workflow-permission, or schema changes. Assertions stay strict.

Required Quality on this head is red. Actions 34004001791: pnpm lint failed, so pnpm test did not run; combined CI failed because quality failed. oxlint typescript(no-floating-promises) on the un-awaited test(...) registrations in tests/production-diagnostics.test.mjs (lines 32, 57, 71, 86, 107, 130, 148, 211, 272) and typescript(no-unsafe-unary-minus) at line 315 (process.kill(-child.pid, …) with pid possibly undefined). Current main wraps node:test registrations with void test(...).

Integration (production) succeeded on this head. That is not a #22 root-cause fix and does not make this PR merge-ready. Keep #22 open.

@sdcarlson
sdcarlson requested a review from sykosyber September 6, 2026 01:33
@sdcarlson

Copy link
Copy Markdown
Contributor Author

Review update at 7b1f4cc: the reviewed diagnostics helper is unchanged from 3d65c2d. The fixture readiness/flush correction was independently source-reviewed at dda4090. I inspected the final test-only lint correction: explicit void test registrations and numeric PID narrowing, preserving all assertions and cleanup semantics. This resolves both pairs of lint findings on the prior merge head. Current GitHub Quality/release build, API, compiled production, dependencies and CodeQL pass; browser and automatic review are still running. No retry or assertion weakening was added. Mateo has been requested as peer reviewer. This supersedes the earlier pending teardown review; issue #22 remains open because diagnostics do not fix the intermittent 503.

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale comment

Prior High findings are addressed at 7b1f4cc. No new actionable defects.

Verified prior findings

  • oxlint no-floating-promises: all 10 cases now use void test(...), matching sibling unit files.
  • oxlint no-unsafe-unary-minus: every -pid SIGKILL is after typeof pid === 'number'.
  • SIGTERM fixture race remains fixed: handler is registered before grandchild=; after-term flushes in the write callback.

Ran

  • node --test tests/production-diagnostics.test.mjs: 10/10 passed on this checkout.

Inspected (not re-executed)

  • Unique 3-file diff vs main (7cfec262...7b1f4cc). This synchronize only changes the test file.
  • expectStatus / readBoundedBody / finishProductionServer / groupAlive / POSIX detached + stdio: pipe.
  • AGENTS.md + docs/threat-model.md: no auth, user-isolation, exact-acceptance, stale-write, migration, or workflow-permission changes. permissions: contents: read unchanged. Request headers and JWTs are not printed; JWT-shaped failure-body tokens are redacted.
  • Did not run pnpm lint here (no node_modules). Actions 34004136177 for this head was still in progress at review time.

These helpers still do not fix issue #22. Human approval remains mandatory.

Open in Web View Automation 

Sent by Cursor Automation: RELAY pull request review

@sdcarlson
sdcarlson enabled auto-merge (squash) September 6, 2026 01:34
@sdcarlson

Copy link
Copy Markdown
Contributor Author

Final validation: all ten GitHub checks, including aggregate CI, browser/API/compiled-production integration, release build, dependency security, CodeQL and automatic Cursor review, passed for exact head 7b1f4cc in run https://github.com/SyberLabs/relay/actions/runs/34004136177. All known review findings are resolved with evidence. Protected squash auto-merge is enabled and Mateo is requested; independent peer approval is still required. Issue #22 remains unresolved.

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale comment

No new actionable defects at 01adb7c. Unique diagnostics files are unchanged from reviewed 7b1f4cc; this synchronize only merges current main (007e0ae).

Verified prior findings

  • SIGTERM fixture race (dda4090): handler is registered before grandchild=; after-term flushes in the write callback.
  • oxlint no-floating-promises: all 10 cases use void test(...).
  • oxlint no-unsafe-unary-minus: every -pid SIGKILL is after typeof pid === 'number'.

Ran

  • node --test tests/production-diagnostics.test.mjs: 10/10 passed on this checkout.
  • Re-read remote head immediately before posting: still 01adb7c.
  • Actions 34004271384 Quality succeeded. Integration (production) failed on the existing owner-a save 503 (Your worker restarted mid-request...). expectStatus recorded operation, status, content-type/retry-after, a 125-byte body, and exit=null. That is issue #22, not a helper or merge regression.

Inspected (not re-executed)

  • Unique 3-file diff vs 007e0ae. Merge delta vs 7b1f4cc does not touch those files.
  • expectStatus / readBoundedBody / finishProductionServer / groupAlive / POSIX detached + stdio: pipe.
  • AGENTS.md + docs/threat-model.md: no auth, user-isolation, exact-acceptance, stale-write, migration, or workflow-permission changes. permissions: contents: read unchanged. Request headers and JWTs are not printed; JWT-shaped failure-body tokens are redacted.

These helpers still do not fix issue #22. Human approval remains mandatory.

Open in Web View Automation 

Sent by Cursor Automation: RELAY pull request review

@sdcarlson

Copy link
Copy Markdown
Contributor Author

Latest sync01adb7c repeated the known owner-a save503; diagnostics captured the explicit Wrangler proxy body (Your worker restarted mid-request...) in run34004271384/job101408524539. This confirms failure capture works, not the underlying cause. Requested one unchanged failed-job rerun; #22 stays open and no runtime assertion/retry is changed.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No new actionable defects at 0d7c68d. Unique diagnostics files are unchanged from reviewed 7b1f4cc; this synchronize only merges current main (13a2e7a).

Verified prior findings

  • SIGTERM fixture race (dda4090): handler is registered before grandchild=; after-term flushes in the write callback.
  • oxlint no-floating-promises: all 10 cases use void test(...).
  • oxlint no-unsafe-unary-minus: every -pid SIGKILL is after typeof pid === 'number'.

Ran

  • node --test tests/production-diagnostics.test.mjs: 10/10 passed on this checkout.
  • Re-read remote head immediately before posting: still 0d7c68d.
  • Actions 34004392334 on this head: Quality, Integration (production), Integration (api), Dependency security, Bundled agent contracts, and CodeQL succeeded. Integration (browser) was still pending at review time. A green production job does not close issue #22.

Inspected (not re-executed)

  • Unique 3-file diff vs 13a2e7a. Merge delta vs 01adb7c does not touch those files.
  • expectStatus / readBoundedBody / finishProductionServer / groupAlive / POSIX detached + stdio: pipe.
  • AGENTS.md + docs/threat-model.md: no auth, user-isolation, exact-acceptance, stale-write, migration, or workflow-permission changes. permissions: contents: read unchanged. Request headers and JWTs are not printed; JWT-shaped failure-body tokens are redacted.

These helpers still do not fix issue #22. Human approval remains mandatory.

Open in Web View Automation 

Sent by Cursor Automation: RELAY pull request review

@sdcarlson

Copy link
Copy Markdown
Contributor Author

Latest head0d7c68dd5dadc702fb72afc1a3faf92c4809d236 incorporates current main13a2e7a. The requested lint corrections are present: all nine node:test registrations are explicitly handled, and the child pid is narrowed before negation. Every required check now passes, including quality and production. No diagnostic source changes were introduced by the main synchronizations. Please re-review and approve this exact head; the old changes-requested review remains respected. #22 stays open.

@sykosyber sykosyber left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Partner Extra High review of 0d7c68dd5dadc702fb72afc1a3faf92c4809d236. Approving this SHA.

Independent of High Fast and of author comments. This supersedes Extra High REQUEST_CHANGES on 0416146 (oxlint). Extra High is not merging and is not closing #22.

Unique diff vs current main (13a2e7a) is still three files. Blob identity Extra High fetched:

  • scripts/ci/production-diagnostics.mjs 1fcbdad851bf120a981aba9ed5046b009dc98f33 — identical at reviewed helper 3d65c2d, at 7b1f4cc, and at this head.
  • scripts/ci/run-production.mjs 5f6e767bf3996a6858e99b553ab224288e3be1cd — identical at 7b1f4cc and this head.
  • tests/production-diagnostics.test.mjs 15254c2da3c21007a1637b77d76745b81e476042 — identical at 7b1f4cc and this head. Merge 0d7c68d only brings already-landed #41 into the branch; it does not touch those files.

Traced contracts:

  • Failure-only expectStatus (scripts/ci/run-production.mjs): success returns the unread response. Failure records operation, status, content-type/retry-after only, a 2s/2048-byte body, and pid/exit/signal, then asserts. Request headers and JWTs are not printed; JWT-shaped tokens in a failure body are redacted. Writes go to the existing outputs/ci/production-server.log artifact.
  • Request sequence is unchanged: spoofed header 401, authenticated document/asset, unauthenticated asset 401, bootstrap, owner-b 404, untrusted origin 403, owner-a save 200, stale 409, forged identity header ignored, exact accepted draft, no event on the rejected stale write.
  • waitForClose waits until stdout/stderr drain and exitCode/signalCode is set. signalGroup still SIGTERM/SIGKILL -pid on POSIX (taskkill /T on Windows). groupAlive (kill(-pid, 0) on POSIX) still KILLs a live group after those pipes end.
  • Fixture race Extra High confirmed on 85dfe36 (Actions 34003642709, log only grandchild=2693\n): process.on('SIGTERM') plus flushed after-term is registered before grandchild=.
  • oxlint Extra High confirmed on 0416146 (Actions 34004001791): all ten registrations are void test(...); every -pid SIGKILL is after typeof pid === 'number'.
  • No app/auth, schema, retry, health-poll, dependency, or workflow-permission change. .github/workflows/ci.yml stays permissions: contents: read. Threat-model private-data-in-logs: credentials are not in the failure dump.

Required checks on this head are green, including combined CI, Quality, Integration (production/api/browser), CodeQL, and dependency security: Actions 34004392334. Ancestor 01adb7c still showed the intermittent owner-a save 503 with a captured Wrangler proxy body; that is #22, not a helper regression. A green production job here does not close #22.

Keep #22 open. Extra High is not merging.

@sdcarlson
sdcarlson merged commit 9296d9d into main Sep 6, 2026
10 checks passed
@sdcarlson
sdcarlson deleted the cursor/issue22-production-diagnostics-41a2 branch September 6, 2026 01:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Diagnose intermittent 503 in compiled production integration test

4 participants