Standalone smoke test depends on a live relaycast backend, and its error path logs raw SQL with bound params

[khaliqgant]

Summary

Two separate defects surfaced by the same CI failure on Package Validation / Standalone macOS Smoke today.

1. The standalone smoke test depends on a live relaycast backend, making a required check non-deterministic

scripts/ci-standalone-smoke.sh runs standalone up with no workspace key:

Workspace: none selected (no --workspace-key, no RELAY_WORKSPACE_KEY, no repository pin,
no active workspace in the machine-global store). A new workspace will be created.

Creating a workspace requires reaching the relaycast backend, so the test passes or fails on backend availability rather than on the code under test. Package Validation is a required check, so when the backend is unhealthy no PR in this repo can go green and the release workflow cannot complete.

Evidence it is non-deterministic rather than a code defect — two runs on the same commit of feat/relay-1387-delivery-observability, 63 seconds apart:

success   2026-08-17T14:47:32Z
failure   2026-08-17T14:48:35Z

Two different failure modes observed on the same run id (32040314356, PR #1551, head ef82fa59), on successive attempts:

Attempt 1 — handshake timeout:

[startup +5004ms]  handshake attempt 1/4 timed out after 5000ms; retrying in 250ms
[startup +10258ms] handshake attempt 2/4 timed out after 5000ms; retrying in 500ms
[startup +15765ms] handshake attempt 3/4 timed out after 5000ms; retrying in 1000ms
Error: relaycast startup handshake timed out after 4 attempt(s) of 5000ms each;
       the relay backend was unreachable or too slow to complete registration

Attempt 3 — workspace insert failure:

Error: failed to initialize relaycast session
Caused by:
Failed query: insert into "workspaces" (...) values (...)

Same dependency, two symptoms.

Worth deciding: should this smoke test provision against a live backend at all? A merge gate that is hostage to an external service's health reports "your code is broken" during someone else's outage. If the standalone lifecycle genuinely needs a workspace, a hermetic or pre-seeded one would keep the check meaningful.

2. The error path dumps raw SQL and bound parameters into the log

The attempt-3 failure printed the full INSERT statement together with its bound parameters, including the workspace api_key_hash:

Failed query: insert into "workspaces" ("id", "name", "api_key_hash", "system_prompt",
"plan", "created_at", "metadata", "retention") values (?, ?, ?, null, ?, (unixepoch()), ?, null)
returning ...
params: <workspace id>,<workspace name>,<api_key_hash>,free,{}

In this instance it is an ephemeral throwaway CI workspace, so the practical exposure is low. The defect is that the error boundary is missing, not that this particular value mattered — the same handler runs wherever workspace creation can fail, and a CI log is not a private surface.

The fix is to stop rendering the query and params in the surfaced error. Redacting only api_key_hash would relocate the problem rather than close it; an allowlist of safe diagnostic fields is the durable shape.

Impact

Blocked today: relay #1551, #1556, and the 11.6.11 publish (which failed on Verify Standalone (macOS) — same workflow, same path).

[khaliqgant]

Third attempt on the same run gave a third distinct error, which narrows this usefully.

32040314356 attempt 4:

=== Smoke: standalone up ===
Standalone broker did not report readiness within 30s
Standalone broker startup command did not become ready
Workspace: none selected (...). A new workspace will be created.
Failed to start broker: Unable to connect. Is the computer able to access the url?

So across three attempts on one commit:

attempt	failure
1	relaycast startup handshake timed out after 4 attempt(s) of 5000ms each
3	failed to initialize relaycast session → Failed query: insert into "workspaces" (...)
4	Unable to connect. Is the computer able to access the url?

The backend edge is not down. Probed https://cast.agentrelay.com/health from a laptop five consecutive times just now: 200 every time, 28–68ms. Control probe on a bogus path returns 404, so /health is a real route and the probe discriminates.

But /health returning 200 only proves the edge is serving. It does not observe the write plane, and attempt 3 is the interesting one precisely because it reached the server and the workspaces INSERT failed there — that is not a connectivity failure.

Worth investigating in that order:

1. Workspace-creation rate limiting or quota. Every Package Validation run on every branch creates a brand-new throwaway workspace. Under today’s CI load — many PRs, many re-runs — that is a lot of workspace creations from a narrow set of runner IPs. We have separately seen 429s on the relayfile ACL seed path and on action downloads today.
2. D1 write health on the workspaces table, given the INSERT failure is server-side.
3. Runner-to-edge connectivity, which attempts 1 and 4 are consistent with but attempt 3 argues against being the whole story.

Whatever the cause, the design point in the issue body stands and gets stronger: a required merge check that provisions against a live shared backend will keep converting someone else’s capacity problem into "your PR is broken."

[kjgbot]

Checkpoint — investigation started (2026-08-17 Europe/Oslo).

Environment:

• host: SF-Mac-Mini
• GitHub auth: kjgbot; scopes admin:public_key,gist,read:org,repo
• agent-relay: /Users/khaliqgant/.local/share/mise/shims/agent-relay, version 11.6.3

I am treating this as an intermittent write-plane failure, not re-testing the already-ruled-out hard-outage/edge-health hypotheses. Next evidence: read the issue/comment and current workflow logs per workflow, fetch current refs, identify creation limiters and D1 write-result handling in relayfile-cloud, then run a discriminating live workspace-create probe. No production limits/capacity changes and no merges.

[kjgbot]

Investigation checkpoint — write plane and production D1 evidence

Live write-plane probe (not /health)

• Single create: POST https://cast.agentrelay.com/v1/workspaces returned 201; the returned key then authenticated GET /v1/workspace at 200 with the same workspace id.
• Controlled burst: 32 creates in four waves of eight → 32/32 HTTP 201; 32/32 authenticated reads → HTTP 200 with matching ids; 32/32 cleanup deletes → HTTP 204. Maximum create latency was 4.444s.
• No creation response carried X-RateLimit-* headers.
• The two failed CI names both currently return the discriminating public-lookup result HTTP 404 workspace_not_found (not an empty/ambiguous 200).

Limiter finding

The production host is the AgentWorkforce/relaycast-cloud worker, deployed from main commit 8ca8e77 at 07:28Z. Its POST /v1/workspaces comes from @relaycast/engine; in both the marker-pinned 7.0.0 source and the repo dependency 8.0.4 source, that route has no rate-limit middleware. The only unauthenticated workspace limiter nearby is GET /v1/workspaces/by-name/:name, fixed at 30 requests/minute/IP. Therefore the repository-configured threshold/window for workspace creation is: none. The 32-create probe also did not surface an upstream/WAF 429.

Production D1 proof

I ran a read-only D1 query through a temporary branch workflow using the repo existing Cloudflare credential. It queried the two failed snowflake ids and the two failed names:

• ids: 215189678721458176, 215193820025860096
• names: relay-290fcdef, relay-f4149553
• result: query_success=true, rows=[], rows_read=41175, rows_written=0

Artifact: https://github.com/AgentWorkforce/relaycast-cloud/actions/runs/32070503852

The workflow intentionally exited 42 during pre-typecheck after printing this result. Deploy, secret-seed, and migration steps were skipped. No production state changed.

This rules out a workspace-name constraint, snowflake primary-key collision, and an INSERT that committed but lost its response. A daily write/storage quota also does not fit: another branch passed during the failure window, current writes pass, and such quota failures persist until reset/capacity change.

What remains / why the exact D1 subtype is absent

The hosted route catches the Drizzle error locally and calls errorResponse. That returns error.message to the client (hence raw SQL + bound params), but does not preserve/surface error.cause; because the route catches it, the global onError telemetry/logger is bypassed. Thus CI captured only Drizzle’s outer Failed query: insert into workspaces ... text, not the underlying D1 code (overloaded, reset, network loss, etc.). The evidence classifies this as a transient D1 write-plane failure, but current code destroys the subtype needed to distinguish overload from another retryable D1 reset/network error.

Cloudflare documents that each D1 database is single-threaded, excess concurrent requests queue and can return overloaded errors, and write queries are not automatically retried. This route performs an unretried workspace INSERT followed by a separate channel INSERT.

Next change I am shaping:

1. server-side safe diagnostics/telemetry that records an allowlist (operation, route, stable error category/request id) and retains the D1 cause internally;
2. never return raw database messages for 5xx responses;
3. bounded retry for known retryable D1 write errors on workspace creation, with idempotent handling so an ambiguous commit cannot mint a second workspace.

I will not change production limits/capacity and will not merge.

[kjgbot]

Environment correction/addendum:

• The mise shim I initially reported is indeed stale at 11.6.3.
• /Users/khaliqgant/.local/bin/agent-relay is newer but is 11.6.7, not the current npm release 11.7.0. /Users/khaliqgant/.agentworkforce/relay/bin/agent-relay is also 11.6.7. Any local standalone reproduction must therefore use an explicit 11.7.0 artifact rather than either installed binary.
• gh lacks the workflow scope. A direct workflow-file push was rejected. The production D1 read was instead run from a temporary branch-only pre-typecheck hook under the unchanged default Deploy workflow; it intentionally exited 42 before deployment. The temporary remote/local branch was deleted after evidence capture (commit 902ad0b; run artifact remains linked above).

Service ownership confirmed from live deploys rather than checkout naming: cast.agentrelay.com is owned by AgentWorkforce/relaycast-cloud production deploy 32005820944 at main 8ca8e77; AgentWorkforce/cloud only points consumers at that canonical host and separately deploys the legacy gateway.relaycast.dev worker.

[khaliqgant]

Fix checkpoint — canonical engine PR open

Draft PR: AgentWorkforce/relaycast#333
Commit: 1795e4e5f1c55a5a150580cb2c00d782c7a381f7

This implements the code fix at the canonical owner of POST /v1/workspaces:

• workspace + default #general channel are one D1 atomic batch instead of two separate writes;
• the batch reuses fixed generated IDs/key across attempts and uses idempotent conflict updates, so a commit whose response is lost replays to the same workspace rather than duplicating or leaving a partial row;
• only Cloudflare's classified transient D1 network/reset/remote-node/queue failures retry, up to five attempts with exponential backoff + jitter;
• D1 type/schema/constraint/quota errors do not retry;
• exhaustion records only operation, storage_error, and attempts through an explicit diagnostic allowlist;
• uncoded infrastructure 5xx bodies are now internal_error / Internal server error, so Drizzle SQL and bound parameters (including api_key_hash) cannot reach CI logs;
• application-authored coded errors retain their existing client messages.

Failure injection covers both sides of the ambiguity boundary:

1. transient overload before commit → second batch succeeds;
2. commit succeeds but response is lost → replay returns exactly one workspace + one general channel;
3. non-transient D1 error → one attempt only;
4. Drizzle-style outer error containing SQL/bound data → generic client body;
5. a would-be secret/query diagnostic field → excluded by the allowlist.

Validation on Node 22.23.2:

• full engine suite: 60 files / 593 tests passed
• focused suite after final retry-class change: 3 files / 12 tests passed
• engine typecheck: exit 0
• engine lint: exit 0
• git diff --check: exit 0

PR CI was read per workflow: CI run 32073167409 is currently in progress; Preview Deploy was skipped and no other branch workflow has a run.

No capacity, limiter, database, or production configuration was changed.

Important release boundary: merging this PR alone does not change cast.agentrelay.com. After review, the engine package must be published, then relaycast-cloud must bump the package and its handler source marker before deployment. The old failed requests destroyed the underlying D1 cause, so the historical subtype cannot be recovered; the safe storage_error category added here makes the next exhausted event distinguish network/reset/remote-node/queue without exposing SQL or parameters.

Cloudflare's current guidance confirms that writes are not automatically retried and recommends application-level exponential backoff with jitter for safe/idempotent operations: https://developers.cloudflare.com/d1/best-practices/retry-queries/

[khaliqgant]

The failure has stopped reproducing — recording this before it gets forgotten

Attempted to capture a production baseline this morning so the fix could be measured before-and-after. It is not reproducing.

Direct probes against POST /v1/workspaces on cast.agentrelay.com, 30 attempts:

• 10 sequential, 3s apart → 10/10 201, 0.73–2.57s
• 4 bursts of 5 concurrent → 20/20 201, 1.93–1.97s

Zero failures. No 429, no 503, no timeout. The design deliberately separated random failure from concurrency-triggered failure; it found neither.

CI corroborates on the exact path the probe does not cover — Package Validation across all branches:

success  2026-08-18T04:05Z  swift-sdk-relaycast-8
success  2026-08-17T21:39Z  fix/relay-1310-echo-verified-ack
success  2026-08-17T20:16Z  main
success  2026-08-17T19:45Z  main
success  2026-08-17T19:27Z  agent/relay-1553-fleet-agent-list
success  2026-08-17T17:07Z  main

Eight consecutive green runs. The last newly-created failing run was 32040314356 at 14:48Z on 2026-08-17 — the one re-run four times through 19:33Z. Nothing created since has failed.

What follows from this

The before/after measurement is unavailable. There is no live failure to compare against, so deploying relaycast#333 and observing green CI will prove nothing — CI has already been green for sixteen hours without it.

So #333 cannot be credited with fixing this incident. It can only be credited with fixing the defects it names, and those stand on their own merits regardless: Cloudflare does not retry writes, and the workspace and its default #general channel were genuinely two non-atomic writes. Worth fixing either way.

The possibility to hold open: the original cause may have been transient Cloudflare-side degradation that resolved itself. As noted earlier in this issue, the failed requests destroyed the underlying D1 cause, so the historical subtype is unrecoverable and we cannot retroactively distinguish that from an application-level defect.

Practical consequence for the PR: #333 should be reviewed as a robustness change standing on its own tests, not as a validated incident fix. Its local test coverage is now the only evidence available — which makes the open review findings on it more important, not less, particularly the one showing the atomicity test injects its failure before BEGIN IMMEDIATE and so never exercises mid-batch rollback.

What would still be worth building: the storage_error diagnostic category added in #333 means the next exhausted event will be classifiable without exposing SQL or parameters. That is the instrument this issue lacked. If this recurs, we will be able to name the subtype instead of guessing.

Leaving this issue open — the flake stopping on its own is not the same as it being understood.

[khaliqgant]

Correction: the D1 failure has NOT stopped. It moved to a different query.

Earlier today I reported this as no longer reproducing, on the basis of 30 clean POST /v1/workspaces probes and eight consecutive green Package Validation runs. That was true of the workspaces INSERT and false of the underlying fault. I measured one query and generalised to the infrastructure.

Live evidence from sf-mini's node stderr log, last written 07:18Z today:

[agent-relay] node provider: serving .../sf-mini-inert-provider.ts failed:
  NodeRegistrationError: Failed query: select "status" from "nodes"
  where ("nodes"."workspace_id" = ? and "nodes"."id" = ?)

Failed to start broker: Fleet node provider ... exited unexpectedly (code 2).

Same signature as the original report — a Drizzle query against D1 failing, surfaced with the raw SQL — on a different table. So the fault is current and observable right now, on nodes rather than workspaces.

Three consequences

1. relaycast#333 is better justified than I said, not worse. I had downgraded it to "two real robustness defects, fixed, but we can never credit it with the incident." The D1 flakiness is live, which restores the link between the fix and an observable failure.

2. But #333 is too narrow. It hardens workspace.create specifically — atomic batch, bounded retry on classified transient errors, error-boundary on the response. select "status" from "nodes" runs on the same infrastructure with no retry and no error boundary at all, and it is failing today. The retry/classify/redact machinery added in d1Retry.ts and httpError.ts is generic; the question worth answering before merge is which other call sites need it, rather than fixing them one incident at a time.

3. The raw-SQL disclosure is broader than the workspaces path. This log line puts a full select ... where (...) into a node's stderr on an unrelated code path. The allowlist added in #333 covers the engine's HTTP error rendering; it does not cover this.

A separate defect this exposes

The parent agent-relay node up process stays alive and keeps heartbeating while the provider child crash-loops and exits code 2. The control plane therefore reports the node status: online, live: true, handlersLive: true — while nothing can actually be placed on it. Any spawn pinned to that node fails; unpinned spawns route elsewhere and succeed, which makes the failure look intermittent and node-specific rather than what it is.

handlersLive: true should not be reported when the handler process has exited. Filed separately in relay#1563 territory (spawn observability), but noting it here because it is what made this fault hard to see.

Method note

The probe design that produced the false all-clear was sound for what it tested — sequential and concurrent arms, discriminating controls — but it only exercised POST /v1/workspaces. A single-endpoint probe cannot clear a shared dependency. Any future "is it fixed" check on D1 should sample more than one query against more than one table, or it will keep producing confident negatives.