fix(frontend): harden exchange-token routing against transient control-plane failures

[igor-kupczynski]

Description

When the app runs with a control plane (Hatchet Cloud), every tenant-scoped API request is transparently authenticated by an axios interceptor: it fetches a short-lived exchange token from the control plane and rewrites the request's baseURL to the tenant shard's API. For users whose shard is in a different region than the control plane, those token/metadata calls are cross-region.

I've observed two failure paths which are worth fixing for a better UX:

1. The "is the control plane enabled?" metadata query caught all errors and returned null, which React Query cached as a successful "no control plane" answer. For the next 60s the interceptor skipped routing/auth entirely, so requests meant for the tenant shard went to the frontend's own origin — where those routes don't exist — producing 404s that are never retried (4xx).
2. Exchange-token fetch failures were swallowed (console.error only) and the request proceeded unauthenticated to the wrong host: same misleading 404.

On top of that, token renewal happened synchronously on the request path (blocking an interactive request on a cross-region round trip every few minutes), there was no client timeout below the browser's ~60s default, and a staleTime calculation bug made tokens go stale at half their intended lifetime, doubling renewal traffic.

Before / after

sequenceDiagram
    participant UI as Browser UI
    participant INT as axios interceptor
    participant RQ as React Query cache
    participant CP as Control plane
    participant SH as Tenant shard

    UI->>INT: tenant API request
    INT->>RQ: control plane enabled? exchange token?
    RQ->>CP: metadata / token fetch (cross-region)
    CP--xRQ: transient failure

    rect rgb(255, 235, 235)
        Note over UI,SH: BEFORE — silent fallbacks
        RQ-->>INT: error swallowed, "control plane disabled" cached 60s
        INT->>CP: request proceeds unrouted + unauthenticated (wrong host)
        CP-->>UI: 404 — a 4xx is never retried, page hard-fails until the cache expires
    end

    rect rgb(235, 245, 235)
        Note over UI,SH: AFTER — fail loudly, recover fast
        RQ-->>INT: last known good answer (or retry transient errors on first load)
        alt token fresh (renewed by 30s background timer)
            INT->>SH: routed to shard with Bearer token
            SH-->>UI: 200
        else control plane genuinely unreachable
            RQ-->>INT: bounded retries exhausted (15s timeout per attempt)
            INT--xUI: request rejected with typed error — no misroute, no retry storm
        end
    end

Loading

Type of change

• Bug fix (non-breaking change which fixes an issue)

What's Changed

More robust handling of control plane blips

• Metadata query: a failed fetch no longer counts as "control plane disabled"; keep the last known answer instead as this never changes.
• Token fetch failed → reject the request with a clear error. Before: silently sent it, unauthenticated, to the wrong host.

Improve retries

• 15s timeout on all control-plane calls (was: browser default, ~60s) (api.ts)
• Token fetch: retry network/5xx/404 errors, fail fast on other 4xx

Keep token renewal off the hot path

• Background timer renews tokens before they expire, so user requests never wait on a cross-region round trip (query-client.tsx)
• Bugfix: a staleTime math error expired tokens at half their lifetime → 2× the renewal traffic (exchange-token.ts)

Checklist

Changes have been:

• Tested (unit, integration, or manually with steps specified)
• Linted and formatted
• Documented (where applicable)
• Added to CHANGELOG (where applicable) -- see Keep a Changelog

---

🤖 AI Disclosure

• I acknowledge that an LLM was used in the creation of this Pull Request, in accordance with Hatchet's AI_POLICY.md.

• Details: Cursor agent (Claude) was used to investigate the incident, implement the fix, and run adversarial correctness and code-quality reviews. All changes were verified with typecheck, unit tests, and lint; final review by a human.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
hatchet-docs	Ready	Preview, Comment	Jul 21, 2026 4:44pm