Skip to content

Commit 9b10389

Browse files
authored
fix(release): keyless subject uniqueness, NO_PROXY IPv6/CIDR, proxied boot secret, and remaining audit follow-ups (#9866)
* docs(api-keys): add rollback guidance to the api-key-read-only flag * docs(api): say v2 workflow execution needs a full-access API key * chore(api): regenerate MCP operations for the execute scope wording * fix(hitl): stop duplicating input fields across listed pause points * fix(api): refuse a personal v2 key with a missing owner as invalid, not a 500 * fix(credentials): refuse a keyless subject another live credential already holds A keyless subject proof stays valid for 24 hours and is not bound to one credential, so a member removed from a keyless AWS or Databricks credential could save a second credential with the same subject and proof and keep assuming the customer's role. Creating or reconnecting a credential onto a subject it did not already hold now refuses (409 conflict) when another non-revoked credential of the same owner holds it. The subject lives only in the encrypted secret, so the check decrypts the owner's keyless credentials inside the write transaction, under a per-subject advisory lock so concurrent saves serialize. A reconnect keeping its stored subject binds nothing new and skips the check. * refactor(tools): detect the saved-credential picker with isSavedCredentialPicker credentialSuppliedParamsFor matched any oauth-input feeding oauthCredential, without the credentialKind === 'service-account' check isSavedCredentialPicker applies. It now uses that predicate. No registered block changes: every block whose loose-match picker lacks the service-account kind has no tool that declares credentialSuppliedParams. * fix(runtime-secrets): fetch the boot secret through the configured proxy The Secrets Manager client used the SDK's Node handler, which ignores HTTPS_PROXY, so a container whose only egress is a proxy could not load its env and never booted. Under Bun with a proxy set, send through Bun's fetch, which honors HTTPS_PROXY/HTTP_PROXY/NO_PROXY. apps/sim's awsRequestHandler is out of reach for packages and the realtime bootstrap needs the same path. * fix(models): mark NVIDIA Nemotron 3 Super retired and move the NVIDIA default NVIDIA's model page states the nvidia/nemotron-3-super-120b-a12b API is no longer supported after 2026-10-19, and Pi 1.0.1 dropped it as its NVIDIA default for the same reason. Mark it deprecated (hidden from pickers, still a known id for pinned workflows) and default the NVIDIA provider to Nemotron 3 Ultra, which the Pi catalog also serves. The Mistral IDs Pi 1.5 dropped (mistral-large-2411, devstral-small-2507, devstral-medium-2507) were retired by Mistral on 2026-05-31 and are already deprecated here, so saved Pi blocks fail before credentials with the choose-a-model error instead of being mapped onto a different model. * fix(credentials): lock keyless subject checks through the advisory-lock helper and refuse on unreadable siblings assertWorkloadIdentitySubjectUnclaimed now takes a DbTransaction and locks through acquireAdvisoryXactLock, so the lock carries its SQLCommenter tag and cannot be taken on the pool by mistake. A sibling keyless credential whose secret does not decrypt or parse used to throw a generic error, failing every keyless create or rotation for the owner with a 500. Since it might hold the subject, the save is now refused as a conflict naming that credential, with a warning logged, and stored subjects are read with a schema safeParse. * fix(network): match NO_PROXY entries like curl, including bare IPv6 and CIDR A bare IPv6 entry such as ::1 was read as host ':' port 1 and never matched, IP entries only matched their exact spelling, and CIDR ranges were ignored. Entries now follow curl: domains cover their subdomains, bracketed IPv6 may carry a port, IP entries match any spelling, and CIDR ranges match IP-literal destinations. NO_PROXY is parsed once per distinct value. The comments and the self-hosting networking page describe these rules, and note that Bun's default fetch applies its own. * docs(network): note the boot secret fetch follows Bun's NO_PROXY rules * fix(credentials): decide subject reuse under the lock on every keyless reconnect A reconnect that kept its subject skipped the uniqueness check, but whether it kept the subject was decided from the secret read before provider verification. If another reconnect moved the credential to a new subject and a second credential claimed the old one meanwhile, the stale reconnect wrote the old subject back unchecked and two credentials held it. Every keyless reconnect now locks the subject it is writing, then the credential row (FOR UPDATE), re-reads the row's current subject, and runs the sibling check whenever the two differ, with the decision and the write in one transaction. * fix(runtime-secrets): pick the boot fetch's proxy with the app's rules Bun's fetch picks a proxy by scheme only, so with just HTTP_PROXY set the https Secrets Manager request went direct and a proxy-only container still could not boot. Move proxy selection (HTTPS_PROXY falling back to HTTP_PROXY, curl-style NO_PROXY) from apps/sim into @sim/security/environment-proxy, and have runtime-secrets pass the selected proxy to Bun's fetch explicitly, or send through the Node handler when NO_PROXY excludes the endpoint. * docs(api): state the credential and role a manual v2 run requires Manual runs need api:write, which read-only personal keys lack, workspace write access, and a personal key or OAuth token; workspace API keys are refused. Regenerate the OpenAPI document and the MCP operation catalog.
1 parent 4266ef6 commit 9b10389

29 files changed

Lines changed: 841 additions & 95 deletions

‎apps/docs/content/docs/platform/self-hosting/networking.mdx‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -301,6 +301,8 @@ A request through the SSRF guard keeps its checks when it is proxied. Before eac
301301
302302
Set `NO_PROXY` for every destination that is not on the public internet, not just model endpoints. The app reaches the realtime server (`SOCKET_SERVER_URL`), the Presidio PII service (`PII_URL`), and itself (`INTERNAL_API_BASE_URL`) over the same default `fetch`, alongside self-hosted Ollama, LiteLLM, and vLLM — so a proxy that cannot reach your internal network breaks live updates and PII redaction, not only inference. AWS clients and OTLP export follow the same rule, so also add an internal S3-compatible store such as MinIO, AWS VPC endpoints, and an in-cluster OpenTelemetry collector.
303303
304+
For the SSRF guard, AWS clients (including the boot-time Secrets Manager fetch), and OTLP export, Sim matches `NO_PROXY` entries the way curl does: `example.com`, `.example.com`, and `*.example.com` each cover the domain and all its subdomains; `host:port` and `[::1]:port` cover one port; IPv6 entries work bare (`::1`) or bracketed; and CIDR ranges such as `10.0.0.0/8` cover IP-literal destinations only, not hostnames that resolve into the range. Bun's default `fetch`, which carries internal service URLs such as the realtime server and PII service, applies Bun's own `NO_PROXY` rules, which may not honor CIDR or bare IPv6 entries, so list those destinations by hostname.
305+
304306
Set it in the application environment, not your shell — under `app.env` on Helm, or the service's `environment:` on Compose. On Helm the suffixes alone are not enough: the chart wires `SOCKET_SERVER_URL`, `PII_URL`, and `OLLAMA_URL` to bare Service names, which no domain suffix matches. Add those names too — `helm template` prints the rendered ones, and the prefix is the release name unless it already contains `sim`, in which case it is the release name alone:
305307
306308
```yaml

‎apps/docs/openapi-v2-workflows.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3958,7 +3958,7 @@
39583958
"post": {
39593959
"operationId": "executeWorkflowV2",
39603960
"summary": "Execute workflow",
3961-
"description": "Run the active deployment, set `run: { source: \"deployment\", version }` to run an earlier deployment version, or set `run.source: \"manual\"` to run the saved draft. Deployed execution requires workspace read access and OAuth `api:write` scope or an API key; public deployments also allow anonymous synchronous or streamed runs. Running a chosen version requires workspace write access and an API key or OAuth access token; an unknown version returns `404`. Manual runs require personal-key or OAuth write access and cannot be async.\n\nUse `async: true` for a queue receipt, or `stream: true` for SSE. For a synchronous run with both flags false, send `Accept: application/x-ndjson` for heartbeat frames and a final frame containing `data`. With `waitTimeoutSeconds`, a synchronous run still going when the wait elapses continues in the background and answers with its run receipt instead: `202` on a JSON response, or a closing `detached` frame on an NDJSON stream, which keeps its `200`. Timeouts return `200` with `status: \"failed\"` and `error.code: \"TIMEOUT\"`; inspect run status. See [Streaming](/workflows/deployment/api#streaming) for event formats. Input descriptions specify compatible modes; invalid combinations return `400`.\n\nSupply `X-Run-Id` to prevent duplicate execution. Reuse returns `409`, without replaying results. A `503` with `error.details.code: \"ASYNC_ENQUEUE_AMBIGUOUS\"` omits `Retry-After` because the run may already have started. Reconcile using the returned run ID; do not start another run with a new or omitted ID.\n\nOAuth scope: `api:write`.",
3961+
"description": "Run the active deployment, set `run: { source: \"deployment\", version }` to run an earlier deployment version, or set `run.source: \"manual\"` to run the saved draft. Deployed execution requires workspace read access and OAuth `api:write` scope or a full-access API key; public deployments also allow anonymous synchronous or streamed runs. Running a chosen version requires workspace write access and a full-access API key or OAuth `api:write` scope; an unknown version returns `404`. Manual runs require a full-access personal API key or OAuth `api:write` scope, and workspace write access; workspace API keys are refused, and manual runs cannot be async.\n\nUse `async: true` for a queue receipt, or `stream: true` for SSE. For a synchronous run with both flags false, send `Accept: application/x-ndjson` for heartbeat frames and a final frame containing `data`. With `waitTimeoutSeconds`, a synchronous run still going when the wait elapses continues in the background and answers with its run receipt instead: `202` on a JSON response, or a closing `detached` frame on an NDJSON stream, which keeps its `200`. Timeouts return `200` with `status: \"failed\"` and `error.code: \"TIMEOUT\"`; inspect run status. See [Streaming](/workflows/deployment/api#streaming) for event formats. Input descriptions specify compatible modes; invalid combinations return `400`.\n\nSupply `X-Run-Id` to prevent duplicate execution. Reuse returns `409`, without replaying results. A `503` with `error.details.code: \"ASYNC_ENQUEUE_AMBIGUOUS\"` omits `Retry-After` because the run may already have started. Reconcile using the returned run ID; do not start another run with a new or omitted ID.\n\nOAuth scope: `api:write`.",
39623962
"x-sim-operation": "workflows.execute",
39633963
"x-oauth-scope": "api:write",
39643964
"tags": ["Workflows"],

‎apps/sim/lib/api/contracts/v2/openapi/workflows.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1282,7 +1282,7 @@ const declaredRoutes = [
12821282
applicationOperation: workflowOperations.execute,
12831283
operationId: 'executeWorkflowV2',
12841284
summary: 'Execute workflow',
1285-
description: `Run the active deployment, set \`run: { source: "deployment", version }\` to run an earlier deployment version, or set \`run.source: "manual"\` to run the saved draft. Deployed execution requires workspace read access and OAuth \`api:write\` scope or an API key; public deployments also allow anonymous synchronous or streamed runs. Running a chosen version requires workspace write access and an API key or OAuth access token; an unknown version returns \`404\`. Manual runs require personal-key or OAuth write access and cannot be async.\n\nUse \`async: true\` for a queue receipt, or \`stream: true\` for SSE. For a synchronous run with both flags false, send \`Accept: application/x-ndjson\` for heartbeat frames and a final frame containing \`data\`. With \`waitTimeoutSeconds\`, a synchronous run still going when the wait elapses continues in the background and answers with its run receipt instead: \`202\` on a JSON response, or a closing \`detached\` frame on an NDJSON stream, which keeps its \`200\`. Timeouts return \`200\` with \`status: "failed"\` and \`error.code: "TIMEOUT"\`; inspect run status. See [Streaming](/workflows/deployment/api#streaming) for event formats. ${EXECUTE_OPTION_CONSTRAINTS}\n\nSupply \`X-Run-Id\` to prevent duplicate execution. Reuse returns \`409\`, without replaying results. A \`503\` with \`error.details.code: "ASYNC_ENQUEUE_AMBIGUOUS"\` omits \`Retry-After\` because the run may already have started. Reconcile using the returned run ID; do not start another run with a new or omitted ID.`,
1285+
description: `Run the active deployment, set \`run: { source: "deployment", version }\` to run an earlier deployment version, or set \`run.source: "manual"\` to run the saved draft. Deployed execution requires workspace read access and OAuth \`api:write\` scope or a full-access API key; public deployments also allow anonymous synchronous or streamed runs. Running a chosen version requires workspace write access and a full-access API key or OAuth \`api:write\` scope; an unknown version returns \`404\`. Manual runs require a full-access personal API key or OAuth \`api:write\` scope, and workspace write access; workspace API keys are refused, and manual runs cannot be async.\n\nUse \`async: true\` for a queue receipt, or \`stream: true\` for SSE. For a synchronous run with both flags false, send \`Accept: application/x-ndjson\` for heartbeat frames and a final frame containing \`data\`. With \`waitTimeoutSeconds\`, a synchronous run still going when the wait elapses continues in the background and answers with its run receipt instead: \`202\` on a JSON response, or a closing \`detached\` frame on an NDJSON stream, which keeps its \`200\`. Timeouts return \`200\` with \`status: "failed"\` and \`error.code: "TIMEOUT"\`; inspect run status. See [Streaming](/workflows/deployment/api#streaming) for event formats. ${EXECUTE_OPTION_CONSTRAINTS}\n\nSupply \`X-Run-Id\` to prevent duplicate execution. Reuse returns \`409\`, without replaying results. A \`503\` with \`error.details.code: "ASYNC_ENQUEUE_AMBIGUOUS"\` omits \`Retry-After\` because the run may already have started. Reconcile using the returned run ID; do not start another run with a new or omitted ID.`,
12861286
errors: [
12871287
'BadRequest',
12881288
'Unauthorized',

‎apps/sim/lib/api/mcp/generated/v2-operations.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1252,7 +1252,7 @@ export const V2_MCP_OPERATIONS = {
12521252
contract: v2ExecuteWorkflowContract,
12531253
summary: 'Execute workflow',
12541254
description:
1255-
'Run the active deployment, set `run: { source: "deployment", version }` to run an earlier deployment version, or set `run.source: "manual"` to run the saved draft. Deployed execution requires workspace read access and OAuth `api:write` scope or an API key; public deployments also allow anonymous synchronous or streamed runs. Running a chosen version requires workspace write access and an API key or OAuth access token; an unknown version returns `404`. Manual runs require personal-key or OAuth write access and cannot be async.\n\nUse `async: true` for a queue receipt, or `stream: true` for SSE. For a synchronous run with both flags false, send `Accept: application/x-ndjson` for heartbeat frames and a final frame containing `data`. With `waitTimeoutSeconds`, a synchronous run still going when the wait elapses continues in the background and answers with its run receipt instead: `202` on a JSON response, or a closing `detached` frame on an NDJSON stream, which keeps its `200`. Timeouts return `200` with `status: "failed"` and `error.code: "TIMEOUT"`; inspect run status. See [Streaming](/workflows/deployment/api#streaming) for event formats. Input descriptions specify compatible modes; invalid combinations return `400`.\n\nSupply `X-Run-Id` to prevent duplicate execution. Reuse returns `409`, without replaying results. A `503` with `error.details.code: "ASYNC_ENQUEUE_AMBIGUOUS"` omits `Retry-After` because the run may already have started. Reconcile using the returned run ID; do not start another run with a new or omitted ID.\n\nOAuth scope: `api:write`.',
1255+
'Run the active deployment, set `run: { source: "deployment", version }` to run an earlier deployment version, or set `run.source: "manual"` to run the saved draft. Deployed execution requires workspace read access and OAuth `api:write` scope or a full-access API key; public deployments also allow anonymous synchronous or streamed runs. Running a chosen version requires workspace write access and a full-access API key or OAuth `api:write` scope; an unknown version returns `404`. Manual runs require a full-access personal API key or OAuth `api:write` scope, and workspace write access; workspace API keys are refused, and manual runs cannot be async.\n\nUse `async: true` for a queue receipt, or `stream: true` for SSE. For a synchronous run with both flags false, send `Accept: application/x-ndjson` for heartbeat frames and a final frame containing `data`. With `waitTimeoutSeconds`, a synchronous run still going when the wait elapses continues in the background and answers with its run receipt instead: `202` on a JSON response, or a closing `detached` frame on an NDJSON stream, which keeps its `200`. Timeouts return `200` with `status: "failed"` and `error.code: "TIMEOUT"`; inspect run status. See [Streaming](/workflows/deployment/api#streaming) for event formats. Input descriptions specify compatible modes; invalid combinations return `400`.\n\nSupply `X-Run-Id` to prevent duplicate execution. Reuse returns `409`, without replaying results. A `503` with `error.details.code: "ASYNC_ENQUEUE_AMBIGUOUS"` omits `Retry-After` because the run may already have started. Reconcile using the returned run ID; do not start another run with a new or omitted ID.\n\nOAuth scope: `api:write`.',
12561256
handler: () =>
12571257
import('@/app/api/v2/workflows/[workflowId]/execute/route').then((route) => route.POST),
12581258
},

‎apps/sim/lib/api/server/routes/v2-api-key-auth.test.ts‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -199,6 +199,28 @@ describe('v2 API key authentication', () => {
199199
)
200200
})
201201

202+
it('refuses a personal key whose owner row is gone as invalid, expired or not', async () => {
203+
for (const expiresAt of [null, new Date(Date.now() - 1)]) {
204+
queueTableRows(schemaMock.apiKey, [
205+
{
206+
id: 'key-orphan',
207+
userId: 'user-gone',
208+
workspaceId: null,
209+
type: 'personal',
210+
access: 'write',
211+
expiresAt,
212+
userBanned: null,
213+
userSuspendedAt: null,
214+
},
215+
])
216+
const refusal = await authenticateV2ApiKey({ apiKey: 'orphan', bearer: null }).catch(
217+
(error: unknown) => error
218+
)
219+
expect(refusal).toBeInstanceOf(V2ApiKeyUnauthenticatedError)
220+
expect((refusal as V2ApiKeyUnauthenticatedError).message).toBe('Invalid API key')
221+
}
222+
})
223+
202224
it('propagates auth-store failures instead of converting them to invalid credentials', async () => {
203225
const failure = new Error('database unavailable')
204226
dbChainMockFns.limit.mockRejectedValueOnce(failure)

‎apps/sim/lib/api/server/routes/v2-api-key-auth.ts‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -85,7 +85,8 @@ function requireValidRow(row: ApiKeyRow | undefined): ApiKeyRow {
8585
if (!row) throw new V2ApiKeyUnauthenticatedError()
8686
if (row.type === 'personal' && row.workspaceId === null) {
8787
if (row.userBanned === null) {
88-
throw new Error(`Personal API key ${row.id} is missing its credential owner`)
88+
logger.error('Personal API key is missing its credential owner', { keyId: row.id })
89+
throw new V2ApiKeyUnauthenticatedError()
8990
}
9091
if (row.userBanned) throw new V2ApiKeyUnauthenticatedError()
9192
/**

‎apps/sim/lib/core/config/feature-flags.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -79,7 +79,7 @@ const FEATURE_FLAGS = {
7979
},
8080
'api-key-read-only': {
8181
description:
82-
'Allow creating read-only API keys. Turn on only after every pod runs a release that enforces key access, since older pods treat a read-only key as full access. Enforcement of existing read-only keys never depends on this flag. Global; disabled by default.',
82+
'Allow creating read-only API keys. Turn on only after every pod runs a release that enforces key access, since older pods treat a read-only key as full access. Before rolling back below the release that added read-only keys, turn this off first: that older code also serves every existing read-only key as full access. Enforcement of existing read-only keys never depends on this flag. Global; disabled by default.',
8383
fallback: 'API_KEY_READ_ONLY',
8484
},
8585
'mothership-model-selector': {

‎apps/sim/lib/core/network/environment-proxy.server.test.ts‎

Lines changed: 61 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,7 @@ vi.mock('@sim/security/dns', async (importOriginal) => {
3232
}
3333
})
3434

35-
import { resolveEnvironmentProxy } from '@/lib/core/network/environment-proxy.server'
35+
import { resolveEnvironmentProxy } from '@sim/security/environment-proxy'
3636
import { awsRequestHandler } from '@/lib/core/network/proxy-agent.server'
3737
import {
3838
createPinnedFetchWithDispatcher,
@@ -341,6 +341,66 @@ describe('proxy selection from the environment', () => {
341341
target: 'https://api.example.com/',
342342
proxy: 'http://p:3128/',
343343
},
344+
{
345+
case: 'a bare NO_PROXY domain also excludes its subdomains',
346+
env: { HTTPS_PROXY: 'http://p:3128', NO_PROXY: 'example.com' },
347+
target: 'https://api.example.com/',
348+
proxy: null,
349+
},
350+
{
351+
case: 'a leading-dot NO_PROXY domain also excludes the apex',
352+
env: { HTTPS_PROXY: 'http://p:3128', NO_PROXY: '.example.com' },
353+
target: 'https://example.com/',
354+
proxy: null,
355+
},
356+
{
357+
case: 'a NO_PROXY domain does not match a longer name that merely ends with it',
358+
env: { HTTPS_PROXY: 'http://p:3128', NO_PROXY: 'example.com' },
359+
target: 'https://notexample.com/',
360+
proxy: 'http://p:3128/',
361+
},
362+
{
363+
case: 'a bare IPv6 NO_PROXY entry excludes that address',
364+
env: { HTTP_PROXY: 'http://p:3128', NO_PROXY: 'localhost,::1' },
365+
target: 'http://[::1]:3000/',
366+
proxy: null,
367+
},
368+
{
369+
case: 'an IPv6 NO_PROXY entry matches any spelling of the address',
370+
env: { HTTP_PROXY: 'http://p:3128', NO_PROXY: '0:0:0:0:0:0:0:1' },
371+
target: 'http://[::1]/',
372+
proxy: null,
373+
},
374+
{
375+
case: 'a bracketed IPv6 NO_PROXY entry with a port only excludes that port',
376+
env: { HTTP_PROXY: 'http://p:3128', NO_PROXY: '[::1]:3000' },
377+
target: 'http://[::1]:4000/',
378+
proxy: 'http://p:3128/',
379+
},
380+
{
381+
case: 'an IPv4 CIDR NO_PROXY entry excludes addresses inside the range',
382+
env: { HTTPS_PROXY: 'http://p:3128', NO_PROXY: '10.0.0.0/8' },
383+
target: 'https://10.20.30.40/',
384+
proxy: null,
385+
},
386+
{
387+
case: 'an IPv4 CIDR NO_PROXY entry leaves addresses outside the range proxied',
388+
env: { HTTPS_PROXY: 'http://p:3128', NO_PROXY: '10.0.0.0/8' },
389+
target: 'https://11.0.0.1/',
390+
proxy: 'http://p:3128/',
391+
},
392+
{
393+
case: 'an IPv6 CIDR NO_PROXY entry excludes addresses inside the range',
394+
env: { HTTP_PROXY: 'http://p:3128', NO_PROXY: 'fd00::/8' },
395+
target: 'http://[fd12::1]/',
396+
proxy: null,
397+
},
398+
{
399+
case: 'an empty lowercase no_proxy does not hide the uppercase one',
400+
env: { HTTPS_PROXY: 'http://p:3128', no_proxy: '', NO_PROXY: 'example.com' },
401+
target: 'https://example.com/',
402+
proxy: null,
403+
},
344404
{
345405
case: 'NO_PROXY=* excludes every destination',
346406
env: { HTTPS_PROXY: 'http://p:3128', NO_PROXY: '*' },

0 commit comments

Comments
 (0)