strix openai-direct fallback 404s at runtime despite gpt-5.4 contract (blocks macos_utility_packs#3)

[seonghobae]

Evidence

Run 32815579699 attempt 2 (macos_utility_packs, head 8348e4f, trusted strix @8fd471a):

1. Primary NIM nemotron-3-super-120b-a12b → provider failure-signal.
2. Fallback openai-direct/gpt-5.4 → LLM CONNECTION FAILED · Error: 404 page not found after 4s.
3. An inner pass had already reported Vulnerabilities 0, but no report artifact was produced → gate correctly fails closed.

Hypotheses (ordered)

1. openai_direct/ is not a recognized strix provider slug; the runner maps openai-direct/gpt-5.4→openai_direct/gpt-5.4 (.github/workflows/strix.yml:784-788) and the unknown-provider default base URL 404s.
2. Base URL for direct OpenAI missing /v1 segment when constructed from STRIX_OPENAI_API_BASE_FILE.
3. gpt-5.4 not exposed on the direct OpenAI API for this key tier (model-id mismatch vs gpt-5.4-turbo style alias).

Ask

Fix provider slug/base-url routing so the contracted final fallback completes; then re-run Strix Security Scan on macos_utility_packs#3 head 8348e4f. Related: #1317, #1318.

[seonghobae]

Root cause confirmed and fixed in #1326: resolved_llm_api_base_for_model had no openai-direct branch, so nvidia_nim-primary runs passed LLM_API_BASE=https://integrate.api.nvidia.com/v1 to the openai-direct/gpt-5.4 fallback attempt. litellm posted openai/gpt-5.4 to NVIDIA's inference endpoint → Go-style '404 page not found' → STRIX_PROVIDER_UNAVAILABLE. The fix returns no custom base for explicit direct-OpenAI models so litellm routes to api.openai.com with the OpenAI fallback key (key selection already existed). Awaiting Checks on #1326 head; macos_utility_packs#3 strix should pass on the next scheduler rerun after merge.

[seonghobae]

Fresh cross-product reproduction on 2026-08-25 confirms this is still a central Strix provider/fallback boundary, not a leaf-code finding.

ContextualWisdomLab/pg-erd-cloud#704 exact head 2c4f891bd71ed74c5a7cff8e511c3b2d746a667a has every other live required context terminal-success, but required Strix run 32830460915 / job 97747751275 failed on trusted Strix 8fd471a31399a914d9cb22a840f4a4c68e010ea6:

• primary nvidia_nim/nvidia/nemotron-3-super-120b-a12b repeatedly hit HTTP 429;
• the NVIDIA 49B fallback completed a zero-vulnerability penetration-test summary, but provider-failure evidence from the attempt kept the fail-closed gate non-passing;
• final openai-direct/gpt-5.4 then failed with the same 404 page not found signature documented here.

I re-ran only the failed Strix job against the unchanged leaf head; the replacement job is 97852761442 and is currently queued, so it is not passing evidence. No pg-erd-cloud source change was made for this control-plane/provider failure.

This gives #1326 a second independent consumer canary for the resolved_llm_api_base_for_model repair. After #1326 integrates normally, re-run Strix on the then-unchanged leaf head and require terminal success before treating the leaf as merge-ready.

[seonghobae]

Fresh downstream canary from ContextualWisdomLab/OriginWeave#124 confirms the same central Strix fallback defect on an exact-current leaf head and adds one distinct transient/tool-boundary symptom that should be preserved during the owner fix.

• OriginWeave PR docs: record aFIPC required workflow coverage gap #124 exact head: 6dd2023ca2f9fd5003564d1603565c05f4d9f111
• protected live base: b05d5acca82b9d916ada2c8e82f59f92a89817e1
• required Strix run/job: 32884789734 / 97922586136
• Strix report artifact: 9577973496, digest sha256:2d0134a7ec1e8f474af480b98f6cac46921740f53850daa31528871a9ef389cf
• native CI 32884793521, SAST 32884793442, and Security Scan 32884793441 all completed successfully on the same head.

The Strix cascade was:

1. primary nvidia_nim/nvidia/nemotron-3-super-120b-a12b repeatedly returned HTTP 429 and exhausted its bounded retry budget;
2. the NVIDIA 49B fallback later hit loginAsGuest failed after 10 attempts / curl exit 7 against the local 127.0.0.1:48080 proxy boundary;
3. final openai-direct/gpt-5.4 reproduced the same 404 page not found signature already traced here to resolved_llm_api_base_for_model routing.

No OriginWeave source workaround is appropriate: this is the organization-required Strix provider/runtime boundary and the leaf gate correctly remains fail-closed. Please use this exact-head canary when validating #1326: preserve bounded 429 handling and fail-closed semantics, ensure the direct-OpenAI fallback no longer inherits the NVIDIA API base, and separately classify the local proxy-readiness failure rather than masking it as a clean scan. After the central fix integrates normally, rerun Strix against the then-unchanged OriginWeave #124 head; only terminal exact-head evidence should count.

[seonghobae]

Fresh ScopeWeave canary confirms the same central Strix provider/fallback defect on an exact PR head; no ScopeWeave source workaround is appropriate.

• leaf PR: ContextualWisdomLab/scopeweave#602
• exact leaf head: 5f9dc371fb3ba7150a8f1e3693bc64794e50ebee
• live protected base observed: 2c328875e00e86537df3e965170be80532571cad
• required Strix run/job: 32819824316 / 97715410727
• trusted .github Strix source: 8fd471a31399a914d9cb22a840f4a4c68e010ea6
• the trusted workflow correctly fetched and scanned PR head 5f9dc371fb3ba7150a8f1e3693bc64794e50ebee; this is not a synthetic-checkout failure.

Observed cascade:

1. primary nvidia_nim/nvidia/nemotron-3-super-120b-a12b exhausted three bounded attempts with HTTP 429 Too Many Requests;
2. fallback nvidia_nim/nvidia/llama-3.3-nemotron-super-49b-v1.5 eventually completed a Vulnerabilities 0 run, but its degraded narrative incorrectly claimed server/server.mjs was absent and full execution was impossible, so the fail-closed gate correctly refused to promote that incomplete/provider-contaminated output;
3. final openai-direct/gpt-5.4 reproduced LLM CONNECTION FAILED · Error: 404 page not found after ~5 seconds;
4. no authoritative structured vulnerability report completed, so the required check correctly ended STRIX_PROVIDER_UNAVAILABLE/failure rather than false-green.

Use this as another exact-head acceptance canary for the existing owner repair. RED should preserve the NIM-primary→direct-OpenAI fallback chain with a non-OpenAI primary base and assert that the direct-OpenAI attempt cannot inherit https://integrate.api.nvidia.com/v1. GREEN requires a valid direct-OpenAI route/key/base, a completed authoritative structured report on the unchanged leaf head, no provider-failure markers, and terminal required-check success without weakening fail-closed behavior. Also retain a fixture proving that a model-produced 0 vulnerabilities summary which simultaneously reports inability to execute the target is not merge evidence.

[seonghobae]

Fresh ScopeWeave consumer evidence adds a distinct direct-OpenAI fallback contract failure that is not covered by the existing NVIDIA-base 404 diagnosis, so the central owner path needs to preserve both regressions.

Consumer tuple:

• ContextualWisdomLab/scopeweave#610
• exact head fb381e86cdb0c6bcc6397a278070cb26e964a4d6
• protected live base 2c328875e00e86537df3e965170be80532571cad
• required Strix run/job 32976934806 / 98203848191

The trusted workflow validated the exact target/base/head and entered Strix normally. The primary nvidia_nim/nvidia/nemotron-3-super-120b-a12b attempt then hit HTTP 429. The final direct fallback was resolved as openai_direct/gpt-5.4, but failed before authoritative analysis with a LiteLLM/OpenAI BadRequestError: the configured reasoning_effort was rejected because the runtime classified the fallback as not a GPT-5-series model. No authoritative Strix report was produced, so the required gate correctly remained fail-closed. There is no ScopeWeave-local causal repair for this provider/runtime contract.

Please extend the existing owner repair/validation rather than treating #1326's API-base fix alone as sufficient. A useful permanent RED contract is: force the primary NIM path to a bounded 429, exercise the same installed Strix/LiteLLM runtime and trusted fallback configuration, and prove openai_direct/gpt-5.4 reaches model execution without either (a) inheriting the NVIDIA API base or (b) failing provider/model-specific reasoning_effort validation. The smallest remedy should align provider slug/model normalization and reasoning-effort options with what the installed runtime actually supports; do not remove the fail-closed/no-artifact guard or downgrade provider failures.

Required GREEN evidence: focused central fallback contracts, full exact-head central gates/reviews, then an unchanged-head rerun of ScopeWeave #610 proving the fallback can reach target analysis and emit authoritative exact-head Strix evidence. This new canary is complementary to the existing macos_utility_packs/pg-erd-cloud/OriginWeave 404 canaries.

[seonghobae]

Fresh convergence check on the three live repair vehicles: #1350 (045f1789...), #1352 (dbf8ae19...), and #1353 (6c26f754...) all target the direct-OpenAI Strix reasoning contract, but #1353 currently has the narrowest demonstrated provider boundary for the ScopeWeave canary. Its first revision was correctly challenged for suppressing configured effort on custom OpenAI-compatible endpoints; exact head 6c26f754... now limits none to explicit direct-OpenAI attempts using the native/default https://api.openai.com/v1 authority and adds regressions proving a custom endpoint preserves configured effort. The thread is resolved/outdated, Strix Changed Path Quality CI is terminal success, while CodeQL/Python Security are still queued and therefore non-authorizing.

By contrast, #1350 keys the override specifically to the GPT-5.4 model spelling and #1352 also edits the workflow-level default plus gate propagation; they overlap the same causal path. Please converge on one owner branch rather than integrating multiple competing variants. Preserve #1353's native-vs-custom endpoint regression plus the existing NIM 429/410 → direct OpenAI fallback regression, and require a post-integration unchanged-head downstream canary on scopeweave#610@fb381e86cdb0c6bcc6397a278070cb26e964a4d6. No downstream success should be claimed until that rerun reaches authoritative Strix analysis/artifact generation.

[seonghobae]

Fresh downstream reproduction from BandScope shows the same canonical Strix fallback lane has moved past the earlier 404 but is still broken at the request-contract boundary.

Exact evidence (2026-08-26 UTC):

• consumer: ContextualWisdomLab/bandscope#1024
• exact PR head: c03001e3af3452057d4ec21dec402ce39fc2fa04
• independently resolved live base: develop@749511c3ad4000090048718f685c6bee6b3d2c25
• trusted central source: ContextualWisdomLab/.github/main@e00bd7964f332b69cf7b430b0cb5ad486eef8258
• Strix run: 32982165723
• failing job: 98221210650

Observed attempt sequence from the current job log:

1. nvidia_nim/nvidia/nemotron-3-super-120b-a12b -> HTTP 429 Too Many Requests.
2. fallback nvidia_nim/nvidia/llama-3.3-nemotron-super-49b-v1.5 -> HTTP 410 Gone: Model is no longer available as of Aug 26, 09Z. Please use an alternative model. This fallback is now permanently EOL, not a transient provider failure.
3. final fallback openai_direct/gpt-5.4 reaches OpenAI, but fails HTTP 400: reasoning_effort is not supported with this model (param: reasoning_effort). Strix then exhausts the pool without an authoritative finding/no-finding report and correctly fails closed.

First causal boundary is central provider/model routing and request-envelope capability handling; there is no correct BandScope-local source fix for this exact failure. Please keep the fail-closed gate and repair the central lane rather than asking the consumer to suppress/retry around it.

Acceptance test for this reproduction:

• remove/replace the EOL NIM fallback with a currently supported model under the existing bounded provider policy;
• make request construction capability-aware so openai_direct/gpt-5.4 does not receive unsupported reasoning_effort (or route it through the officially supported endpoint/envelope for that capability);
• add focused regressions covering 410-permanent-model-retirement and model-specific unsupported request parameters without converting real provider failures/findings to success;
• rerun Strix against unchanged BandScope head c03001e3af3452057d4ec21dec402ce39fc2fa04 and require a terminal authoritative finding/no-finding report bound to that exact head before treating the gate as recovered.

[seonghobae]

Fresh ScopeWeave exact-head evidence after the protected-main GPT-5.4 reasoning repair exposes a remaining, separate Strix fallback lifecycle defect that still belongs to this central owner boundary.

ScopeWeave PR #610 exact head fb381e86cdb0c6bcc6397a278070cb26e964a4d6 failed required Strix run 32976934806, job 98203848191, while the trusted workflow source was predecessor .github@e00bd7964f332b69cf7b430b0cb5ad486eef8258:

1. primary nvidia_nim/nvidia/nemotron-3-super-120b-a12b exhausted bounded retries on HTTP 429;
2. fallback nvidia_nim/nvidia/llama-3.3-nemotron-super-49b-v1.5 returned HTTP 410 and explicitly reported end-of-life at 2026-08-26T09:00:00Z;
3. final openai-direct/gpt-5.4 then hit the function-tools/reasoning-effort HTTP 400 that has since been repaired on protected .github/main by merged fix(strix): honor GPT-5.4 tool reasoning contract #1350.

Fresh protected .github/main is now f655a901f7ccdfef0d62694c818ad2896a2f5da1, so the GPT-5.4 reasoning failure is historical for the next rerun. However, that exact protected-main strix.yml still hard-codes the EOL nvidia/llama-3.3-nemotron-super-49b-v1.5 in the nvidia_nim fallback list. A new ScopeWeave run can therefore still waste a complete fallback slot on a deterministically unavailable provider model before reaching direct OpenAI.

Please treat this as a RED acceptance case for the central Strix owner, not a ScopeWeave source defect. Smallest acceptable remedy: make the Strix NVIDIA fallback lifecycle-aware (or otherwise remove the retired pin through the existing trusted model-selection boundary) without broadening credentials, weakening fail-closed evidence, or accepting a provider error as a clean scan. Required GREEN evidence: central focused/full Strix contracts on the owner head plus a rerun of required Strix on the unchanged ScopeWeave #610 head under the then-current protected central workflow. The canary must show the EOL model is not attempted and the authoritative scan still fails closed on genuine provider exhaustion or missing vulnerability evidence.

[seonghobae]

Fresh OriginWeave canary confirms the remaining Strix failure is still central provider-lifecycle state, not an OriginWeave source defect.

• consumer: ContextualWisdomLab/OriginWeave#37
• exact consumer head: 34b825f6959b0ba08ab8f2785601a2e5033c1b3b
• exact protected base: b05d5acca82b9d916ada2c8e82f59f92a89817e1
• required Strix workflow/job: 32944287472 / 98281866567
• trusted central workflow SHA from the job: 874f47b3856ca6bdb6bc71d48173b8f34ba7b9ca

The exact attempt sequence is:

1. primary nvidia_nim/nvidia/nemotron-3-super-120b-a12b exhausted bounded retries on HTTP 429;
2. fallback nvidia_nim/nvidia/llama-3.3-nemotron-super-49b-v1.5 returned HTTP 410 Gone and states the model reached EOL at 2026-08-26T09:00:00Z;
3. final openai-direct/gpt-5.4 reached the direct provider but exhausted on HTTP 429 insufficient_quota / credit_balance_exhausted.

No authoritative vulnerability finding/no-finding report was produced, so the gate correctly remains fail-closed as STRIX_PROVIDER_UNAVAILABLE. OriginWeave native CI, SAST, and Security Scan are separate evidence and must not substitute for Strix.

The permanently retired NVIDIA fallback is the actionable owner defect: remove/replace it through the reviewed central model-selection boundary while preserving bounded attempts, exact-head binding, credential separation, and fail-closed exhaustion. This canary should be rerun only after that central owner repair reaches protected .github/main; the unchanged consumer head must then produce terminal authoritative Strix evidence before promotion.

[seonghobae]

Fresh Orgmetra exact-head canary confirms the Strix failure is still owned by the central provider/fallback boundary, not by the leaf PR.

Consumer tuple:

• leaf PR: ContextualWisdomLab/Orgmetra#45
• exact head: 90aa2284a43b5f2491ee3f20c6946c4a78520c36
• live base observed: develop@9e3e4847510e1e612b48474ba42b177b8ed824df
• required Strix run/job: 33010253596 / 98314385714
• Strix report artifact: 9622747073, digest sha256:45aeeae8700b0e31cd9d8adcb8b4aec451d2eed35f3882686fa92187d804ca78

The exact job acquired a runner, materialized the target workspace, fetched the PR head, installed Strix, and reached Run Strix (quick). The fail-closed cascade was then entirely provider-side:

1. primary nvidia_nim/nvidia/nemotron-3-super-120b-a12b repeatedly returned HTTP 429 Too Many Requests and exhausted bounded retries;
2. fallback openrouter/free failed with OpenRouter API 502 Invalid URL (provider metadata: Stealth);
3. final openai-direct/gpt-5.4 now reaches the direct OpenAI service rather than the old NVIDIA-base 404 path, but fails with HTTP 429 insufficient_quota / credit_balance_exhausted (You have no credits remaining).

No authoritative vulnerability report was produced. One attempt displayed Vulnerabilities 0, but the gate correctly refused to promote it because provider-failure markers remained and the structured report was incomplete. That fail-closed behavior must be preserved.

This is useful evidence that the API-base routing repair alone is insufficient for reliable required-check completion: the owner boundary also needs a healthy provider/key/credit fallback chain (and the OpenRouter invalid-URL route should be classified separately). No Orgmetra source workaround exists without weakening the required security gate.

Acceptance canary after owner repair: rerun Strix on unchanged Orgmetra #45 head 90aa2284...; require a completed authoritative structured report with no provider-failure markers and terminal required-check success. Do not treat log-only 0 vulnerabilities output as passing evidence.

[seonghobae]

Fresh OriginWeave consumer evidence after central Strix 548a97560070d31b03d14bee0ac98a990bd88482 (post-#1297) shows the fallback contract is still not robust enough; no OriginWeave source workaround is appropriate.

Exact consumer:

• ContextualWisdomLab/OriginWeave#229
• exact head 585a7d5545b13f18d76f79100ff4d47ac423e861
• exact live base b05d5acca82b9d916ada2c8e82f59f92a89817e1
• Strix run 33000778334, attempt 2; job 98341504012
• report artifact 9625898775, SHA-256 28b8a28f664dbc7e43ebc4ef6b7122b217d8aa8f013c1da698fb5ffb291ac7bd

The bounded provider cascade failed closed without authoritative vulnerability evidence:

1. primary nvidia_nim/nvidia/nemotron-3-super-120b-a12b repeatedly returned HTTP 429;
2. the live-NVIDIA fallback resolver selected nvidia_nim/nvidia/llama-3.1-nemotron-ultra-253b-v1, but NVIDIA returned HTTP 404 / Function ... Not found for account ..., indicating that catalog selection did not prove account-invocable availability;
3. openrouter/free repeatedly returned HTTP 502 Invalid URL from provider Stealth;
4. openai-direct/gpt-5.4 reached the direct OpenAI route but repeatedly returned HTTP 429 credit_balance_exhausted / insufficient quota.

This is materially different from the old inherited-NVIDIA-base 404 fixed by #1326: direct OpenAI routing is now correct enough to reach OpenAI, but all authority-equivalent fallbacks exhausted. The NVIDIA 404 is also stronger than ordinary transient rate limiting: a live resolver must not promote a model as usable merely because it appears in a catalog when the configured account cannot invoke it.

Please keep Strix fail-closed, but add/extend test-first central owner coverage so model resolution validates account-invocable capability (or rejects a 404-selected candidate from the current run), records the finite provider-unavailable class, and advances only to reviewed authority-equivalent fallbacks under existing bounded attempt/deadline/resource rules. Repeated OpenRouter Stealth 502 should remain a typed provider-unavailable outcome rather than be retried indefinitely; OpenAI billing/quota remains an external fail-closed condition, not a success fallback.

The same job also logged inability to compute the PR merge base and used the existing direct base/head diff fallback for changed-file enumeration. That did retain one scannable changed file, so I am not attributing the provider failure to OriginWeave diff selection; central owner should nevertheless preserve exact base/head identity when validating the eventual fix.

[seonghobae]

Fresh Orgmetra post-owner-repair acceptance canary has been started without changing leaf source.

• consumer: ContextualWisdomLab/Orgmetra#45
• unchanged exact head: 90aa2284a43b5f2491ee3f20c6946c4a78520c36
• live base: develop@9e3e4847510e1e612b48474ba42b177b8ed824df
• prior Strix run/job: 33010253596 / 98314385714 (provider/fallback failure already documented here)
• safe re-run attempt 2 job: 98365671632
• current state: queued, no runner assigned yet; therefore non-passing

All Orgmetra-local exact-head product/Foundation/Recovery/SAST/Security gates remain terminal GREEN. This rerun is the required unchanged-head canary after the central Strix owner repairs; do not treat it as GREEN unless it produces a complete authoritative structured Strix report with no provider-failure markers and a terminal successful required check. No .github source/ref/workflow/PR state was modified from the Orgmetra loop.

[seonghobae]

Fresh downstream canary from ContextualWisdomLab/OriginWeave#43 confirms the required Strix failure is still entirely in the central provider/runtime boundary, not the OriginWeave MV3 product code.

• OriginWeave PR Refresh governance audit after clearfolio rollout #43 exact head: 14728625b180c5ffe6ba0aa1b8d61a16f9347252
• protected live base used by the run: b05d5acca82b9d916ada2c8e82f59f92a89817e1
• required Strix run/job: 33025899963 / 98367053503
• trusted central Strix source: .github@548a97560070d31b03d14bee0ac98a990bd88482
• Strix report artifact: 9628800510, digest sha256:0e81d1aafd01af8cc9f0300d237d42f1ed79038cd62028dc0d46c2a2fde8e5de

The exact-head cascade was:

1. primary nvidia_nim/nvidia/nemotron-3-super-120b-a12b repeatedly returned HTTP 429;
2. NVIDIA fallback nvidia_nim/nvidia/llama-3.1-nemotron-ultra-253b-v1 returned 404 Function ... Not found for account;
3. openrouter/free repeatedly returned 502 Invalid URL from provider Stealth;
4. final openai-direct/gpt-5.4 repeatedly returned 429 insufficient_quota / credit_balance_exhausted.

One NVIDIA attempt printed Vulnerabilities 0, but it also emitted a provider-failure traceback and did not yield authoritative clean report evidence; the central gate correctly kept the run fail-closed as STRIX_PROVIDER_UNAVAILABLE. No OriginWeave source workaround is appropriate. Please retain this exact-head canary for the central provider/fallback repair and rerun against the unchanged leaf head after the owner fix; only a terminal authoritative exact-head Strix result should count.