Skip to content

docs: confirm the family_cap/max_tokens fixes work end-to-end (one further gap remains open) - #1444

Open
seonghobae wants to merge 8 commits into
mainfrom
docs/sidecar-outage-resolved-confirmation-20260830
Open

docs: confirm the family_cap/max_tokens fixes work end-to-end (one further gap remains open)#1444
seonghobae wants to merge 8 commits into
mainfrom
docs/sidecar-outage-resolved-confirmation-20260830

Conversation

@seonghobae

@seonghobae seonghobae commented Aug 30, 2026

Copy link
Copy Markdown
Contributor

Summary

Item 1 of the post-#1434-merge priority list: verify end-to-end that noema-review/opencode-review/strix actually recover now that main has both #1436's max_tokens fix and #1434's family_cap raise — not just structurally argued.

Update (softened from this PR's original "confirmed resolved" title/framing): after opening this PR, a second, independently-found, timestamped counter-example surfaced and was verified directly against its own job log — a strix run on .github's own main (job 99247611184, completed 2026-08-30T11:16:17Z, predating this PR's original claim) failed with a genuinely separate, unfixed defect: Strix's OpenAI Agents SDK client sends stream_options.include_usage=true together with tools/response_format, and whichever orchestrator/free candidate it routed to rejected that exact combination with HTTP 400 invalid_stream_options on all retries, exhausting the pool (STRIX_PROVIDER_UNAVAILABLE). This is a request-shape incompatibility, not a candidate-selection or token-budget problem — genuinely distinct from what #1434/#1436 fixed. Root-causing it has been delegated elsewhere and is not duplicated in this PR.

Accurate combined claim, checked against both pieces of evidence:

  • The family_cap/max_tokens fixes are confirmed working end-to-end on ContextualWisdomLab/contextual-orchestrator#921 (unrelated to this investigation, picked because it doesn't itself hit the pull_request_target self-test trust boundary fix(zdr): cite NVIDIA's own Trial ToS for the nvidia_nim not-ZDR classification #1434/fix(strix): add missing github-token fallback for self-target status publish #1441 do): noema-review and strix both succeeded on a fresh head. noema-review's job log confirms a real, complete sidecar cycle (not vacuous): CONTEXTUAL_ORCHESTRATOR_BASE_URL was exported, which only happens after the sidecar's own preflight found a viable route and the server became healthy — the exact stage that failed 100% of the time (three independent reproductions) before these fixes landed.
  • orchestrator/free is not reliable in general yet. With family_cap=8 giving more provider diversity per run, which candidate a given run draws varies, and at least one candidate family has the separate, live, unfixed stream_options gap above that can still fail Strix (though evidently not every candidate — contextual-orchestrator#921's own strix run succeeded). Two permanently-retired google/gemma-3-*-it model ids are also still admitted into the pool and will still individually fail when the alphabetical sort reaches them.

Also includes a second correction (real Devin finding, verified against the actual merged diff): updates the pingora_edge_policy.py binary-evidence entry from "two competing open fixes, unresolved" to reflect that #1435 already merged — a third, better implementation that network-verifies the real %PDF- magic prefix rather than trusting the .pdf extension alone.

Test plan

Docs-only change (plus CHANGELOG bullets). tests/test_pr_governance_audit_contract.py passes.

🤖 Generated with Claude Code

https://claude.ai/code/session_015Gs7KmNvH75nxz1sL8mKjw

…ed run

Item 1 of the coordinator's post-merge priority list: verify end-to-end
that noema-review/opencode-review/strix actually recover now that main has
both #1436's max_tokens fix and this investigation's family_cap raise, not
just structurally argued.

Checked ContextualWisdomLab/contextual-orchestrator#921 (deliberately
unrelated to this investigation, so it does not itself hit the
pull_request_target self-test trust boundary #1434/#1441 do) on a fresh
head pushed after main had all three fixes: noema-review and strix both
succeeded. noema-review's job log confirms a real sidecar cycle
(CONTEXTUAL_ORCHESTRATOR_BASE_URL was exported, which only happens after
the sidecar's own preflight found a viable route and the server became
healthy -- the exact stage that failed 100% of the time before these
fixes). opencode-review still failed, but for an already-understood,
unrelated, benign timing gap (it ran before the separate async
opencode-review-dispatch flow had posted a verdict for that exact head),
not a sidecar regression.

Co-Authored-By: Claude <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Aug 30, 2026

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 1e397d86-b974-4189-9f3b-ef64f37ba682


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

devin-ai-integration[bot]

This comment was marked as resolved.

…ne entries

1. Soften "the sidecar-preflight outage is confirmed resolved" to accurately
   reflect a second, independently-found, timestamped counter-example: a
   strix run on .github's own main (job 99247611184, completed
   2026-08-30T11:16:17Z -- predating this PR's original claim) failed with a
   genuinely separate, unfixed defect verified directly from its job log:
   Strix's OpenAI Agents SDK client sends stream_options.include_usage=true
   together with tools/response_format, and the routed orchestrator/free
   candidate rejected that combination with HTTP 400 invalid_stream_options
   on all retries, exhausting the pool (STRIX_PROVIDER_UNAVAILABLE). The
   family_cap/max_tokens fixes are confirmed working end-to-end on
   contextual-orchestrator#921's real run; that specific, previously
   100%-reproducible failure mode is closed. It is not true that
   orchestrator/free is now reliable in general. Root-causing the
   stream_options gap has been delegated elsewhere; not duplicated here.

2. Update the pingora_edge_policy.py binary-evidence entry from "two
   competing open fixes, unresolved" to reflect that #1435 already merged
   -- verified against its actual diff, not assumed from its title: a
   third, better implementation that network-verifies the real %PDF- magic
   prefix rather than trusting the .pdf extension alone, falling back to
   path+suffix trust only for the one case that cannot be verified by
   content at all (a genuinely oversized file). #1420/#1427 remain open for
   the separate, non-blocking question of the other binary formats.

Co-Authored-By: Claude <noreply@anthropic.com>
@seonghobae seonghobae changed the title docs: confirm the sidecar-preflight outage is resolved on a live hosted run docs: confirm the family_cap/max_tokens fixes work end-to-end (one further gap remains open) Aug 30, 2026
claude added 2 commits August 30, 2026 11:25
…ted gap-baseline entry

Companion to the prior commit's gap-baseline correction -- the CHANGELOG
bullet had the same overclaim and needed the same softening.

Co-Authored-By: Claude <noreply@anthropic.com>
…t timeout)

Verified directly from #1441's noema-review job
log (99249903390, head d877886): healthz and the launcher's own internal
preflight succeeded (past the family-cap fix's stage entirely), but the
shell script's separate post-healthz gateway smoke request -- max_tokens
now 4096 per #1436's fix, up from 16 -- got zero bytes back for the full
120-second curl timeout. Distinct from both other evidence in this entry:
not a bad-request rejection (no response at all) and not Strix's own
client code path. Documented with plausible causes (the 120s ceiling never
re-examined when max_tokens grew 256x; a real server-side hang for a
specific request shape; transient flakiness) but not pursued further this
pass -- noted as evidence for whoever picks this up next.

Co-Authored-By: Claude <noreply@anthropic.com>
devin-ai-integration[bot]

This comment was marked as resolved.

The "one further known gap remains" conclusion undercounted: the same
entry already documented two separate, unfixed failure modes (the
stream_options 400 and the 120s-timeout gap). Also corrects a second,
premature claim that contextual-orchestrator#924 already fixed the
stream_options gap -- checked directly: that PR is still open and
mergeable_state=blocked, not merged, so it is a proposed fix only.

Co-Authored-By: Claude <noreply@anthropic.com>
devin-ai-integration[bot]

This comment was marked as resolved.

claude and others added 3 commits August 30, 2026 11:48
…ding)

Bare #924/#921/.github#1441 refs in the text I just added violate the
CLAUDE.md §7 / CWL-MASTER-CONTEXT.md §7 cross-repo reference convention.
Also fixes a leftover "below" that should have said "above" for the
timeout-gap section, missed in the previous correction pass.

Co-Authored-By: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants