Skip to content

Revert LLM_DISABLE_STREAMING=true in strix.yml once contextual-orchestrator fixes the stale invalid_stream_options gate #1446

Description

@seonghobae

Incident

Since Strix moved to contextual-orchestrator's orchestrator/free pool today (#1434), every Strix scan org-wide has been failing closed with openai.BadRequestError: 400 invalid_stream_options — stream_options.include_usage=true is not supported with tools or response_format. Example: https://github.com/ContextualWisdomLab/.github/actions/runs/33307905354/job/99247611184

Root cause (confirmed by reading actual source, not guessing):

  • strix-agent 1.5.3 hardcodes ModelSettings(include_usage=True) for every turn (strix/core/inputs.py::make_model_settings()), with no supported env var, CLI flag, or config key to disable just that flag while keeping streaming. Verified against strix v1.5.3 and openai-agents v0.19.4 source directly.
  • contextual_orchestrator/server.py (around line 6583-6589) unconditionally rejects stream=true + stream_options.include_usage=true whenever tools or response_format are present — a check introduced in commit e7618a3 alongside an unrelated fix, justified only by a test docstring ("Structured passthrough cannot emit usage SSE") that is factually wrong: _chat_response_sse_chunks already builds a genuine usage_source: "reported" SSE chunk from the provider's real usage field, and the tool-loop path (proxy_completion, orchestrator.py) already forces the upstream call to non-streaming and already captures real per-call usage — the honest-usage plumbing this check thinks doesn't exist already exists and already runs on this exact path.

Mitigation shipped now

This PR/commit sets LLM_DISABLE_STREAMING: "true" in the Strix job env in .github/workflows/strix.yml. This is a genuine, documented strix env var (strix/config/models.py::_NonStreamingModel) that makes every turn a single non-streaming call, so stream_options is never sent and the gateway's check never fires. This restores the required Strix gate org-wide immediately, at the cost of Strix losing real-time SSE streaming (all turns become synchronous).

What must happen before this is reverted

  1. A PR on ContextualWisdomLab/contextual-orchestrator that removes or narrows the stale guard at server.py (~line 6583-6589), per evidence that both the tool-loop and non-tool-loop branches already source real, honest usage from the provider response (never fabricated) — no orchestrator.py change is required, the plumbing already works.
  2. That PR flips the two tests currently pinning the rejection (tests/test_chat_tools_passthrough_controls_http_honesty.py::test_http_structured_stream_usage_fails_closed_before_execution, tests/test_stream_options_null_flags_noop_http_honesty.py::test_http_chat_structured_streams_include_usage) to assert a real usage_source: "reported" SSE chunk, adds a positive honesty-invariant test, and corrects the now-stale CHANGELOG.md entry describing the old rejection as intentional.
  3. Once that PR is merged and deployed, verify with a manual Strix workflow_dispatch run (or a direct request reproducing Strix's exact request shape) showing 200 + usage_source: "reported", then remove LLM_DISABLE_STREAMING: "true" from strix.yml here so Strix regains real streaming.

Do not let this mitigation quietly become permanent — it trades away working infrastructure for no architectural reason once the real fix lands.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions