SCA-365: route remaining HTTP LLM surfaces through the gateway - #12337
Merged
Git-on-my-level merged 10 commits intoAug 28, 2026
Merged
Conversation
OpenAI retired Assistants on 2026-08-26, so every /v1/threads create 404s and non-vision file chat is down. Stream PDFs as Chat Completions file parts on gpt-4.1 (documented file-input contract) and keep images on the verified gpt-5.6-luna vision lane. Failure-Class: FC-pinned-vendor-request-contract-retired Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: multica-agent <github@multica.ai>
Unpacking a token-budget dict into create() is untyped and fails pyright. Failure-Class: FC-pinned-vendor-request-contract-retired Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: multica-agent <github@multica.ai>
CI ran each unit file in its own process; incomplete BYOK/LLM stubs and a duration-guard race on desktop transcribe failed the backend suite. Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: multica-agent <github@multica.ai>
Upstream 7c900a9 moved the Soniox client into utils.stt.soniox but left SafeSonioxSocket / process_audio_soniox imported in streaming.py without a re-export marker; receiver.py and test_soniox_streaming.py import them through utils.stt.streaming, so strict pyright fails main and every PR merge ref. Carry the one-line marker so this backend CI lane can go green; matches the existing SafeDeepgramSocket compat re-export convention on the previous line. Co-authored-by: multica-agent <github@multica.ai>
…-365-llm-gateway-surfaces # Conflicts: # backend/tests/unit/test_byok_security.py # backend/tests/unit/test_chat_file_upload_unsupported.py # backend/tests/unit/test_chat_quota.py # backend/tests/unit/test_desktop_transcribe.py # backend/tests/unit/test_paywall_reconnect_gate.py # backend/utils/stt/streaming.py
File-chat completions, company-paid desktop Vertex traffic (text + single embed), and both embedding surfaces now hop backend/llm_gateway so accounting (build_accounting_event / gateway metrics) is the single spend ledger when OMI_LLM_GATEWAY_FEATURE_MODE=gateway: - gateway: OpenAI-shaped /v1/embeddings surface (omi:auto:openai-embeddings, omi:auto:gemini-embeddings) with accounting; desktop-vertex-* chat lanes generated from vertex_pt_routing; VertexGeminiProvider owns the PT policy (pin, promotion latch, overflow ladder, reachability, capacity header, regional vs multi-region host split) and gains tools/toolConfig translation plus :predict embeddings; validator accepts file parts, json_object, and the extra_body-style google options field. - backend: chat_file streams/sync completions through the file-chat lanes (OpenAI Files upload/download stays direct by design); the OpenAI embeddings proxy and gemini_embed_query hop the gateway lanes with BYOK kept on documented thin direct paths; desktop_proxy stays the BFF (auth/metering/limits) and translates Gemini JSON via utils/llm/desktop_gemini_gateway.py, keeping BYOK and batchEmbedContents direct. - FEATURE_MODE=off keeps every legacy direct path (kill switch unchanged); inventory + guardrails updated; omni WS stays blocked.
…ut of the ratcheted files providers.py and desktop_proxy.py grew past the product line-count ratchet; the vertex adapter's pure wire translation (vertex_wire.py), the PT policy mixin (vertex_pt_policy.py), shared provider types (provider_types.py), and the desktop BFF's gateway hop + Gemini body sanitization (utils/llm/desktop_gemini_gateway.py) move into focused modules with the original call sites and test seams preserved via re-exports.
…ile-chat spend The gateway Vertex provider was sharing the PT monotonic clock with ADC token expiry, so tokens never refreshed after the first fetch. File-chat hops also omitted the user uid, leaving ledger rows unattributed, and the desktop tool-loop minted a new tool_call_id after the ordinal advanced. Failure-Class: none Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: multica-agent <github@multica.ai>
…t stubs Direct-path aembed_query/aembed_documents now notify and fall back to the Omi key on BYOK 401 like the sync methods. Hermetic app-integration harnesses stub utils.llm.temporal so collection survives the main merge that imports current_date_for_uid. Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: multica-agent <github@multica.ai>
Live probe confirmed Luna accepts the Chat Completions file-part contract, so the documents lane leaves gpt-4.1. Keep the separate file-chat-documents lane because the request shape still differs. Failure-Class: none Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: multica-agent <github@multica.ai>
Git-on-my-level
added a commit
that referenced
this pull request
Aug 28, 2026
…12374) ## What The Auto Deploy Desktop Backend to Development promotion gate fail-closed on every candidate since #12337: the probe's `REAL_GEMINI_PROVIDER_ROUTES` did not admit the `llm_gateway` provider route the desktop proxy now stamps on `X-Omi-Provider` for company-paid Gemini traffic (run [33219770852](https://github.com/BasedHardware/omi/actions/runs/33219770852), step "Prove candidate chat compatibility", `gemini_proxy: response did not come from an admitted provider route`). Dev stayed 100% on the stale revision `desktop-backend-cdca0c865fd1-33169706190-1`; the candidate was correctly held tag-only. This PR admits the real post-gateway route in the probe allowlist. No backend behavior changes. ## Why this is a real route, not a weakened gate The probed surface is `POST /v1/proxy/gemini/models/gemini-2.5-flash:generateContent`. Post-#12337, that request hops the gateway: - `backend/routers/desktop_proxy.py` → `company_paid_via_gateway` → `proxy_company_paid_via_gateway` stamps `telemetry.provider = 'llm_gateway'` → `X-Omi-Provider: llm_gateway` (`backend/utils/llm/desktop_gemini_gateway.py:595`). - `gemini-2.5-flash` maps to lane `omi:auto:desktop-vertex-flash` (`utils/llm/vertex_pt_routing.py` DESKTOP_TEXT_LANES), whose route pins primary provider `gemini` = `VertexGeminiProvider` with **no fallbacks** (`backend/llm_gateway/gateway/config_loader.py:330`) — real Gemini-on-Vertex. - `offline_stub`, `desktop_llm_stub`, unknown, and empty header values remain rejected fail-closed; the existing offline-stub rejection test is unchanged. ## Test plan - `python3 .github/scripts/test_desktop_backend_candidate_probe.py` — 19 tests OK locally, including: - updated `test_gemini_probe_rejects_stub_or_unknown_provider_routes`: rejects `offline_stub` / `desktop_llm_stub` / `unknown` / `""`, admits `vertex_ai` / `ai_studio` / `ai_studio_byok`; - new `test_gemini_probe_admits_post_gateway_llm_gateway_route`: a gateway-served response (`x-omi-provider: llm_gateway`) passes `_gemini_request` end to end and reports `provider_route: llm_gateway` — the exact live failure mode, now pinned as passing. - After merge: Auto Deploy Desktop Backend to Development must build, probe, and shift 100% dev traffic to the merge-SHA revision; `gcloud run services describe desktop-backend --project=based-hardware-dev --region=us-central1` is the independent serving proof (tracked on SCA-368). ## Product invariants none (per `scripts/pr-preflight --suggest`) ## Failure-Class Failure-Class: FC-client-model-outside-proxy-allowlist #12337 retargeted the desktop proxy at a new provider route without widening the gate that admits it in the same change, so the promotion gate rejected a value it did not know instead of degrading. This PR widens the admission set in data and pins both directions (real route admitted, stub class still rejected) in the probe's test file. <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/BasedHardware/omi/pull/12374?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. -->
pull Bot
pushed a commit
to codingwatching/omi
that referenced
this pull request
Aug 29, 2026
![https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source](https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source){kind=link}
![https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img](https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img){kind=link}
![https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a](https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a){kind=link}
BasedHardware#12337 routed company-paid desktop Gemini traffic through the LLM gateway; since then the desktop proxy stamps X-Omi-Provider: llm_gateway (backend/utils/llm/desktop_gemini_gateway.py), a value the Auto Deploy candidate probe's REAL_GEMINI_PROVIDER_ROUTES did not admit, so "Prove candidate chat compatibility" fail-closed on every promotion (run 33219770852) and dev desktop-backend stayed on the stale revision. Admit llm_gateway: the probed surface (gemini-2.5-flash generateContent) maps to the gateway's desktop-vertex-flash lane, whose primary provider is VertexGeminiProvider with no fallbacks (config_loader.py), so the hop is real Gemini-on-Vertex. Stubs, unknown, and empty routes stay rejected fail-closed; the offline-stub rejection test is unchanged and a new test pins the post-gateway route as admitted end to end through _gemini_request. Verification: python3 .github/scripts/test_desktop_backend_candidate_probe.py 19 tests OK (incl. test_gemini_probe_admits_post_gateway_llm_gateway_route). Failure-Class: FC-client-model-outside-proxy-allowlist Co-authored-by: multica-agent <github@multica.ai>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When
OMI_LLM_GATEWAY_FEATURE_MODE=gateway, the remaining Omi-paid HTTP LLM surfaces hopbackend/llm_gatewaysobuild_accounting_event/ gateway metrics are the spend ledger.omi:auto:file-chat-vision(gpt-5.6-luna) andomi:auto:file-chat-documents(gpt-4.1). OpenAI Files upload/download stays direct (bytes/file_idlifecycle). BYOK OpenAI is not used on this surface because the Files namespace is company-owned.generateContent/streamGenerateContent/embedContenthop generatedomi:auto:desktop-vertex-*lanes andomi:auto:gemini-embeddings.desktop_proxystays the BFF. PT pin/overflow/host policy lives inVertexGeminiProviderviavertex_pt_routing.py(no second policy). Mac Gemini JSON is translated in the BFF; no Desktop Swift change. Gemini BYOK stays thin direct AI Studio.POST /v1/embeddingsfor OpenAItext-embedding-3-largeand Vertexgemini-embedding-001(:predict).batchEmbedContentsstays on AI Studio (Vertex batch wire shape incompatible).This branch merges open #12284 (Assistants sunset → Chat Completions) so file-chat hops the completions path, not a re-deleted Assistants stack. Do not comment on or reassign SCA-362.
Review follow-ups on top of the OMP implementation:
time.time); the PT probe clock is monotonic and must not be shared or tokens never expire-refresh.X-Omi-User-Uidso ledger rows are attributed.tool_call_idinstead of minting a new ordinal.Leftover exceptions (intentional)
omni_relay.py) remains blocked-when-gateway-on.OMI_LLM_GATEWAY_FEATURE_MODE=offkeeps every legacy direct path (kill switch).:generateContentroutes on the gateway.Test plan
pytest tests/unit/test_chat_file_gateway_surface.py tests/unit/test_desktop_gemini_gateway.py tests/unit/test_llm_gateway_vertex_provider.py tests/unit/test_embeddings_gateway.py tests/unit/test_llm_gateway_embeddings_route.py tests/unit/test_llm_gateway_coverage_guardrails.py tests/unit/test_desktop_proxy.py— 162 passedpytest tests/unit/test_chat_file_completions.py tests/unit/test_vertex_pt_routing.py tests/unit/test_llm_gateway_validator.py— 84 passedscripts/pr-preflight --suggestthenOMI_PR_BODY_FILE=… make preflightCould not exercise live Vertex/OpenAI hops from this host (no production credentials / no traffic shift).
Closes SCA-365
Product invariants affected
none
Failure class (fixes)
Failure-Class: none