Skip to content

feat(deploy): reach the same models through hosted APIs - #20

Merged
Odiethebest merged 2 commits into
mainfrom
feat/managed-model-providers
Aug 3, 2026
Merged

Odiethebest merged 2 commits into
mainfrom
feat/managed-model-providers

Conversation

@Odiethebest

Copy link
Copy Markdown
Owner

Stacked on #18 and #19. Until those merge this PR also shows their commits.

Closes the second half of F-13. Delivers D-1 and D-5 in Deploy.md.

What this adds

A second way to reach the embedding and reranker models — Jina AI's hosted endpoint and SiliconFlow's — selected by EMBEDDING_PROVIDER / RERANKER_PROVIDER. Both default to sidecar, so no existing .env, test or compose invocation changes behaviour, and apps/embedding, apps/reranker, the compose profiles, the host scripts and both Http*Client classes are untouched (Deploy.md §5.1).

The models do not change. jina-embeddings-v2-base-code at 768 dimensions and BAAI/bge-reranker-v2-m3 are what the recorded evaluation measured, and every figure the UI displays is a claim about that configuration. The provider selects how a model is reached, never which one runs.

Both wire formats were observed before any client was written

And both differ from the sidecar contract in the same load-bearing way: results are identified by a per-item index and do not arrive in input order. A three-document rerank probe came back [1, 0, 2].

Zipping arrival order against the input would attach the wrong embedding to a chunk, or score every passage with another passage's relevance — no error, no log line, just worse retrieval. Both clients place results by index and reject a response that is short, duplicated, or out of range. This is what D-5 anticipated: neither API is OpenAI-compatible in the part that actually matters.

429 is retryable here and is not in the sidecar clients — a service we run does not rate-limit us, a metered one does. 401 stays non-retryable and surfaces on the first attempt.

SiliconFlow runs two platforms and they are not interchangeable

api.siliconflow.cn api.siliconflow.com
Rerankers BGE v2-m3 + four Qwen3 Qwen3 only, no BGE
.com key 401 "Api key is invalid" works

The failure misleads in both directions: wrong region reads as a bad key, right region with the wrong model reads as 400 Model does not exist. Taking the .com path would have meant deploying a Qwen reranker under BGE's measured numbers, which §2 does not permit. Recorded in Deploy.md §2.2.

Verification

21 mocked-transport tests cover reordering, cross-batch ordering, every malformed-response branch, both retry policies, provider selection, missing-key and unknown-provider errors, and stub short-circuiting. CI stays keyless (§5.3); the backoff sleep is captured rather than waited out, so the suite does not get slower.

Live, from a running stack switched to both hosted providers and queried against the existing 768-dim index written by the local sidecar — that combination is the parity claim itself:

Check Result
mode=dense, the zero-keyword-overlap query from Operations.md identical ranking; cosine differs in the third decimal (0.5155 → 0.5182)
mode=hybrid same five chunks; top two swap, and they were tied at 0.0042 in the baseline — the hosted run broke the tie
Full agent path 3 tool calls, 2 citations, both verified, groundedness 1.0
Against the documented run reproduces Operations.md Verified Current Path — 2026-07-30 citation for citation: sessions.py:186, sessions.py:557

The third-decimal drift is the caveat §2 already recorded: same weights, different serving stack, not bit-identical. Visible, and it does not move the ranking.

Two gaps, named rather than glossed

  • No index has been built through the hosted provider. The verification above is query-side. 726 chunks at batch 32 is ~23 requests, which is where rate limiting and the embed stage's Redis cache interaction would appear. Deliberate — re-indexing would replace the verified reference index — and it will happen at the first Railway index.
  • The sidecar path was not re-run live after the switch. Its code is untouched and its selection is covered by tests, but that is a weaker claim than having watched it work, and it is stated as the weaker one.

make check green: 310 passed / 5 skipped, 73 frontend tests, mypy strict over 83 files.

Adds a second way to reach the embedding and reranker models: Jina AI's
hosted endpoint and SiliconFlow's, selected by EMBEDDING_PROVIDER and
RERANKER_PROVIDER. Both default to `sidecar`, so no existing .env, test or
compose invocation changes behaviour, and apps/embedding, apps/reranker,
the compose profiles, the host scripts and both HttpXClient classes are
untouched.

The models do not change. jina-embeddings-v2-base-code at 768 dimensions
and BAAI/bge-reranker-v2-m3 are what the recorded evaluation measured, and
every figure the UI displays is a claim about that configuration. The
provider selects how a model is reached, never which one runs.

Both wire formats were observed against the live APIs before any client was
written, and both differed from the sidecar contract in the same
load-bearing way: results are identified by a per-item `index` and do not
arrive in input order. A three-document rerank probe came back [1, 0, 2].
Zipping arrival order against the input would attach the wrong embedding to
a chunk, or score every passage with another passage's relevance — no
error, no log line, just worse retrieval. Both clients place results by
index and reject a response that is short, duplicated, or out of range.

429 is retryable here and is not in the sidecar clients. A service we run
does not rate-limit us; a metered one does, and on a free tier it is the
normal response under load. 401 stays non-retryable and surfaces on the
first attempt, because a wrong key does not improve by asking again.

Two findings worth the reader's time, both in Deploy.md. SiliconFlow runs
two independent platforms: .cn serves BAAI/bge-reranker-v2-m3, .com served
only Qwen rerankers, and keys are not interchangeable — the failure reads
as a bad key in one direction and a missing model in the other. Taking the
.com path would have meant deploying a Qwen reranker under BGE's measured
numbers, which §2 does not permit.

Verification, stated precisely. 21 mocked-transport tests cover reordering,
cross-batch ordering, every malformed-response branch, both retry
policies, provider selection and stub short-circuiting; CI stays keyless.
The production classes were then run against both live APIs: 5 inputs over
3 batches returned 5 distinct non-zero 768-dim vectors, and re-embedding
input 2 alone landed 0.000000 from its batched slot against 0.586 for the
nearest other slot, which is what makes the ordering claim a measurement
rather than an assertion.

Not done: no index has been built through the hosted provider. 726 chunks
at batch 32 is where rate limiting and the embed stage's Redis cache
interaction would appear, and neither has been exercised. This commit
proves the clients are correct, not that an index built through them is.
Deploy.md marks PR 2 acceptance as partially met for that reason.
The running stack was switched to both hosted providers and queried against
the existing 768-dimension index — the one the local sidecar wrote. That
combination is the parity claim itself: a query vector from Jina's API
searching vectors produced by the local model.

Dense retrieval returned an identical ranking, the same five chunks in the
same order, with cosine differing in the third decimal (0.5155 -> 0.5182).
Hybrid returned the same five with the top two swapped, and those two were
tied at 0.0042 in the baseline; the hosted run broke the tie 0.0043/0.0042.
A tie broken differently is not a behaviour change and is not reported as
one.

The full agent path then reproduced the documented sidecar run citation for
citation: sessions.py:186 and sessions.py:557, both verified, groundedness
1.0, three tool calls. Hosted BGE also returns scores in the same magnitude
band as the local BGE sidecar, where the Qwen reranker on SiliconFlow's
other platform returned two orders of magnitude higher on a comparable
query — circumstantial, but it is the kind of evidence available.

The third-decimal drift is the caveat §2 already recorded: same weights,
different serving stack, not bit-identical. It is visible and it does not
move the ranking.

Two gaps are named rather than glossed. No index has been built through the
hosted provider, so the bulk-embedding path — 23 requests, rate limiting,
the embed stage's Redis cache interaction — is still unproven; that is
deliberate, because re-indexing would replace the verified reference index,
and it will happen at the first Railway index. And the sidecar path was not
re-run live after the switch: its code is untouched and its selection is
covered by tests, but that is a weaker claim than having watched it work,
so it is stated as the weaker one.
@Odiethebest
Odiethebest merged commit 797bb1e into main Aug 3, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant