Repository navigation
feat(deploy): reach the same models through hosted APIs - #20
Merged
Merged
Conversation
Adds a second way to reach the embedding and reranker models: Jina AI's hosted endpoint and SiliconFlow's, selected by EMBEDDING_PROVIDER and RERANKER_PROVIDER. Both default to `sidecar`, so no existing .env, test or compose invocation changes behaviour, and apps/embedding, apps/reranker, the compose profiles, the host scripts and both HttpXClient classes are untouched. The models do not change. jina-embeddings-v2-base-code at 768 dimensions and BAAI/bge-reranker-v2-m3 are what the recorded evaluation measured, and every figure the UI displays is a claim about that configuration. The provider selects how a model is reached, never which one runs. Both wire formats were observed against the live APIs before any client was written, and both differed from the sidecar contract in the same load-bearing way: results are identified by a per-item `index` and do not arrive in input order. A three-document rerank probe came back [1, 0, 2]. Zipping arrival order against the input would attach the wrong embedding to a chunk, or score every passage with another passage's relevance — no error, no log line, just worse retrieval. Both clients place results by index and reject a response that is short, duplicated, or out of range. 429 is retryable here and is not in the sidecar clients. A service we run does not rate-limit us; a metered one does, and on a free tier it is the normal response under load. 401 stays non-retryable and surfaces on the first attempt, because a wrong key does not improve by asking again. Two findings worth the reader's time, both in Deploy.md. SiliconFlow runs two independent platforms: .cn serves BAAI/bge-reranker-v2-m3, .com served only Qwen rerankers, and keys are not interchangeable — the failure reads as a bad key in one direction and a missing model in the other. Taking the .com path would have meant deploying a Qwen reranker under BGE's measured numbers, which §2 does not permit. Verification, stated precisely. 21 mocked-transport tests cover reordering, cross-batch ordering, every malformed-response branch, both retry policies, provider selection and stub short-circuiting; CI stays keyless. The production classes were then run against both live APIs: 5 inputs over 3 batches returned 5 distinct non-zero 768-dim vectors, and re-embedding input 2 alone landed 0.000000 from its batched slot against 0.586 for the nearest other slot, which is what makes the ordering claim a measurement rather than an assertion. Not done: no index has been built through the hosted provider. 726 chunks at batch 32 is where rate limiting and the embed stage's Redis cache interaction would appear, and neither has been exercised. This commit proves the clients are correct, not that an index built through them is. Deploy.md marks PR 2 acceptance as partially met for that reason.
The running stack was switched to both hosted providers and queried against the existing 768-dimension index — the one the local sidecar wrote. That combination is the parity claim itself: a query vector from Jina's API searching vectors produced by the local model. Dense retrieval returned an identical ranking, the same five chunks in the same order, with cosine differing in the third decimal (0.5155 -> 0.5182). Hybrid returned the same five with the top two swapped, and those two were tied at 0.0042 in the baseline; the hosted run broke the tie 0.0043/0.0042. A tie broken differently is not a behaviour change and is not reported as one. The full agent path then reproduced the documented sidecar run citation for citation: sessions.py:186 and sessions.py:557, both verified, groundedness 1.0, three tool calls. Hosted BGE also returns scores in the same magnitude band as the local BGE sidecar, where the Qwen reranker on SiliconFlow's other platform returned two orders of magnitude higher on a comparable query — circumstantial, but it is the kind of evidence available. The third-decimal drift is the caveat §2 already recorded: same weights, different serving stack, not bit-identical. It is visible and it does not move the ranking. Two gaps are named rather than glossed. No index has been built through the hosted provider, so the bulk-embedding path — 23 requests, rate limiting, the embed stage's Redis cache interaction — is still unproven; that is deliberate, because re-indexing would replace the verified reference index, and it will happen at the first Railway index. And the sidecar path was not re-run live after the switch: its code is untouched and its selection is covered by tests, but that is a weaker claim than having watched it work, so it is stated as the weaker one.
This was referenced Aug 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #18 and #19. Until those merge this PR also shows their commits.
Closes the second half of F-13. Delivers D-1 and D-5 in
Deploy.md.What this adds
A second way to reach the embedding and reranker models — Jina AI's hosted endpoint and SiliconFlow's — selected by
EMBEDDING_PROVIDER/RERANKER_PROVIDER. Both default tosidecar, so no existing.env, test or compose invocation changes behaviour, andapps/embedding,apps/reranker, the compose profiles, the host scripts and bothHttp*Clientclasses are untouched (Deploy.md§5.1).The models do not change.
jina-embeddings-v2-base-codeat 768 dimensions andBAAI/bge-reranker-v2-m3are what the recorded evaluation measured, and every figure the UI displays is a claim about that configuration. The provider selects how a model is reached, never which one runs.Both wire formats were observed before any client was written
And both differ from the sidecar contract in the same load-bearing way: results are identified by a per-item
indexand do not arrive in input order. A three-document rerank probe came back[1, 0, 2].Zipping arrival order against the input would attach the wrong embedding to a chunk, or score every passage with another passage's relevance — no error, no log line, just worse retrieval. Both clients place results by index and reject a response that is short, duplicated, or out of range. This is what D-5 anticipated: neither API is OpenAI-compatible in the part that actually matters.
429is retryable here and is not in the sidecar clients — a service we run does not rate-limit us, a metered one does.401stays non-retryable and surfaces on the first attempt.SiliconFlow runs two platforms and they are not interchangeable
api.siliconflow.cnapi.siliconflow.com.comkey401 "Api key is invalid"The failure misleads in both directions: wrong region reads as a bad key, right region with the wrong model reads as
400 Model does not exist. Taking the.compath would have meant deploying a Qwen reranker under BGE's measured numbers, which §2 does not permit. Recorded inDeploy.md§2.2.Verification
21 mocked-transport tests cover reordering, cross-batch ordering, every malformed-response branch, both retry policies, provider selection, missing-key and unknown-provider errors, and stub short-circuiting. CI stays keyless (§5.3); the backoff sleep is captured rather than waited out, so the suite does not get slower.
Live, from a running stack switched to both hosted providers and queried against the existing 768-dim index written by the local sidecar — that combination is the parity claim itself:
mode=dense, the zero-keyword-overlap query from Operations.mdmode=hybrid1.0sessions.py:186,sessions.py:557The third-decimal drift is the caveat §2 already recorded: same weights, different serving stack, not bit-identical. Visible, and it does not move the ranking.
Two gaps, named rather than glossed
make checkgreen: 310 passed / 5 skipped, 73 frontend tests, mypy strict over 83 files.