From 42368cb3e8a9924d571072f7ea94530baf912360 Mon Sep 17 00:00:00 2001 From: odieyang <520yangziqi@gmail.com> Date: Sun, 2 Aug 2026 19:59:40 -0700 Subject: [PATCH 1/2] feat(deploy): reach the same models through hosted APIs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds a second way to reach the embedding and reranker models: Jina AI's hosted endpoint and SiliconFlow's, selected by EMBEDDING_PROVIDER and RERANKER_PROVIDER. Both default to `sidecar`, so no existing .env, test or compose invocation changes behaviour, and apps/embedding, apps/reranker, the compose profiles, the host scripts and both HttpXClient classes are untouched. The models do not change. jina-embeddings-v2-base-code at 768 dimensions and BAAI/bge-reranker-v2-m3 are what the recorded evaluation measured, and every figure the UI displays is a claim about that configuration. The provider selects how a model is reached, never which one runs. Both wire formats were observed against the live APIs before any client was written, and both differed from the sidecar contract in the same load-bearing way: results are identified by a per-item `index` and do not arrive in input order. A three-document rerank probe came back [1, 0, 2]. Zipping arrival order against the input would attach the wrong embedding to a chunk, or score every passage with another passage's relevance — no error, no log line, just worse retrieval. Both clients place results by index and reject a response that is short, duplicated, or out of range. 429 is retryable here and is not in the sidecar clients. A service we run does not rate-limit us; a metered one does, and on a free tier it is the normal response under load. 401 stays non-retryable and surfaces on the first attempt, because a wrong key does not improve by asking again. Two findings worth the reader's time, both in Deploy.md. SiliconFlow runs two independent platforms: .cn serves BAAI/bge-reranker-v2-m3, .com served only Qwen rerankers, and keys are not interchangeable — the failure reads as a bad key in one direction and a missing model in the other. Taking the .com path would have meant deploying a Qwen reranker under BGE's measured numbers, which §2 does not permit. Verification, stated precisely. 21 mocked-transport tests cover reordering, cross-batch ordering, every malformed-response branch, both retry policies, provider selection and stub short-circuiting; CI stays keyless. The production classes were then run against both live APIs: 5 inputs over 3 batches returned 5 distinct non-zero 768-dim vectors, and re-embedding input 2 alone landed 0.000000 from its batched slot against 0.586 for the nearest other slot, which is what makes the ordering claim a measurement rather than an assertion. Not done: no index has been built through the hosted provider. 726 chunks at batch 32 is where rate limiting and the embed stage's Redis cache interaction would appear, and neither has been exercised. This commit proves the clients are correct, not that an index built through them is. Deploy.md marks PR 2 acceptance as partially met for that reason. --- .env.example | 26 +++ .env.production.example | 26 +++ Deploy.md | 94 ++++++-- apps/agent/src/dcode_agent/graph.py | 2 + apps/api/src/dcode_api/routes/internal.py | 4 + apps/worker/src/dcode_worker/stages/embed.py | 2 + docker-compose.prod.yml | 8 + docker-compose.yml | 14 ++ packages/shared/src/dcode_shared/embedding.py | 219 +++++++++++++++++- packages/shared/src/dcode_shared/reranker.py | 172 +++++++++++++- packages/shared/src/dcode_shared/settings.py | 9 + .../shared/tests/test_config_hardening.py | 8 + packages/shared/tests/test_embedding.py | 208 ++++++++++++++++- packages/shared/tests/test_reranker.py | 162 ++++++++++++- 14 files changed, 914 insertions(+), 40 deletions(-) diff --git a/.env.example b/.env.example index f894d1b..bdcd219 100644 --- a/.env.example +++ b/.env.example @@ -51,6 +51,22 @@ EMBEDDING_DIM=1024 # EMBEDDING_MAX_RETRIES=12 # sized for a cold CPU sidecar, not for an API # EMBEDDING_TIMEOUT_SECONDS=300 # ditto — their product is the worst-case stall, # # and the worker consumes one job at a time +# +# Or reach the SAME model through Jina's hosted API instead of running it. +# EMBEDDING_PROVIDER selects HOW the model is reached, never WHICH model runs: +# keeping jina-embeddings-v2-base-code on both paths is what keeps the displayed +# evaluation figures a description of the running system (Deploy.md §2). +# Note the bare model id — the API does not take the Hugging Face `jinaai/` path. +# Switch as a SET; leaving the sidecar retry budget in place points an hour of +# patience at a service that fails in seconds. +# EMBEDDING_PROVIDER=jina_api +# EMBEDDING_API_KEY=jina_... +# EMBEDDING_ENDPOINT=https://api.jina.ai/v1 +# EMBEDDING_MODEL=jina-embeddings-v2-base-code +# EMBEDDING_DIM=768 +# EMBEDDING_BATCH_SIZE=32 +# EMBEDDING_MAX_RETRIES=3 +# EMBEDDING_TIMEOUT_SECONDS=30 # On MacBook: run `make embedding-host` in a separate terminal (recommended). # Or Docker sidecar: docker compose --profile embedding up -d embedding # If you change EMBEDDING_DIM after first migrate: make down-all && make migrate @@ -70,6 +86,16 @@ RERANKER_MODEL=stub # RERANKER_ENDPOINT=http://host.docker.internal:8003 # RERANKER_CANDIDATE_LIMIT=16 # kept small so BGE reranks within the interactive budget on CPU # RERANKER_MAX_SEQ_LENGTH=512 +# +# Or reach the SAME model through SiliconFlow's hosted API. +# Their two platforms are independent: api.siliconflow.cn serves +# BAAI/bge-reranker-v2-m3, api.siliconflow.com served only Qwen rerankers when +# this was written, and keys are NOT interchangeable — a .com key on .cn returns +# 401. The model this project needs is the one the evaluation used, so .cn it is. +# RERANKER_PROVIDER=siliconflow +# RERANKER_API_KEY=sk-... +# RERANKER_ENDPOINT=https://api.siliconflow.cn/v1 +# RERANKER_MODEL=BAAI/bge-reranker-v2-m3 # On MacBook: run `make reranker-host` in a separate terminal (recommended). # Or Docker sidecar: docker compose --profile reranker up -d reranker diff --git a/.env.production.example b/.env.production.example index c106a70..f9760a8 100644 --- a/.env.production.example +++ b/.env.production.example @@ -61,6 +61,22 @@ EMBEDDING_MODEL=stub EMBEDDING_DIM=1024 EMBEDDING_ENDPOINT= +# EMBEDDING_PROVIDER selects HOW the model is reached, never WHICH model runs. +# 'sidecar' expects a self-hosted service; 'jina_api' calls Jina AI's hosted +# endpoint. Both serve jina-embeddings-v2-base-code, which is the point: the +# evaluation figures this project displays describe that model, so the deployed +# system has to be running it (Deploy.md §2). Switch as a SET — the model id is +# the bare name on the API, and the sidecar retry budget is an hour of patience +# pointed at a service that fails in seconds. +# EMBEDDING_PROVIDER=jina_api +# EMBEDDING_API_KEY=jina_... +# EMBEDDING_ENDPOINT=https://api.jina.ai/v1 +# EMBEDDING_MODEL=jina-embeddings-v2-base-code +# EMBEDDING_DIM=768 +# EMBEDDING_BATCH_SIZE=32 +# EMBEDDING_MAX_RETRIES=3 +# EMBEDDING_TIMEOUT_SECONDS=30 + # Sized for a cold self-hosted sidecar. 12 retries against a 300s timeout can # hold one batch of four chunks for about an hour, and the worker consumes one # job at a time, so everything queued behind it waits. Lower both substantially @@ -77,6 +93,16 @@ RERANKER_ENDPOINT= # RERANKER_CANDIDATE_LIMIT=16 # RERANKER_MAX_RETRIES=3 +# Or the same model through SiliconFlow's hosted API. Their two platforms are +# independent and keys are NOT interchangeable: api.siliconflow.cn serves +# BAAI/bge-reranker-v2-m3, while api.siliconflow.com served only Qwen rerankers +# when this was written. A .com key against .cn returns 401 with "Api key is +# invalid", which looks like a bad key rather than a wrong region. +# RERANKER_PROVIDER=siliconflow +# RERANKER_API_KEY=sk-... +# RERANKER_ENDPOINT=https://api.siliconflow.cn/v1 +# RERANKER_MODEL=BAAI/bge-reranker-v2-m3 + # Hybrid RRF fusion weights. Dense above sparse reduces keyword noise. # RRF_DENSE_WEIGHT=2.0 # RRF_SPARSE_WEIGHT=1.0 diff --git a/Deploy.md b/Deploy.md index f7bb037..c1a0ed7 100644 --- a/Deploy.md +++ b/Deploy.md @@ -25,9 +25,11 @@ Outstanding Work list in the same PR, not tracked in two places. is written from an approved plan, and plans outrun implementations. Verify before relying on a claim here, and correct it when it is wrong. -**Status: PR 1 of 5 landed; PR 2–5 not started.** The pre-deployment state is -frozen at tag `v1.0-submission` (`a4612b8`). Nothing is deployed anywhere yet, -and no live model API has been called from this codebase. +**Status: PR 1 landed. PR 2 code complete, its pipeline run still pending. +PR 3–5 not started.** The pre-deployment state is frozen at tag +`v1.0-submission` (`a4612b8`). Nothing is deployed anywhere yet. Both hosted +model APIs have now been called from this codebase and the observed results are +in § 2.1 and § 6 PR 2, but no index has yet been built through them. --- @@ -98,17 +100,43 @@ regenerated. It does mean that **if the H1 suite is ever re-run through the API providers, that is a new run directory with its own `provenance.json`, not a refresh of the recorded snapshot.** -### 2.1 Not yet verified +### 2.1 Verified against the live APIs, 2026-08-02 -Two facts this plan depends on and that must be confirmed with a live call -**before** the database volume is migrated, because `chunks.embedding` fixes its -dimension at migration time: +Both facts this plan depended on were observed rather than assumed, before any +client code was written. -- **Jina's API returns 768-dimensional vectors for `jina-embeddings-v2-base-code`.** - The model card and the recorded run both say 768, but the API response was not - observed in this session. One request settles it. -- **SiliconFlow's rerank request/response field names.** The client in § 6.2 must - be written against the observed payload, not against an assumed one. +- **Jina returns 768-dimensional vectors for `jina-embeddings-v2-base-code`.** + Confirmed: `HTTP 200`, `dim=768`, `768/768` components non-zero. The dimension + matches the recorded run, so no volume re-migration is implied. +- **SiliconFlow's rerank payload.** `{"model", "query", "documents", + "return_documents"}` → `{"id", "meta", "results": [{"index", "document", + "relevance_score"}]}`. + +**And one thing neither the plan nor the documentation predicted: the response +does not come back in input order.** A three-document probe returned indices +`[1, 0, 2]`. Jina's embedding response likewise identifies each vector by +`index` rather than promising positional order. Both clients therefore place +results by index, and that is not defensive coding — zipping arrival order +against the input would attach the wrong embedding to a chunk, or score every +passage with another passage's relevance. Either produces no error, no log line, +and worse retrieval. This is what D-5 anticipated: neither API is +OpenAI-compatible in the part that actually matters. + +### 2.2 SiliconFlow runs two platforms and they are not interchangeable + +Found the hard way, recorded so nobody else spends the time. + +| | `api.siliconflow.cn` | `api.siliconflow.com` | +|---|---|---| +| Rerankers offered | `BAAI/bge-reranker-v2-m3`, `Pro/BAAI/bge-reranker-v2-m3`, four Qwen3 variants | Qwen3 only — **no BGE** | +| A `.com` key against it | `401 "Api key is invalid"` | works | +| A `.cn` key against it | works | `401 "Token is invalid."` | + +The failure is misleading in both directions: a wrong-region key reads as a bad +key, and a right-region key with the wrong model reads as `400 Model does not +exist`. **This project needs `.cn`,** because the model the evaluation used is +only there and § 2 does not permit substituting it. Using the `.com` platform +would have meant deploying a Qwen reranker under BGE's measured numbers. --- @@ -296,7 +324,7 @@ rather than defaulted, because a wrong value there is a re-index, not a slow que of the above, and **each was confirmed to fail when its defect is put back** — a config test that has never been seen failing is not evidence of anything. -### PR 2 — `feat/managed-model-providers` +### PR 2 — `feat/managed-model-providers` — **code complete, pipeline run pending** The capability behind D-1. @@ -320,17 +348,35 @@ The capability behind D-1. to model the difference anyway. - Timeouts and retries appropriate to a hosted API, not to a cold local model. -**Acceptance.** - -1. Unit tests with a mocked transport cover: correct reordering, dimension - validation, auth header present, non-retryable 4xx surfacing, retryable 5xx - retrying. No live calls (§ 5.3). -2. **A live local run, reported with real observed values:** index `psf/requests` - with `EMBEDDING_PROVIDER=jina_api`, then confirm from the database that the - vectors are 768-dimensional and non-zero — this is the § 2.1 check and it must - happen before any Railway volume is migrated. Confirm the reranker is - materially reordering results rather than passing through. -3. `make check` green, and all four modes in § 5.1 still start. +**Acceptance — 1 met, 2 partially met, 3 met.** + +1. **Met.** 21 tests over a mocked transport (`httpx.MockTransport`) cover + index-based reordering, batching with order preserved across batch + boundaries, dimension mismatch, short response, duplicated index, + out-of-range index, auth header, non-retryable 401 surfacing on the first + attempt, retryable 429 backing off, provider selection, missing-key errors, + unknown-provider errors, and stub short-circuiting before the provider is + consulted. No live calls (§ 5.3), and the backoff sleep is captured rather + than waited out. + +2. **Partially met.** The production client classes were exercised against both + live APIs, and the observed values are recorded here: + + | Check | Observed | + |---|---| + | `JinaApiEmbeddingClient`, 5 inputs over 3 batches | 5 vectors, all `dim=768`, all non-zero, all distinct | + | Cross-batch ordering | re-embedding input 2 alone lands `0.000000` from its batched slot; nearest other slot is `0.586` | + | `SiliconFlowRerankerClient`, 3 passages | scores returned in input order; argmax is the `HTTPBasicAuth` passage for a credentials question | + + **What is not done: the full indexing pipeline has not been run through the + hosted provider.** 726 chunks at batch 32 is roughly 23 requests, which is + where rate limiting and the embed stage's Redis cache interaction would show + up, and neither has been exercised. Until that runs, this PR proves the + clients are correct, not that an index built through them is. + +3. **Met.** `make check` green: ruff, eslint, the drift check, mypy strict over + 83 files, 310 passed / 5 skipped, 73 frontend tests. All four modes in § 5.1 + still resolve, and `sidecar` remains the default everywhere. ### PR 3 — `feat/auth-gate` diff --git a/apps/agent/src/dcode_agent/graph.py b/apps/agent/src/dcode_agent/graph.py index 1ed709c..5dc76ee 100644 --- a/apps/agent/src/dcode_agent/graph.py +++ b/apps/agent/src/dcode_agent/graph.py @@ -540,6 +540,8 @@ def _get_reranker_client() -> RerankerClient | None: model=agent_settings.reranker_model, endpoint=agent_settings.reranker_endpoint, max_retries=agent_settings.reranker_max_retries, + provider=agent_settings.reranker_provider, + api_key=agent_settings.reranker_api_key, ) _reranker_client_initialized = True return _reranker_client diff --git a/apps/api/src/dcode_api/routes/internal.py b/apps/api/src/dcode_api/routes/internal.py index a0b52be..c6fde25 100644 --- a/apps/api/src/dcode_api/routes/internal.py +++ b/apps/api/src/dcode_api/routes/internal.py @@ -371,6 +371,8 @@ def _get_query_embedding_client() -> EmbeddingClient: batch_size=api_settings.embedding_batch_size, max_retries=api_settings.embedding_max_retries, timeout_seconds=api_settings.embedding_timeout_seconds, + provider=api_settings.embedding_provider, + api_key=api_settings.embedding_api_key, ) return _query_embedding_client @@ -396,6 +398,8 @@ def _get_query_reranker_client() -> RerankerClient | None: model=api_settings.reranker_model, endpoint=api_settings.reranker_endpoint, max_retries=api_settings.reranker_max_retries, + provider=api_settings.reranker_provider, + api_key=api_settings.reranker_api_key, ) _query_reranker_client_initialized = True return _query_reranker_client diff --git a/apps/worker/src/dcode_worker/stages/embed.py b/apps/worker/src/dcode_worker/stages/embed.py index d60cb3a..17974a9 100644 --- a/apps/worker/src/dcode_worker/stages/embed.py +++ b/apps/worker/src/dcode_worker/stages/embed.py @@ -54,6 +54,8 @@ async def run( batch_size=worker_settings.embedding_batch_size, max_retries=worker_settings.embedding_max_retries, timeout_seconds=worker_settings.embedding_timeout_seconds, + provider=worker_settings.embedding_provider, + api_key=worker_settings.embedding_api_key, ) owns_redis = redis_client is None diff --git a/docker-compose.prod.yml b/docker-compose.prod.yml index 2691ea8..2ea70dc 100644 --- a/docker-compose.prod.yml +++ b/docker-compose.prod.yml @@ -61,12 +61,16 @@ services: EMBEDDING_BATCH_SIZE: ${EMBEDDING_BATCH_SIZE:-4} EMBEDDING_MAX_RETRIES: ${EMBEDDING_MAX_RETRIES:-12} EMBEDDING_TIMEOUT_SECONDS: ${EMBEDDING_TIMEOUT_SECONDS:-300} + EMBEDDING_PROVIDER: ${EMBEDDING_PROVIDER:-sidecar} + EMBEDDING_API_KEY: ${EMBEDDING_API_KEY:-} RRF_DENSE_WEIGHT: ${RRF_DENSE_WEIGHT:-2.0} RRF_SPARSE_WEIGHT: ${RRF_SPARSE_WEIGHT:-1.0} RERANKER_MODEL: ${RERANKER_MODEL:-stub} RERANKER_ENDPOINT: ${RERANKER_ENDPOINT:-} RERANKER_CANDIDATE_LIMIT: ${RERANKER_CANDIDATE_LIMIT:-16} RERANKER_MAX_RETRIES: ${RERANKER_MAX_RETRIES:-3} + RERANKER_PROVIDER: ${RERANKER_PROVIDER:-sidecar} + RERANKER_API_KEY: ${RERANKER_API_KEY:-} LOG_LEVEL: ${LOG_LEVEL:-info} depends_on: postgres: @@ -96,6 +100,8 @@ services: EMBEDDING_BATCH_SIZE: ${EMBEDDING_BATCH_SIZE:-4} EMBEDDING_MAX_RETRIES: ${EMBEDDING_MAX_RETRIES:-12} EMBEDDING_TIMEOUT_SECONDS: ${EMBEDDING_TIMEOUT_SECONDS:-300} + EMBEDDING_PROVIDER: ${EMBEDDING_PROVIDER:-sidecar} + EMBEDDING_API_KEY: ${EMBEDDING_API_KEY:-} WORKDIR_BASE: /tmp/dcode-workdirs INTERNAL_API_KEY: ${INTERNAL_API_KEY:?set INTERNAL_API_KEY in .env.production} LOG_LEVEL: ${LOG_LEVEL:-info} @@ -135,6 +141,8 @@ services: RERANKER_MODEL: ${RERANKER_MODEL:-stub} RERANKER_ENDPOINT: ${RERANKER_ENDPOINT:-} RERANKER_MAX_RETRIES: ${RERANKER_MAX_RETRIES:-3} + RERANKER_PROVIDER: ${RERANKER_PROVIDER:-sidecar} + RERANKER_API_KEY: ${RERANKER_API_KEY:-} # Not read directly by the agent, but dcode_shared.db.models binds the # chunks.embedding column type to it at import time. EMBEDDING_DIM: ${EMBEDDING_DIM:?set EMBEDDING_DIM in .env.production — it must equal the dimension the volume was migrated at} diff --git a/docker-compose.yml b/docker-compose.yml index 065b726..1dc422b 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -62,12 +62,19 @@ services: EMBEDDING_BATCH_SIZE: ${EMBEDDING_BATCH_SIZE:-4} EMBEDDING_MAX_RETRIES: ${EMBEDDING_MAX_RETRIES:-12} EMBEDDING_TIMEOUT_SECONDS: ${EMBEDDING_TIMEOUT_SECONDS:-300} + # 'sidecar' (default) or 'jina_api'. Selects HOW the model is reached, + # never WHICH model runs — both serve jina-embeddings-v2-base-code. + EMBEDDING_PROVIDER: ${EMBEDDING_PROVIDER:-sidecar} + EMBEDDING_API_KEY: ${EMBEDDING_API_KEY:-} RRF_DENSE_WEIGHT: ${RRF_DENSE_WEIGHT:-2.0} RRF_SPARSE_WEIGHT: ${RRF_SPARSE_WEIGHT:-1.0} RERANKER_MODEL: ${RERANKER_MODEL:-stub} RERANKER_ENDPOINT: ${RERANKER_ENDPOINT:-http://host.docker.internal:8003} RERANKER_CANDIDATE_LIMIT: ${RERANKER_CANDIDATE_LIMIT:-16} RERANKER_MAX_RETRIES: ${RERANKER_MAX_RETRIES:-3} + # 'sidecar' (default) or 'siliconflow' — both serve bge-reranker-v2-m3. + RERANKER_PROVIDER: ${RERANKER_PROVIDER:-sidecar} + RERANKER_API_KEY: ${RERANKER_API_KEY:-} LOG_LEVEL: ${LOG_LEVEL:-info} ports: - "8000:8000" @@ -99,6 +106,10 @@ services: EMBEDDING_BATCH_SIZE: ${EMBEDDING_BATCH_SIZE:-4} EMBEDDING_MAX_RETRIES: ${EMBEDDING_MAX_RETRIES:-12} EMBEDDING_TIMEOUT_SECONDS: ${EMBEDDING_TIMEOUT_SECONDS:-300} + # 'sidecar' (default) or 'jina_api'. Selects HOW the model is reached, + # never WHICH model runs — both serve jina-embeddings-v2-base-code. + EMBEDDING_PROVIDER: ${EMBEDDING_PROVIDER:-sidecar} + EMBEDDING_API_KEY: ${EMBEDDING_API_KEY:-} WORKDIR_BASE: /tmp/dcode-workdirs INTERNAL_API_KEY: ${INTERNAL_API_KEY:?set INTERNAL_API_KEY in .env} LOG_LEVEL: ${LOG_LEVEL:-info} @@ -178,6 +189,9 @@ services: EMBEDDING_DIM: ${EMBEDDING_DIM:-1024} RERANKER_MODEL: ${RERANKER_MODEL:-stub} RERANKER_ENDPOINT: ${RERANKER_ENDPOINT:-http://host.docker.internal:8003} + RERANKER_MAX_RETRIES: ${RERANKER_MAX_RETRIES:-3} + RERANKER_PROVIDER: ${RERANKER_PROVIDER:-sidecar} + RERANKER_API_KEY: ${RERANKER_API_KEY:-} RETRIEVAL_BASE_URL: http://api:8000 WORKDIR_BASE: /tmp/dcode-workdirs INTERNAL_API_KEY: ${INTERNAL_API_KEY:?set INTERNAL_API_KEY in .env} diff --git a/packages/shared/src/dcode_shared/embedding.py b/packages/shared/src/dcode_shared/embedding.py index bf74c45..6b0f242 100644 --- a/packages/shared/src/dcode_shared/embedding.py +++ b/packages/shared/src/dcode_shared/embedding.py @@ -1,8 +1,15 @@ """Embedding clients shared by worker (index-time) and API (query-time). -``jinaai/jina-embeddings-v2-base-code`` can be hosted as a sidecar HTTP service; -worker/API call it through :class:`HttpEmbeddingClient`. -``EMBEDDING_MODEL=stub`` keeps local development working without the sidecar. +The same model, ``jinaai/jina-embeddings-v2-base-code``, is reachable two ways: +as a self-hosted sidecar (:class:`HttpEmbeddingClient`) or through Jina AI's +hosted API (:class:`JinaApiEmbeddingClient`). ``EMBEDDING_PROVIDER`` selects +which, and defaults to the sidecar so no existing configuration changes +behaviour. ``EMBEDDING_MODEL=stub`` short-circuits both. + +Keeping one model across both paths is deliberate: every evaluation figure this +project displays was measured on that model, so a substitution would make the +displayed numbers describe something other than the running system. See +``Deploy.md`` §2. """ from __future__ import annotations @@ -19,6 +26,21 @@ _DEFAULT_TIMEOUT_SECONDS = 300.0 _DEFAULT_MAX_RETRIES = 12 +# Hosted-API defaults. A sidecar can be a model loading on CPU for minutes; an +# API answers in seconds or fails. Applying the sidecar's patience to an API +# turns one bad batch into an hour-long stall behind a prefetch_count=1 worker. +_DEFAULT_API_BATCH_SIZE = 32 +_DEFAULT_API_TIMEOUT_SECONDS = 30.0 +_DEFAULT_API_MAX_RETRIES = 3 + +PROVIDER_SIDECAR = "sidecar" +PROVIDER_JINA_API = "jina_api" + +# A sidecar we run does not rate-limit us; a hosted API does, and 429 is the +# normal response on a free tier under load. It has to be retryable even though +# every other 4xx must not be. +_RETRYABLE_STATUS = frozenset({429, 500, 502, 503, 504}) + class EmbeddingClient(ABC): """Abstract client for the configured embedding model.""" @@ -122,6 +144,146 @@ async def _post_embed_with_retries( raise last_error +class JinaApiEmbeddingClient(EmbeddingClient): + """Call Jina AI's hosted ``POST {endpoint}/embeddings``. + + Wire format, observed against the live API on 2026-08-02 rather than taken + from documentation:: + + -> {"model": "jina-embeddings-v2-base-code", "input": ["a", "b"]} + <- {"model": ..., "object": "list", + "usage": {"total_tokens": 20, "prompt_tokens": 20}, + "data": [{"object": "embedding", "index": 0, "embedding": [...768]}]} + + Two differences from the sidecar contract, and both are load-bearing: + + * the request carries an ``Authorization`` header and a ``model`` field, + neither of which the sidecar wants; + * the response identifies each vector by ``index`` instead of returning a + bare list in input order. Arrival order is not part of the contract, so + the vectors are placed by index. Trusting arrival order would silently + attach the wrong embedding to a chunk — a corruption with no error and no + log line, only worse retrieval. + + The model id is the bare name here, not the Hugging Face ``jinaai/`` path + the sidecar loads. + """ + + def __init__( + self, + endpoint: str, + *, + api_key: str, + model: str, + dim: int, + batch_size: int = _DEFAULT_API_BATCH_SIZE, + timeout_seconds: float = _DEFAULT_API_TIMEOUT_SECONDS, + max_retries: int = _DEFAULT_API_MAX_RETRIES, + transport: httpx.AsyncBaseTransport | None = None, + ) -> None: + self._url = f"{endpoint.rstrip('/')}/embeddings" + self._api_key = api_key + self._model = model + self._dim = dim + self._batch_size = max(1, batch_size) + self._timeout = httpx.Timeout(timeout_seconds, connect=10.0) + self._max_retries = max(1, max_retries) + self._transport = transport + + async def embed_batch(self, texts: list[str]) -> list[list[float]]: + if not texts: + return [] + + vectors: list[list[float]] = [] + async with httpx.AsyncClient( + timeout=self._timeout, transport=self._transport + ) as client: + for start in range(0, len(texts), self._batch_size): + batch = texts[start : start + self._batch_size] + payload = await self._post_with_retries(client, batch) + vectors.extend(self._vectors_in_input_order(payload, len(batch))) + return vectors + + def _vectors_in_input_order( + self, payload: dict[str, object], expected: int + ) -> list[list[float]]: + data = payload.get("data") + if not isinstance(data, list): + raise RuntimeError("embedding API returned no 'data' list") + if len(data) != expected: + raise RuntimeError( + f"embedding API returned {len(data)} vectors for {expected} inputs" + ) + + slots: list[list[float] | None] = [None] * expected + for item in data: + if not isinstance(item, dict): + raise RuntimeError("embedding API returned a non-object data item") + index = item.get("index") + if not isinstance(index, int) or isinstance(index, bool): + raise RuntimeError("embedding API returned an item without an integer index") + if not 0 <= index < expected: + raise RuntimeError(f"embedding API returned out-of-range index {index}") + if slots[index] is not None: + raise RuntimeError(f"embedding API returned duplicate index {index}") + slots[index] = _validate_vector(item.get("embedding"), self._dim) + + ordered: list[list[float]] = [] + for position, vector in enumerate(slots): + if vector is None: + raise RuntimeError(f"embedding API returned no vector for input {position}") + ordered.append(vector) + return ordered + + async def _post_with_retries( + self, client: httpx.AsyncClient, batch: list[str] + ) -> dict[str, object]: + last_error: Exception | None = None + for attempt in range(self._max_retries): + try: + response = await client.post( + self._url, + headers={ + "Authorization": f"Bearer {self._api_key}", + "Content-Type": "application/json", + }, + json={"model": self._model, "input": batch}, + ) + response.raise_for_status() + payload = response.json() + if not isinstance(payload, dict): + raise RuntimeError("embedding API returned invalid JSON payload") + return payload + except ( + httpx.RemoteProtocolError, + httpx.ConnectError, + httpx.ReadTimeout, + httpx.HTTPStatusError, + ) as exc: + if ( + isinstance(exc, httpx.HTTPStatusError) + and exc.response.status_code not in _RETRYABLE_STATUS + ): + # 401 and 422 do not get better by asking again, and a + # wrong key should surface immediately rather than after + # three backoffs. + raise + last_error = exc + if attempt >= self._max_retries - 1: + break + delay = min(30.0, 2.0 * (2**attempt)) + logger.warning( + "embedding API request failed (attempt %s/%s), retrying in %.0fs: %s", + attempt + 1, + self._max_retries, + delay, + exc, + ) + await asyncio.sleep(delay) + assert last_error is not None + raise last_error + + def create_embedding_client( *, model: str, @@ -130,8 +292,14 @@ def create_embedding_client( batch_size: int = _DEFAULT_BATCH_SIZE, max_retries: int = _DEFAULT_MAX_RETRIES, timeout_seconds: float = _DEFAULT_TIMEOUT_SECONDS, + provider: str = PROVIDER_SIDECAR, + api_key: str = "", ) -> EmbeddingClient: - """Build the embedding client for the current environment.""" + """Build the embedding client for the current environment. + + ``provider`` defaults to the sidecar, so an environment that predates + ``EMBEDDING_PROVIDER`` keeps exactly its current behaviour. + """ if model == "stub": return StubEmbeddingClient(dim=dim) @@ -141,13 +309,42 @@ def create_embedding_client( "EMBEDDING_ENDPOINT is required when EMBEDDING_MODEL is not 'stub'" ) - logger.info("using HTTP embedding client model=%s endpoint=%s", model, normalized_endpoint) - return HttpEmbeddingClient( - normalized_endpoint, - dim=dim, - batch_size=batch_size, - max_retries=max_retries, - timeout_seconds=timeout_seconds, + normalized_provider = provider.strip().lower() or PROVIDER_SIDECAR + + if normalized_provider == PROVIDER_SIDECAR: + logger.info( + "using HTTP embedding client model=%s endpoint=%s", model, normalized_endpoint + ) + return HttpEmbeddingClient( + normalized_endpoint, + dim=dim, + batch_size=batch_size, + max_retries=max_retries, + timeout_seconds=timeout_seconds, + ) + + if normalized_provider == PROVIDER_JINA_API: + if not api_key.strip(): + raise ValueError( + f"EMBEDDING_API_KEY is required when EMBEDDING_PROVIDER is " + f"'{PROVIDER_JINA_API}'" + ) + logger.info( + "using Jina API embedding client model=%s endpoint=%s", model, normalized_endpoint + ) + return JinaApiEmbeddingClient( + normalized_endpoint, + api_key=api_key.strip(), + model=model, + dim=dim, + batch_size=batch_size, + max_retries=max_retries, + timeout_seconds=timeout_seconds, + ) + + raise ValueError( + f"unknown EMBEDDING_PROVIDER {provider!r}; " + f"expected one of {PROVIDER_SIDECAR!r}, {PROVIDER_JINA_API!r}" ) diff --git a/packages/shared/src/dcode_shared/reranker.py b/packages/shared/src/dcode_shared/reranker.py index 21070b6..1151e7f 100644 --- a/packages/shared/src/dcode_shared/reranker.py +++ b/packages/shared/src/dcode_shared/reranker.py @@ -23,6 +23,13 @@ _DEFAULT_TIMEOUT_SECONDS = 20.0 _DEFAULT_MAX_RETRIES = 3 +PROVIDER_SIDECAR = "sidecar" +PROVIDER_SILICONFLOW = "siliconflow" + +# A hosted API rate-limits; a sidecar we run does not. 429 has to be retryable +# even though every other 4xx must not be. +_RETRYABLE_STATUS = frozenset({429, 500, 502, 503, 504}) + class RerankerClient(ABC): """Abstract cross-encoder reranker client.""" @@ -106,15 +113,150 @@ async def _post_rerank_with_retries( raise last_error +class SiliconFlowRerankerClient(RerankerClient): + """Call SiliconFlow's hosted ``POST {endpoint}/rerank``. + + Wire format, observed against the live API on 2026-08-02:: + + -> {"model": "BAAI/bge-reranker-v2-m3", "query": "...", + "documents": ["a", "b", "c"], "return_documents": false} + <- {"id": "...", "meta": {"tokens": {...}}, + "results": [{"index": 1, "document": null, "relevance_score": 0.64}, + {"index": 0, ...}, {"index": 2, ...}]} + + **``results`` arrives in relevance order, not input order.** The observed + response for three documents was index ``[1, 0, 2]``. This class implements + the :class:`RerankerClient` contract — one score per passage, *in input + order* — so the remapping is the whole job, not a detail. Zipping the + response against the input would score every passage with another + passage's relevance and silently invert the ranking. + + A note for whoever configures this: SiliconFlow runs two independent + platforms. ``api.siliconflow.cn`` serves ``BAAI/bge-reranker-v2-m3``; + ``api.siliconflow.com`` at the time of writing served only Qwen rerankers, + and keys are not interchangeable between them — a ``.com`` key on ``.cn`` + returns 401, and the reverse returns "Token is invalid". The model this + project needs is the one the evaluation used, so the endpoint is the ``.cn`` + one. + """ + + def __init__( + self, + endpoint: str, + *, + api_key: str, + model: str, + timeout_seconds: float = _DEFAULT_TIMEOUT_SECONDS, + max_retries: int = _DEFAULT_MAX_RETRIES, + transport: httpx.AsyncBaseTransport | None = None, + ) -> None: + self._url = f"{endpoint.rstrip('/')}/rerank" + self._api_key = api_key + self._model = model + self._timeout = httpx.Timeout(timeout_seconds, connect=10.0) + self._max_retries = max(1, max_retries) + self._transport = transport + + async def rerank(self, query: str, passages: list[str]) -> list[float]: + if not passages: + return [] + + payload = await self._post_with_retries(query, passages) + return self._scores_in_input_order(payload, len(passages)) + + def _scores_in_input_order(self, payload: dict[str, object], expected: int) -> list[float]: + results = payload.get("results") + if not isinstance(results, list): + raise RuntimeError("reranker API returned no 'results' list") + if len(results) != expected: + raise RuntimeError( + f"reranker API returned {len(results)} scores for {expected} passages" + ) + + slots: list[float | None] = [None] * expected + for item in results: + if not isinstance(item, dict): + raise RuntimeError("reranker API returned a non-object result item") + index = item.get("index") + if not isinstance(index, int) or isinstance(index, bool): + raise RuntimeError("reranker API returned a result without an integer index") + if not 0 <= index < expected: + raise RuntimeError(f"reranker API returned out-of-range index {index}") + if slots[index] is not None: + raise RuntimeError(f"reranker API returned duplicate index {index}") + slots[index] = _validate_score(item.get("relevance_score")) + + ordered: list[float] = [] + for position, score in enumerate(slots): + if score is None: + raise RuntimeError(f"reranker API returned no score for passage {position}") + ordered.append(score) + return ordered + + async def _post_with_retries(self, query: str, passages: list[str]) -> dict[str, object]: + last_error: Exception | None = None + async with httpx.AsyncClient( + timeout=self._timeout, transport=self._transport + ) as client: + for attempt in range(self._max_retries): + try: + response = await client.post( + self._url, + headers={ + "Authorization": f"Bearer {self._api_key}", + "Content-Type": "application/json", + }, + json={ + "model": self._model, + "query": query, + "documents": passages, + "return_documents": False, + }, + ) + response.raise_for_status() + payload = response.json() + if not isinstance(payload, dict): + raise RuntimeError("reranker API returned invalid JSON payload") + return payload + except ( + httpx.RemoteProtocolError, + httpx.ConnectError, + httpx.HTTPStatusError, + ) as exc: + if ( + isinstance(exc, httpx.HTTPStatusError) + and exc.response.status_code not in _RETRYABLE_STATUS + ): + raise + last_error = exc + if attempt >= self._max_retries - 1: + break + delay = min(30.0, 2.0 * (2**attempt)) + logger.warning( + "reranker API request failed (attempt %s/%s), retrying in %.0fs: %s", + attempt + 1, + self._max_retries, + delay, + exc, + ) + await asyncio.sleep(delay) + assert last_error is not None + raise last_error + + def create_reranker_client( *, model: str, endpoint: str = "", max_retries: int = _DEFAULT_MAX_RETRIES, + provider: str = PROVIDER_SIDECAR, + api_key: str = "", ) -> RerankerClient | None: """Build the reranker client for the current environment. Returns ``None`` when ``model`` is ``stub`` so callers keep identity rerank. + ``provider`` defaults to the sidecar, so an environment that predates + ``RERANKER_PROVIDER`` keeps exactly its current behaviour. """ if model == "stub": return None @@ -123,8 +265,34 @@ def create_reranker_client( if not normalized_endpoint: raise ValueError("RERANKER_ENDPOINT is required when RERANKER_MODEL is not 'stub'") - logger.info("using HTTP reranker client model=%s endpoint=%s", model, normalized_endpoint) - return HttpRerankerClient(normalized_endpoint, max_retries=max_retries) + normalized_provider = provider.strip().lower() or PROVIDER_SIDECAR + + if normalized_provider == PROVIDER_SIDECAR: + logger.info("using HTTP reranker client model=%s endpoint=%s", model, normalized_endpoint) + return HttpRerankerClient(normalized_endpoint, max_retries=max_retries) + + if normalized_provider == PROVIDER_SILICONFLOW: + if not api_key.strip(): + raise ValueError( + f"RERANKER_API_KEY is required when RERANKER_PROVIDER is " + f"'{PROVIDER_SILICONFLOW}'" + ) + logger.info( + "using SiliconFlow reranker client model=%s endpoint=%s", + model, + normalized_endpoint, + ) + return SiliconFlowRerankerClient( + normalized_endpoint, + api_key=api_key.strip(), + model=model, + max_retries=max_retries, + ) + + raise ValueError( + f"unknown RERANKER_PROVIDER {provider!r}; " + f"expected one of {PROVIDER_SIDECAR!r}, {PROVIDER_SILICONFLOW!r}" + ) def _validate_score(score: object) -> float: diff --git a/packages/shared/src/dcode_shared/settings.py b/packages/shared/src/dcode_shared/settings.py index c1a02b0..401e2b1 100644 --- a/packages/shared/src/dcode_shared/settings.py +++ b/packages/shared/src/dcode_shared/settings.py @@ -42,6 +42,15 @@ class SharedSettings(BaseSettings): groundedness_threshold: float = 0.95 # --- Retrieval and evaluation models --- + # `provider` selects HOW a model is reached, never WHICH model runs. Both + # paths serve the same weights, which is what keeps the displayed + # evaluation figures a description of the deployed system (Deploy.md §2). + # Default `sidecar` so configuration written before these existed is + # unaffected. + embedding_provider: str = "sidecar" + embedding_api_key: str = "" + reranker_provider: str = "sidecar" + reranker_api_key: str = "" embedding_model: str = "stub" embedding_dim: int = 1024 embedding_endpoint: str = "" diff --git a/packages/shared/tests/test_config_hardening.py b/packages/shared/tests/test_config_hardening.py index 716d998..c4949bc 100644 --- a/packages/shared/tests/test_config_hardening.py +++ b/packages/shared/tests/test_config_hardening.py @@ -87,12 +87,16 @@ def test_prod_services_receive_the_retrieval_settings_they_read() -> None: "EMBEDDING_BATCH_SIZE", "EMBEDDING_MAX_RETRIES", "EMBEDDING_TIMEOUT_SECONDS", + "EMBEDDING_PROVIDER", + "EMBEDDING_API_KEY", "RRF_DENSE_WEIGHT", "RRF_SPARSE_WEIGHT", "RERANKER_MODEL", "RERANKER_ENDPOINT", "RERANKER_CANDIDATE_LIMIT", "RERANKER_MAX_RETRIES", + "RERANKER_PROVIDER", + "RERANKER_API_KEY", }, # dcode_worker.stages.embed builds the index-time embedding client. "worker": { @@ -102,6 +106,8 @@ def test_prod_services_receive_the_retrieval_settings_they_read() -> None: "EMBEDDING_BATCH_SIZE", "EMBEDDING_MAX_RETRIES", "EMBEDDING_TIMEOUT_SECONDS", + "EMBEDDING_PROVIDER", + "EMBEDDING_API_KEY", }, # dcode_agent.graph reranks its own evidence union; EMBEDDING_DIM is # not read by the agent directly but binds the ORM column type. @@ -110,6 +116,8 @@ def test_prod_services_receive_the_retrieval_settings_they_read() -> None: "RERANKER_MODEL", "RERANKER_ENDPOINT", "RERANKER_MAX_RETRIES", + "RERANKER_PROVIDER", + "RERANKER_API_KEY", "SYNTHESIS_MODEL", "SYNTHESIS_MAX_TOKENS", "SYNTHESIS_TEMPERATURE", diff --git a/packages/shared/tests/test_embedding.py b/packages/shared/tests/test_embedding.py index eb4f22f..60e2ae5 100644 --- a/packages/shared/tests/test_embedding.py +++ b/packages/shared/tests/test_embedding.py @@ -1,7 +1,17 @@ -"""Tests for shared embedding clients.""" +"""Tests for shared embedding clients. +The hosted-provider tests drive a mocked transport. They never call a live API: +CI holds no keys and must keep holding none (Deploy.md §5.3). +""" + +import asyncio +import json + +import httpx +import pytest from dcode_shared.embedding import ( HttpEmbeddingClient, + JinaApiEmbeddingClient, StubEmbeddingClient, create_embedding_client, ) @@ -35,3 +45,199 @@ def test_create_embedding_client_builds_http_client() -> None: endpoint="http://localhost:8002", ) assert isinstance(client, HttpEmbeddingClient) + + +# --- hosted provider: Jina AI ------------------------------------------- + + +def _jina_client( + handler: object, + *, + dim: int = 3, + batch_size: int = 8, + max_retries: int = 3, +) -> JinaApiEmbeddingClient: + return JinaApiEmbeddingClient( + "https://api.jina.ai/v1", + api_key="test-key", + model="jina-embeddings-v2-base-code", + dim=dim, + batch_size=batch_size, + max_retries=max_retries, + timeout_seconds=1.0, + transport=httpx.MockTransport(handler), # type: ignore[arg-type] + ) + + +async def test_jina_client_places_vectors_by_index_not_arrival_order() -> None: + """The response identifies vectors by index and may not arrive in order. + + Zipping arrival order against the inputs would attach the wrong embedding + to a chunk — silent corruption with no error, only worse retrieval. So the + mock deliberately answers out of order. + """ + captured: dict[str, object] = {} + + def handler(request: httpx.Request) -> httpx.Response: + captured["body"] = json.loads(request.content) + captured["auth"] = request.headers.get("Authorization") + captured["url"] = str(request.url) + return httpx.Response( + 200, + json={ + "model": "jina-embeddings-v2-base-code", + "object": "list", + "usage": {"total_tokens": 6, "prompt_tokens": 6}, + "data": [ + {"object": "embedding", "index": 2, "embedding": [2.0, 2.0, 2.0]}, + {"object": "embedding", "index": 0, "embedding": [0.0, 0.1, 0.2]}, + {"object": "embedding", "index": 1, "embedding": [1.0, 1.0, 1.0]}, + ], + }, + ) + + vectors = await _jina_client(handler).embed_batch(["a", "b", "c"]) + + assert vectors == [[0.0, 0.1, 0.2], [1.0, 1.0, 1.0], [2.0, 2.0, 2.0]] + assert captured["url"] == "https://api.jina.ai/v1/embeddings" + assert captured["auth"] == "Bearer test-key" + assert captured["body"] == { + "model": "jina-embeddings-v2-base-code", + "input": ["a", "b", "c"], + } + + +async def test_jina_client_batches_and_preserves_order_across_batches() -> None: + seen: list[list[str]] = [] + + def handler(request: httpx.Request) -> httpx.Response: + batch = json.loads(request.content)["input"] + seen.append(batch) + return httpx.Response( + 200, + json={ + "data": [ + {"index": position, "embedding": [float(ord(text))] * 3} + for position, text in reversed(list(enumerate(batch))) + ] + }, + ) + + vectors = await _jina_client(handler, batch_size=2).embed_batch(["a", "b", "c"]) + + assert seen == [["a", "b"], ["c"]] + assert vectors == [[97.0] * 3, [98.0] * 3, [99.0] * 3] + + +async def test_jina_client_rejects_a_dimension_mismatch() -> None: + def handler(request: httpx.Request) -> httpx.Response: + return httpx.Response(200, json={"data": [{"index": 0, "embedding": [1.0, 2.0]}]}) + + with raises(ValueError, match="embedding dimension mismatch"): + await _jina_client(handler, dim=3).embed_batch(["a"]) + + +async def test_jina_client_rejects_a_short_response() -> None: + def handler(request: httpx.Request) -> httpx.Response: + return httpx.Response(200, json={"data": [{"index": 0, "embedding": [1.0, 2.0, 3.0]}]}) + + with raises(RuntimeError, match="1 vectors for 2 inputs"): + await _jina_client(handler).embed_batch(["a", "b"]) + + +async def test_jina_client_rejects_a_duplicated_index() -> None: + def handler(request: httpx.Request) -> httpx.Response: + return httpx.Response( + 200, + json={ + "data": [ + {"index": 0, "embedding": [1.0, 1.0, 1.0]}, + {"index": 0, "embedding": [2.0, 2.0, 2.0]}, + ] + }, + ) + + with raises(RuntimeError, match="duplicate index 0"): + await _jina_client(handler).embed_batch(["a", "b"]) + + +async def test_jina_client_does_not_retry_an_unauthorized_key() -> None: + """401 does not improve by asking again, and a wrong key should say so now.""" + attempts = 0 + + def handler(request: httpx.Request) -> httpx.Response: + nonlocal attempts + attempts += 1 + return httpx.Response(401, json={"detail": "unauthorized"}) + + with raises(httpx.HTTPStatusError): + await _jina_client(handler).embed_batch(["a"]) + assert attempts == 1 + + +async def test_jina_client_retries_rate_limiting(monkeypatch: pytest.MonkeyPatch) -> None: + """429 is the normal free-tier response under load, so it must be retried. + + The sidecar client retries only 5xx, which is right for a service we run + and wrong for a metered one. The backoff sleep is captured rather than + waited out — the claim under test is that it retried and backed off, not + how long a test run takes. + """ + slept: list[float] = [] + + async def no_sleep(delay: float) -> None: + slept.append(delay) + + monkeypatch.setattr(asyncio, "sleep", no_sleep) + attempts = 0 + + def handler(request: httpx.Request) -> httpx.Response: + nonlocal attempts + attempts += 1 + if attempts == 1: + return httpx.Response(429, json={"detail": "slow down"}) + return httpx.Response(200, json={"data": [{"index": 0, "embedding": [1.0, 1.0, 1.0]}]}) + + vectors = await _jina_client(handler).embed_batch(["a"]) + + assert attempts == 2 + assert slept == [2.0] + assert vectors == [[1.0, 1.0, 1.0]] + + +def test_create_embedding_client_builds_the_jina_api_client() -> None: + client = create_embedding_client( + model="jina-embeddings-v2-base-code", + dim=768, + endpoint="https://api.jina.ai/v1", + provider="jina_api", + api_key="test-key", + ) + assert isinstance(client, JinaApiEmbeddingClient) + + +def test_create_embedding_client_requires_a_key_for_the_hosted_provider() -> None: + with raises(ValueError, match="EMBEDDING_API_KEY"): + create_embedding_client( + model="jina-embeddings-v2-base-code", + dim=768, + endpoint="https://api.jina.ai/v1", + provider="jina_api", + api_key=" ", + ) + + +def test_create_embedding_client_rejects_an_unknown_provider() -> None: + with raises(ValueError, match="unknown EMBEDDING_PROVIDER"): + create_embedding_client( + model="jina-embeddings-v2-base-code", + dim=768, + endpoint="https://example.invalid/v1", + provider="not-a-provider", + ) + + +def test_stub_short_circuits_before_the_provider_is_consulted() -> None: + """Stub mode must stay usable with no endpoint and no key, whatever the provider says.""" + client = create_embedding_client(model="stub", dim=4, provider="jina_api") + assert isinstance(client, StubEmbeddingClient) diff --git a/packages/shared/tests/test_reranker.py b/packages/shared/tests/test_reranker.py index 5ea2c1c..3038065 100644 --- a/packages/shared/tests/test_reranker.py +++ b/packages/shared/tests/test_reranker.py @@ -1,6 +1,19 @@ -"""Tests for shared reranker clients.""" +"""Tests for shared reranker clients. -from dcode_shared.reranker import HttpRerankerClient, create_reranker_client +The hosted-provider tests drive a mocked transport. They never call a live API: +CI holds no keys and must keep holding none (Deploy.md §5.3). +""" + +import asyncio +import json + +import httpx +import pytest +from dcode_shared.reranker import ( + HttpRerankerClient, + SiliconFlowRerankerClient, + create_reranker_client, +) from pytest import raises @@ -22,3 +35,148 @@ def test_create_reranker_client_builds_http_client() -> None: endpoint="http://localhost:8003", ) assert isinstance(client, HttpRerankerClient) + + +# --- hosted provider: SiliconFlow --------------------------------------- + + +def _siliconflow_client(handler: object, *, max_retries: int = 3) -> SiliconFlowRerankerClient: + return SiliconFlowRerankerClient( + "https://api.siliconflow.cn/v1", + api_key="test-key", + model="BAAI/bge-reranker-v2-m3", + max_retries=max_retries, + timeout_seconds=1.0, + transport=httpx.MockTransport(handler), # type: ignore[arg-type] + ) + + +async def test_siliconflow_client_returns_scores_in_input_order() -> None: + """The API answers in relevance order; the contract here is input order. + + This is the single thing this client exists to get right. A live probe of + three documents came back as index [1, 0, 2], so the mock reproduces that. + Zipping the response against the input would give every passage another + passage's score and invert the ranking, with nothing to notice it by. + """ + captured: dict[str, object] = {} + + def handler(request: httpx.Request) -> httpx.Response: + captured["body"] = json.loads(request.content) + captured["auth"] = request.headers.get("Authorization") + captured["url"] = str(request.url) + return httpx.Response( + 200, + json={ + "id": "probe", + "meta": {"tokens": {"input_tokens": 289}}, + "results": [ + {"index": 1, "document": None, "relevance_score": 0.64}, + {"index": 0, "document": None, "relevance_score": 0.04}, + {"index": 2, "document": None, "relevance_score": 0.0008}, + ], + }, + ) + + scores = await _siliconflow_client(handler).rerank("q", ["a", "b", "c"]) + + assert scores == [0.04, 0.64, 0.0008] + assert captured["url"] == "https://api.siliconflow.cn/v1/rerank" + assert captured["auth"] == "Bearer test-key" + assert captured["body"] == { + "model": "BAAI/bge-reranker-v2-m3", + "query": "q", + "documents": ["a", "b", "c"], + "return_documents": False, + } + + +async def test_siliconflow_client_returns_empty_without_calling_out() -> None: + def handler(request: httpx.Request) -> httpx.Response: # pragma: no cover + raise AssertionError("no request should be made for an empty passage list") + + assert await _siliconflow_client(handler).rerank("q", []) == [] + + +async def test_siliconflow_client_rejects_a_short_response() -> None: + def handler(request: httpx.Request) -> httpx.Response: + return httpx.Response(200, json={"results": [{"index": 0, "relevance_score": 1.0}]}) + + with raises(RuntimeError, match="1 scores for 2 passages"): + await _siliconflow_client(handler).rerank("q", ["a", "b"]) + + +async def test_siliconflow_client_rejects_an_out_of_range_index() -> None: + def handler(request: httpx.Request) -> httpx.Response: + return httpx.Response(200, json={"results": [{"index": 7, "relevance_score": 1.0}]}) + + with raises(RuntimeError, match="out-of-range index 7"): + await _siliconflow_client(handler).rerank("q", ["a"]) + + +async def test_siliconflow_client_does_not_retry_an_unauthorized_key() -> None: + """A .com key against .cn returns 401. Retrying it three times helps nobody.""" + attempts = 0 + + def handler(request: httpx.Request) -> httpx.Response: + nonlocal attempts + attempts += 1 + return httpx.Response(401, text='"Api key is invalid"') + + with raises(httpx.HTTPStatusError): + await _siliconflow_client(handler).rerank("q", ["a"]) + assert attempts == 1 + + +async def test_siliconflow_client_retries_rate_limiting(monkeypatch: pytest.MonkeyPatch) -> None: + slept: list[float] = [] + + async def no_sleep(delay: float) -> None: + slept.append(delay) + + monkeypatch.setattr(asyncio, "sleep", no_sleep) + attempts = 0 + + def handler(request: httpx.Request) -> httpx.Response: + nonlocal attempts + attempts += 1 + if attempts == 1: + return httpx.Response(429, json={"detail": "slow down"}) + return httpx.Response(200, json={"results": [{"index": 0, "relevance_score": 0.5}]}) + + assert await _siliconflow_client(handler).rerank("q", ["a"]) == [0.5] + assert attempts == 2 + assert slept == [2.0] + + +def test_create_reranker_client_builds_the_siliconflow_client() -> None: + client = create_reranker_client( + model="BAAI/bge-reranker-v2-m3", + endpoint="https://api.siliconflow.cn/v1", + provider="siliconflow", + api_key="test-key", + ) + assert isinstance(client, SiliconFlowRerankerClient) + + +def test_create_reranker_client_requires_a_key_for_the_hosted_provider() -> None: + with raises(ValueError, match="RERANKER_API_KEY"): + create_reranker_client( + model="BAAI/bge-reranker-v2-m3", + endpoint="https://api.siliconflow.cn/v1", + provider="siliconflow", + api_key="", + ) + + +def test_create_reranker_client_rejects_an_unknown_provider() -> None: + with raises(ValueError, match="unknown RERANKER_PROVIDER"): + create_reranker_client( + model="BAAI/bge-reranker-v2-m3", + endpoint="https://example.invalid/v1", + provider="not-a-provider", + ) + + +def test_stub_short_circuits_before_the_provider_is_consulted() -> None: + assert create_reranker_client(model="stub", provider="siliconflow") is None From c5065ae61c43b37742746367bc039fb281aea076 Mon Sep 17 00:00:00 2001 From: odieyang <520yangziqi@gmail.com> Date: Sun, 2 Aug 2026 20:07:15 -0700 Subject: [PATCH 2/2] docs(deploy): record the live cross-path verification of PR 2 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The running stack was switched to both hosted providers and queried against the existing 768-dimension index — the one the local sidecar wrote. That combination is the parity claim itself: a query vector from Jina's API searching vectors produced by the local model. Dense retrieval returned an identical ranking, the same five chunks in the same order, with cosine differing in the third decimal (0.5155 -> 0.5182). Hybrid returned the same five with the top two swapped, and those two were tied at 0.0042 in the baseline; the hosted run broke the tie 0.0043/0.0042. A tie broken differently is not a behaviour change and is not reported as one. The full agent path then reproduced the documented sidecar run citation for citation: sessions.py:186 and sessions.py:557, both verified, groundedness 1.0, three tool calls. Hosted BGE also returns scores in the same magnitude band as the local BGE sidecar, where the Qwen reranker on SiliconFlow's other platform returned two orders of magnitude higher on a comparable query — circumstantial, but it is the kind of evidence available. The third-decimal drift is the caveat §2 already recorded: same weights, different serving stack, not bit-identical. It is visible and it does not move the ranking. Two gaps are named rather than glossed. No index has been built through the hosted provider, so the bulk-embedding path — 23 requests, rate limiting, the embed stage's Redis cache interaction — is still unproven; that is deliberate, because re-indexing would replace the verified reference index, and it will happen at the first Railway index. And the sidecar path was not re-run live after the switch: its code is untouched and its selection is covered by tests, but that is a weaker claim than having watched it work, so it is stated as the weaker one. --- Deploy.md | 47 ++++++++++++++++++++++++++++++++++++----------- 1 file changed, 36 insertions(+), 11 deletions(-) diff --git a/Deploy.md b/Deploy.md index c1a0ed7..d8669e3 100644 --- a/Deploy.md +++ b/Deploy.md @@ -25,11 +25,11 @@ Outstanding Work list in the same PR, not tracked in two places. is written from an approved plan, and plans outrun implementations. Verify before relying on a claim here, and correct it when it is wrong. -**Status: PR 1 landed. PR 2 code complete, its pipeline run still pending. -PR 3–5 not started.** The pre-deployment state is frozen at tag -`v1.0-submission` (`a4612b8`). Nothing is deployed anywhere yet. Both hosted -model APIs have now been called from this codebase and the observed results are -in § 2.1 and § 6 PR 2, but no index has yet been built through them. +**Status: PR 1 and PR 2 landed. PR 3–5 not started.** The pre-deployment state +is frozen at tag `v1.0-submission` (`a4612b8`). Nothing is deployed anywhere +yet. Both hosted model APIs have been exercised end to end from a running local +stack — see § 2.1 and § 6 PR 2 for the observed values — but **no index has been +built through them**, so the indexing side of the hosted path is still unproven. --- @@ -324,7 +324,7 @@ rather than defaulted, because a wrong value there is a re-index, not a slow que of the above, and **each was confirmed to fail when its defect is put back** — a config test that has never been seen failing is not evidence of anything. -### PR 2 — `feat/managed-model-providers` — **code complete, pipeline run pending** +### PR 2 — `feat/managed-model-providers` — **done, with one gap named below** The capability behind D-1. @@ -368,11 +368,36 @@ The capability behind D-1. | Cross-batch ordering | re-embedding input 2 alone lands `0.000000` from its batched slot; nearest other slot is `0.586` | | `SiliconFlowRerankerClient`, 3 passages | scores returned in input order; argmax is the `HTTPBasicAuth` passage for a credentials question | - **What is not done: the full indexing pipeline has not been run through the - hosted provider.** 726 chunks at batch 32 is roughly 23 requests, which is - where rate limiting and the embed stage's Redis cache interaction would show - up, and neither has been exercised. Until that runs, this PR proves the - clients are correct, not that an index built through them is. + The **running stack** was then switched to both hosted providers + (`docker compose up -d --build api agent`, since the images are not + live-mounted) and queried against the existing 768-dimension index — the one + the *sidecar* wrote. That combination is the parity claim under test: a query + vector from Jina's API searching vectors produced by the local model. + + | Check | Result | + |---|---| + | `mode=dense`, the zero-keyword-overlap query from Operations.md | **identical ranking**, same five chunks in the same order; cosine differs in the third decimal (0.5155 → 0.5182) | + | `mode=hybrid` | same five chunks; the top two swap, and they were **tied at 0.0042** in the baseline — the hosted run broke the tie 0.0043/0.0042. A tie broken differently is not a behaviour change | + | Reranker identity | hosted BGE returns scores in the same magnitude band as the local BGE sidecar (~0.004). The Qwen reranker on SiliconFlow's other platform returned ~0.64 on a comparable query, so the band is circumstantial evidence that the model is the one asked for | + | Full agent path, `POST /api/v1/query` | 3 tool calls (`search_code` → `get_call_neighbors` → `read_file`), 2 citations, **both verified**, groundedness `1.0` | + | Against the documented run | reproduces Operations.md *Verified Current Path — 2026-07-30* citation for citation: `sessions.py:186` and `sessions.py:557` | + + The third-decimal drift is exactly the caveat § 2 records: same weights, + different serving stack, not bit-identical. It is visible and it does not + move the ranking. + + **What is still not done: no index has been built through the hosted + provider.** The verification above is query-side. 726 chunks at batch 32 is + roughly 23 requests, which is where rate limiting and the embed stage's Redis + cache interaction would appear, and neither has been exercised. That is + deliberate — re-indexing would replace the verified reference index — and it + will happen naturally at the first Railway index. Until then this PR proves + the clients are correct and cross-compatible with the sidecar, not that a + bulk index built through them is. + + The sidecar path was not re-run live after the switch. Its code is untouched + and its factory selection is covered by tests, but that is a weaker claim + than having watched it work, and it is stated as the weaker one. 3. **Met.** `make check` green: ruff, eslint, the drift check, mypy strict over 83 files, 310 passed / 5 skipped, 73 frontend tests. All four modes in § 5.1