Skip to content

fix(deploy): give every service the retrieval settings it reads - #19

Merged
Odiethebest merged 3 commits into
mainfrom
fix/prod-compose-model-env
Aug 3, 2026
Merged

Odiethebest merged 3 commits into
mainfrom
fix/prod-compose-model-env

Conversation

@Odiethebest

Copy link
Copy Markdown
Owner

Stacked on #18. Until that merges, this PR also shows its two documentation commits; afterwards the diff reduces to the single fix commit here.

Closes F-02, F-11, F-12 and half of F-13 in Deploy.md §7.

The defect

The production compose file configured almost none of the retrieval stack, and every failure mode is silent.

api had no embedding or reranker configuration at all. Two consequences:

  • make prod-migrate runs alembic inside that container, and 001_initial_schema reads shared_settings.embedding_dim to size the pgvector column. Production therefore built a vector(1024) column whatever .env.production said, while the worker inserted at its own configured dimension.
  • api is the service that embeds the search query and reranks candidates. With no configuration it ran the stub path: an all-zero query vector against real stored vectors. That is dense retrieval returning noise, with nothing in the log.

The other two were short as well — this is wider than the finding as originally written, and Deploy.md records the correction:

Service Was missing Effect
worker EMBEDDING_ENDPOINT, EMBEDDING_BATCH_SIZE, EMBEDDING_MAX_RETRIES a model and a dimension, but no way to reach a service
agent RERANKER_MODEL, RERANKER_MAX_RETRIES, EMBEDDING_DIM, synthesis knobs RERANKER_ENDPOINT alone selects nothing — create_reranker_client returns None for a stub model whatever the endpoint says — so the agent's evidence-union rerank was identity ordering in production

EMBEDDING_DIM is now required in production

Not defaulted. It is not only the migration that depends on it: dcode_shared/db/models.py:128 binds the chunks.embedding column type at import time, so every service touching the ORM needs it correct. A wrong value is a re-index, not a slow query, and that is not something to inherit from a default that happens to suit stub vectors.

Also here

  • The dead RERANKER_ENDPOINT=http://localhost:9999 default is replaced by an empty one, which fails loudly for a real model and is ignored by a stub.
  • The two documented compose invocations are reconciled on the standalone form. The overlay form the env example documented republishes Postgres, Redis and the RabbitMQ management UI to the host.
  • EMBEDDING_TIMEOUT_SECONDS becomes env-tunable instead of hardcoded. The value is unchanged. Twelve retries against a 300s timeout can hold one batch for about an hour while prefetch_count=1 stalls the queue behind it — but those defaults are correct for a cold CPU sidecar, and the path they are wrong for is the hosted API, which does not exist until PR 2. PR 2 sets them.

Tests

Three new tests in packages/shared/tests/test_config_hardening.py, and each was confirmed to fail when its defect is put back. A config test that has never been seen failing is not evidence of anything.

They parse the compose file by hand rather than importing PyYAML, which is only a transitive dependency here and is imported by no other code in the repository.

Verified / not verified

Verified: docker compose config resolves 12 retrieval variables on api, 6 on worker, 9 on agent, mutually consistent; omitting EMBEDDING_DIM aborts with the reason attached; make check green — ruff, eslint, the eval-artifact drift check, mypy strict over 83 files, 289 passed / 5 skipped, 73 frontend tests.

Not verified: no container was started, the production stack was not run, and no live model API was called. This round is configuration resolution and unit tests only.

Deploy.md is the authority for taking this online. It records the three
decisions taken (real models via hosted APIs, a login gate between landing
and workbench, Railway), the platform constraints that were verified rather
than assumed, the invariants this workstream may not break, five PRs with
binding acceptance criteria, and a findings register.

Two things about the register are deliberate. Every row carries a file:line
so it can be checked instead of believed, and each row is marked `direct`
(confirmed in-session by reading the file or resolving the compose config)
or `audit` (reported with a citation by a code-audit pass, not
independently re-checked). Treating those two as the same kind of claim is
how a plan acquires facts nobody verified.

The register is not a second backlog. Final_Report.md keeps the single
outstanding-work list for the product; Deploy.md covers only what blocks
deployment, and where an item is both it gets struck from Outstanding Work
in the PR that fixes it.

The constraint that outranks the rest is model parity: every figure the UI
and the docs display is generated from results/eval-h1-repeat3-2026-07-31/
and is a claim about that exact configuration. All three of its models are
available as hosted APIs, so the deployment can match it without
substituting anything — and if a future change substitutes one, it says so
on the landing page and /methodology or it reverts.

CLAUDE.md gets a pointer, not a copy: §2 carries only the seven invariants
that stay true whichever PR is in flight, and says explicitly that the
backlog lives in Deploy.md. Three copies of "what's left" is the drift
pattern this project keeps having to fix. Sections renumbered 2-6 -> 3-7;
the two internal §1 references still resolve.
D-5: explicit per-provider client classes rather than a generic
OpenAI-compatible adapter. Neither Jina's embeddings response nor
SiliconFlow's rerank response is OpenAI-shaped in the part that matters —
both carry per-item indices that have to be mapped back to input order — so
a shared abstraction would have to model the difference anyway. The cost is
recorded: a fourth provider later means writing a class, not setting a
variable.

D-6: one shared account read from the environment. No user table, no
migration. The consequences are written down rather than discovered later:
the daily query cap becomes per session rather than per person, and a
shared password cannot be revoked for one holder, only rotated. That is
adequate for a gated demo and is not an authorization model.

Open questions drop from five to two. Team ownership stays unresolved and
is the reason this branch is being pushed; the durable fix for the Railway
shared-volume constraint stays deferred and recorded.
The production compose file configured almost none of the retrieval stack,
and the failure mode is silence rather than an error.

The api service had no embedding or reranker configuration at all. Two
consequences. `make prod-migrate` runs alembic inside that container, and
001_initial_schema reads shared_settings.embedding_dim to size the pgvector
column — so production built a vector(1024) column whatever .env.production
said, while the worker inserted at its own configured dimension. And the
api is the service that embeds the search query and reranks candidates, so
it silently ran the stub path: an all-zero query vector against real stored
vectors, which is dense retrieval returning noise with nothing in the log.

The other two services were short as well. The worker had a model and a
dimension but no EMBEDDING_ENDPOINT, so it could not reach a service. The
agent had RERANKER_ENDPOINT alone, which selects nothing — create_reranker_
client returns None for a stub model regardless of the endpoint — so the
evidence-union rerank the agent performs was identity ordering in
production.

EMBEDDING_DIM is now required in production rather than defaulted. It is
not only the migration that depends on it: dcode_shared.db.models binds the
chunks.embedding column type at import time, so every service touching the
ORM needs it right. A wrong value is a re-index, not a slow query, and that
is not a thing to inherit from a default that happens to suit stub vectors.

Also here: the dead RERANKER_ENDPOINT=http://localhost:9999 default is
replaced by an empty one, which fails loudly for a real model and is
ignored by a stub; the two documented compose invocations are reconciled on
the standalone form, because the overlay form the env example documented
republishes Postgres, Redis and the RabbitMQ UI to the host.

EMBEDDING_TIMEOUT_SECONDS becomes env-tunable instead of hardcoded at 300s.
The value is unchanged. Twelve retries against a 300s timeout can hold one
batch for about an hour while prefetch_count=1 stalls the queue behind it,
but those defaults are correct for a cold CPU sidecar; the path they are
wrong for is the hosted API, which does not exist until PR 2. PR 2 sets
them.

Three tests pin all of this, and each was confirmed to fail when its defect
is put back. They parse the compose file by hand rather than importing
PyYAML, which is only a transitive dependency here and is imported by no
other code in the repository.

Deploy.md records a correction: F-02 was written as "the api service has no
embedding configuration" and was wider than that. The register and the
acceptance criteria are updated from what was observed, not from what was
planned.
@Odiethebest
Odiethebest merged commit ca18b4a into main Aug 3, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant