Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,9 @@ EMBEDDING_DIM=1024
# EMBEDDING_ENDPOINT=http://host.docker.internal:8002
# EMBEDDING_BATCH_SIZE=1
# EMBEDDING_MAX_SEQ_LENGTH=1024 # caps Jina token window to avoid OOM (0 = disable)
# EMBEDDING_MAX_RETRIES=12 # sized for a cold CPU sidecar, not for an API
# EMBEDDING_TIMEOUT_SECONDS=300 # ditto — their product is the worst-case stall,
# # and the worker consumes one job at a time
# On MacBook: run `make embedding-host` in a separate terminal (recommended).
# Or Docker sidecar: docker compose --profile embedding up -d embedding
# If you change EMBEDDING_DIM after first migrate: make down-all && make migrate
Expand Down
83 changes: 73 additions & 10 deletions .env.production.example
Original file line number Diff line number Diff line change
@@ -1,8 +1,15 @@
# ============================================================
# Dcode production environment variables.
# Copy to `.env.production` on the deployment host and review
# every placeholder before `docker compose -f docker-compose.yml
# -f docker-compose.prod.yml up -d --build`.
#
# Copy to `.env.production` on the deployment host and review every
# placeholder before:
#
# docker compose --env-file .env.production -f docker-compose.prod.yml up -d --build
#
# `docker-compose.prod.yml` is a COMPLETE stack, not an overlay. Do not
# combine it with `-f docker-compose.yml`: the developer file publishes
# Postgres, Redis and the RabbitMQ management UI to the host, and overlaying
# would republish them on a public machine.
# ============================================================

# --- Public entrypoint ---
Expand All @@ -13,26 +20,82 @@ PUBLIC_ORIGIN=http://dcode.odieyang.com
POSTGRES_USER=dcode
POSTGRES_PASSWORD=__SET_PROD_POSTGRES_PASSWORD__
POSTGRES_DB=dcode
DATABASE_URL=postgresql+asyncpg://dcode:__SET_PROD_POSTGRES_PASSWORD__@postgres:5432/dcode

# --- Internal services ---
REDIS_URL=redis://redis:6379/0
RABBITMQ_USER=__SET_PROD_RABBITMQ_USER__
RABBITMQ_PASSWORD=__SET_PROD_RABBITMQ_PASSWORD__

# The compose file builds DATABASE_URL / REDIS_URL / RABBITMQ_URL / AGENT_URL
# from the values above and the internal service names, so these four are NOT
# read by the running stack. They are here for commands run against this env
# file outside compose (an alembic invocation on the host, for example). If you
# change a credential above, change it here too or leave these out entirely.
DATABASE_URL=postgresql+asyncpg://dcode:__SET_PROD_POSTGRES_PASSWORD__@postgres:5432/dcode
REDIS_URL=redis://redis:6379/0
RABBITMQ_URL=amqp://__SET_PROD_RABBITMQ_USER__:__SET_PROD_RABBITMQ_PASSWORD__@rabbitmq:5672/
AGENT_URL=http://agent:8001

# --- Internal service auth ---
# Protects every /internal/* retrieval and graph route. The code default is a
# placeholder published in this repository, so this must be set to a real
# secret — the compose file refuses to start without it.
INTERNAL_API_KEY=__SET_PROD_INTERNAL_API_KEY__

# --- Model / runtime knobs ---
# Production defaults keep model serving disabled. If production should use
# real embedding/reranker services, add those services to the deployment and
# set endpoint, model, and EMBEDDING_DIM consistently before indexing.
LOG_LEVEL=info

# ============================================================
# Retrieval models
#
# These are read by api, worker AND agent, and they must AGREE. The API embeds
# the search query, the worker embeds the corpus, and the agent reranks its own
# evidence union. Configuring one service and not another produces no error —
# it produces real vectors in the database being searched with a stub all-zero
# query vector, i.e. dense retrieval silently returning noise.
# ============================================================

# EMBEDDING_DIM is fixed into the pgvector column by the migration and cannot
# be changed afterwards without a new volume. It has no default in production
# for that reason: a wrong value here is not a slow query, it is a re-index.
# stub = 1024 zero vectors; jinaai/jina-embeddings-v2-base-code = 768.
EMBEDDING_MODEL=stub
EMBEDDING_DIM=1024
EMBEDDING_ENDPOINT=

# Sized for a cold self-hosted sidecar. 12 retries against a 300s timeout can
# hold one batch of four chunks for about an hour, and the worker consumes one
# job at a time, so everything queued behind it waits. Lower both substantially
# when the endpoint is a hosted API rather than a model loading on CPU.
# EMBEDDING_BATCH_SIZE=4
# EMBEDDING_MAX_RETRIES=12
# EMBEDDING_TIMEOUT_SECONDS=300

# Leave RERANKER_ENDPOINT empty unless RERANKER_MODEL is a real model. A
# non-stub model with an empty endpoint fails fast and loudly, which is the
# intended behaviour; a stub model ignores the endpoint entirely.
RERANKER_MODEL=stub
RERANKER_ENDPOINT=http://localhost:9999
RERANKER_ENDPOINT=
# RERANKER_CANDIDATE_LIMIT=16
# RERANKER_MAX_RETRIES=3

# Hybrid RRF fusion weights. Dense above sparse reduces keyword noise.
# RRF_DENSE_WEIGHT=2.0
# RRF_SPARSE_WEIGHT=1.0

# ============================================================
# Answer synthesis
# ============================================================

# 'stub' keeps rule-based template answers. Set an OpenAI-compatible model and
# key for prose answers; citations are verified against the index either way.
SYNTHESIS_MODEL=stub
OPENAI_API_KEY=
OPENAI_BASE_URL=
# SYNTHESIS_MAX_TOKENS=700
# SYNTHESIS_TEMPERATURE=0.1

# Keep above 0 in production so the groundedness guardrail stays enforced.
# GROUNDEDNESS_THRESHOLD=0.95

# Defined and passed through, but read by no code today. Left in place rather
# than removed, because removing dead configuration is a product decision.
JUDGE_MODEL=stub
60 changes: 54 additions & 6 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,12 @@ rules, and how to not break things.
**Read this, then `git log --oneline`.** The work is committed in small scoped
slices and the messages carry the reasoning.

Project documentation is five files, all authoritative over this one:
Project documentation is six files, all authoritative over this one:

| Document | Read it for |
|---|---|
| [`docs/en/Final_Report.md`](docs/en/Final_Report.md) | The H1 verdict, **the single outstanding-work list**, known limits, what was verified |
| [`Deploy.md`](Deploy.md) | **Everything about taking this online.** Decisions, verified platform constraints, the invariants, the PR breakdown, the findings register — see §2 below |
| [`docs/en/Honesty_Constraints.md`](docs/en/Honesty_Constraints.md) | The eleven rules governing what the UI may assert, with reasoning. Most are pinned by tests — read before touching the thread, the chips, or the inspector |
| [`docs/en/Technical_Design.md`](docs/en/Technical_Design.md) | Repository layout, architecture, data model, API contracts |
| [`docs/en/Operations.md`](docs/en/Operations.md) | Running the stack, the real-model path, the eval harness, operational gotchas |
Expand Down Expand Up @@ -56,7 +57,54 @@ its numbers are generated rather than translated.
flows, and write drafts to files rather than printing them inline. Pure
frontend work runs clean.

## 2. Where things are
## 2. Deployment work is governed by `Deploy.md` — follow it strictly

If the task touches deployment, hosting, model providers, auth, or production
configuration, **read [`Deploy.md`](Deploy.md) first and execute against it.** It
carries the decisions, the verified Railway constraints, the five-PR breakdown
with binding acceptance criteria, and a findings register where every row has a
`file:line`. Do not re-derive that plan, and do not improvise around it.

**The backlog is not copied here.** `Deploy.md` §6 and §7 are the single list;
duplicating them into this file is the exact drift pattern this project keeps
having to fix. What is repeated below is only the set of invariants — the things
that stay true no matter which PR is in flight.

1. **The deployed models must be the evaluated models.**
`jinaai/jina-embeddings-v2-base-code` at 768 dimensions, `BAAI/bge-reranker-v2-m3`,
`gpt-4o-mini`. Every figure the UI and the docs display is generated from
`results/eval-h1-repeat3-2026-07-31/` and is a claim about *that* configuration.
Substituting a model without saying so on the landing page and `/methodology`
is the same class of error as hand-typing a figure into prose. `Deploy.md` §2.
2. **The local multi-container path must not regress.** All four modes keep
working: stub, Docker sidecars, host sidecars, and the new API-provider mode.
`apps/embedding/`, `apps/reranker/`, the Compose profiles, the host scripts,
`HttpEmbeddingClient` and `HttpRerankerClient` are untouched. In particular the
agent+worker merge is a **Railway-only entrypoint** — Railway cannot share a
volume between services and local Docker can, so `docker-compose.yml` keeps two
separate containers. Do not degrade the local path to match the constrained
one. `Deploy.md` §5.1.
3. **The provider switch is additive.** `EMBEDDING_PROVIDER` / `RERANKER_PROVIDER`
default to `sidecar`; existing `.env` files and tests keep their behaviour with
no edit. `Deploy.md` §5.2.
4. **CI stays keyless.** New client tests mock the transport; they never call a
live API. `Deploy.md` §5.3.
5. **Embedding configuration must match on the API and the worker.** Setting it on
one only produces no error — it produces real vectors in the database searched
with a stub all-zero query vector, i.e. dense retrieval silently returning
noise. `Deploy.md` §5.4.
6. **No fork, no long-lived divergent branch.** `Deploy` merges to `main` as small
PRs, each independently green under `make check`. Secrets never enter the
repository. `Deploy.md` §5.5.
7. **`docker-compose.prod.yml` is fixed, not deleted.** `Final_Report.md` lists it
as delivered. `Deploy.md` §5.6.

The pre-deployment state is frozen at tag `v1.0-submission` (`a4612b8`). Two
Railway constraints are load-bearing and were verified on 2026-08-02: a volume
attaches to exactly one service, and an HTTP request is cut after 5 minutes with
no data (~15 minutes with heartbeats).

## 3. Where things are

```
apps/frontend/ React 18 + TS strict + Vite + Tailwind + Router v6 + TanStack Query (~4.2k TS/TSX lines)
Expand Down Expand Up @@ -101,7 +149,7 @@ which matches `dcode_shared.events.sse_encode` emitting pure LF.
Two data-fetching paradigms coexist deliberately: TanStack Query for status
polling, hand-rolled streaming for the query. That is not an inconsistency.

## 3. Visual identity — deliberate, do not "improve"
## 4. Visual identity — deliberate, do not "improve"

Tokens are a CSS-vars layer in `apps/frontend/src/index.css`, with
`tailwind.config.ts` pointing at the vars so `color-mix` translucency still works.
Expand Down Expand Up @@ -134,7 +182,7 @@ let it fill the `1fr` column: serif prose at 100+ characters per line is
unreadable, and on a wide monitor a full-bleed column reads as an empty void. The
rails (262px left, 384px right) are correct as-is.

## 4. Current state
## 5. Current state

Everything below is committed and green: `make check`, `make frontend-build`,
71 frontend tests, full pytest suite.
Expand Down Expand Up @@ -199,7 +247,7 @@ duplicate `psf/requests` rows already in the local DB are not cleaned up
retroactively by the `POST /repos` idempotency fix, and `make down-all` wipes the
volumes, which means re-indexing at whatever `EMBEDDING_DIM` you migrate at.

## 5. Environment
## 6. Environment

```bash
# All three are required — .env points the model endpoints at host.docker.internal,
Expand Down Expand Up @@ -242,7 +290,7 @@ vectors — real, varied embeddings, not stubs. Inspector line numbers and
call-graph neighbours were manually verified against this index and matched
exactly.

## 6. Working agreements that produced good results
## 7. Working agreements that produced good results

- **Where this file and the code disagree, the code is the truth.** These notes
are written from the approved intent of a session, and intent occasionally
Expand Down
Loading
Loading