diff --git a/CLAUDE.md b/CLAUDE.md index f37f45a..7ffe409 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -99,10 +99,16 @@ that stay true no matter which PR is in flight. 7. **`docker-compose.prod.yml` is fixed, not deleted.** `Final_Report.md` lists it as delivered. `Deploy.md` §5.6. -The pre-deployment state is frozen at tag `v1.0-submission` (`a4612b8`). Two -Railway constraints are load-bearing and were verified on 2026-08-02: a volume -attaches to exactly one service, and an HTTP request is cut after 5 minutes with -no data (~15 minutes with heartbeats). +**A gated deployment is live** (Railway, HTTPS, one shared account, the same +three models the recorded run measured). The pre-deployment state is frozen at +tag `v1.0-submission` (`a4612b8`). + +Four things failed on the first deploy and all four were wrong assumptions in +the plan, not platform faults — `Deploy.md` §10.3 has them. The two worth +carrying in your head, because they present as something else entirely: a volume +mounted under `/tmp` fails the deployment **with no log output at all**, and +`HOST=::` gives uvicorn an IPv6-only socket while the health check arrives over +IPv4, so you get "never became healthy" beside a log showing a healthy server. ## 3. Where things are @@ -313,4 +319,17 @@ exactly. that would have been p-hacking. Do that again. - **Be honest about what you could not verify** — no screenshots, no end-to-end from the shell — and say explicitly what a human needs to eyeball. +- **`make check` green and CI green are two different claims. Check CI after + every merge, and say which one you are reporting.** This was learned the + expensive way: main's CI was red for five merged PRs while every report said + "make check green". Both statements were true. The local run passes because a + developer's `.env` supplies a real `INTERNAL_API_KEY` and a Postgres is + running; CI has neither, which is the entire point of CI. A test that quietly + depends on the machine it runs on is invisible locally by construction, so + local success is evidence about your laptop and not about the change. + + `gh run list --branch main --limit 3` after a merge. When something only fails + in CI, reproduce it before fixing it — moving `.env` aside and pointing + `DATABASE_URL` at a dead port is the closest local approximation, and it turns + a guess into a measurement. - **Never fabricate data or fake success.** Verified means verified. diff --git a/Deploy.md b/Deploy.md index 891bad4..1cf8969 100644 --- a/Deploy.md +++ b/Deploy.md @@ -25,8 +25,10 @@ Outstanding Work list in the same PR, not tracked in two places. is written from an approved plan, and plans outrun implementations. Verify before relying on a claim here, and correct it when it is wrong. -**Status: PR 1–5 landed. Nothing is deployed yet** — every remaining item is -in § 10.2 and needs a Railway account. The pre-deployment state +**Status: deployed and verified, 2026-08-03.** All five PRs landed and the stack +is live at [`frontend-production-55c4.up.railway.app`](https://frontend-production-55c4.up.railway.app). § 10 records what was checked +and how; § 10.3 records the four things that failed first, all of them mistakes +in this plan rather than in the platform. The pre-deployment state is frozen at tag `v1.0-submission` (`a4612b8`). Nothing is deployed anywhere yet. Both hosted model APIs have been exercised end to end from a running local stack — see § 2.1 and § 6 PR 2 for the observed values — but **no index has been @@ -387,14 +389,11 @@ The capability behind D-1. different serving stack, not bit-identical. It is visible and it does not move the ranking. - **What is still not done: no index has been built through the hosted - provider.** The verification above is query-side. 726 chunks at batch 32 is - roughly 23 requests, which is where rate limiting and the embed stage's Redis - cache interaction would appear, and neither has been exercised. That is - deliberate — re-indexing would replace the verified reference index — and it - will happen naturally at the first Railway index. Until then this PR proves - the clients are correct and cross-compatible with the sidecar, not that a - bulk index built through them is. + **Closed 2026-08-03: an index has been built through the hosted provider**, + on the deployment. 773 chunks in roughly 24 requests to Jina's API, every one + `200 OK`, **no retries and no rate limiting**. Read back from the deployed + database: all 773 vectors at 768 dimensions, **zero all-zero vectors**, and + 770 distinct prefixes across 773 rows — real, varied embeddings, not stubs. The sidecar path was not re-run live after the switch. Its code is untouched and its factory selection is covered by tests, but that is a weaker claim @@ -487,15 +486,17 @@ done.** Everything below was observed; the row that needs Railway needs Railway. | Heartbeats and the request budget | 5 tests, including that `close()` still ends a stream while the heartbeat loop runs, and that an overrun emits `TIMEOUT` and **no** `final_answer` | | The local stack is unchanged | `docker compose up -d --build api` still healthy; `make check` green | -**Not done: no Railway deployment exists.** The runbook is § 10.2, written from -the code rather than from a completed deploy — read it as intended sequence and -correct it against what happens. These are unverified until then: private -networking over IPv6, the platform's request limits against real heartbeats, the -pgvector template, and the first hosted-provider index (§ 6 PR 2). +**Deployed 2026-08-03.** § 10.2 is corrected against what happened rather than +written from the code. The combined `agent-worker` container runs, and its +`wait -n` behaviour was confirmed the hard way: it exited with `a child exited +with 1; stopping the other` when the broker was unreachable, rather than serving +an agent beside a dead worker. -The combined `agent-worker` container has **not been run**, locally or -otherwise. Its entrypoint is Railway-only and the developer stack does not use -it, so nothing here exercises it. +Two things this PR assumed that turned out backwards, both in § 10.3: private +networking works over **IPv4**, and `HOST=::` is what broke the health check. +One thing remains unobserved — the R-2 heartbeat. Every live query so far +finished well inside the platform's idle window, so the keep-alive is +correct-by-construction and has never been needed. ### PR 5 — `fix/audit-followups` — **done** @@ -680,17 +681,23 @@ neither is a reason to soften a report: Before calling the deployment done: -| # | Check | How | +Checked against the live deployment on 2026-08-03. + +| # | Check | Result | |---|---|---| -| 1 | Vectors in the deployed database are 768-dimensional and non-zero | `SELECT vector_dims(embedding), left(embedding::text, 20) FROM chunks LIMIT 1;` | -| 2 | The reranker is actually reordering, not passing through | Compare `score_components` on a query with and without rerank | -| 3 | `api` and `agent-worker` agree on embedding configuration | § 5.4 readiness check reports ready | -| 4 | A query streams to completion without being cut | Watch a full run; confirm no truncation at 5 minutes (R-2) | -| 5 | Citations are verified and clickable, and open real indexed source at the cited line | Human, in a browser | -| 6 | Call-graph neighbours are clickable and chain | Human, in a browser | -| 7 | Unauthenticated access to `/workbench` and to each protected route is refused | Private window | -| 8 | `/` and `/methodology` load without a session, and their figures match `results/` | `python3 scripts/sync_eval_artifacts.py --check` | -| 9 | The four local modes in § 5.1 all still start | Locally, after the final PR | +| 1 | Vectors in the deployed database are 768-dimensional and non-zero | **verified** — 773/773 at `dims=768`, 0 all-zero, 770 distinct prefixes | +| 2 | The reranker is actually reordering, not passing through | **not verified on the deployment.** `/internal/*` is not publicly routed, so the two-mode comparison cannot be run from outside. It was verified locally in § 6 PR 2 | +| 3 | `api` and `agent-worker` agree on embedding configuration | **verified** — `/readyz` returns 200 with all three checks green | +| 4 | A query streams to completion without being cut | **verified** for a normal query; the heartbeat path is unobserved because nothing has yet run long enough to need it | +| 5 | Citations are verified and clickable, and open real indexed source at the cited line | **verified by a human in a browser** | +| 6 | Call-graph neighbours are clickable and chain | **verified by a human in a browser** | +| 7 | Unauthenticated access to `/workbench` and to each protected route is refused | **verified** — 401 on every protected route; `/workbench` redirects to `/login` | +| 8 | `/` and `/methodology` load without a session, and their figures match `results/` | **verified** — both 200 unauthenticated, drift check green | +| 9 | The four local modes in § 5.1 all still start | **verified** — compose stack healthy after the final PR | + +One live query end to end: 3 tool calls, **3 citations all verified**, +groundedness **1.0**, reproducing the two citations the local run found and +adding a third. --- @@ -790,9 +797,8 @@ confirms you are back. ### 10.2 Railway runbook -Written from the code, not from a completed deployment — **nothing has been -deployed yet** (§ 6 PR 4). Treat it as the intended sequence, and correct it -against what actually happens. +Corrected against a completed deployment on 2026-08-03. The four things that +failed the first time are § 10.3; each is called out below where it bites. **The configuration is committed.** `infra/railway/{api,agent-worker,frontend}.toml` carry the build and deploy settings, and @@ -857,6 +863,51 @@ rebinding is not closed without it or network egress filtering. --- +### 10.3 What failed on the first deploy + +Four failures, in the order they were found. **All four were mistakes in this +plan, not in the platform** — three of them were things I had written from +documentation and not measured. + +**1. `uv run` was installing dependencies at container start.** `api` spent +twelve minutes and then failed with `Failed to spawn: alembic`. Two compounding +causes: `uv sync --no-dev` at a workspace root syncs the *root* project, and the +root declares no dependencies, so the build step installed nothing — the images +had two entries in `site-packages`. Everything was really installed by `uv run` +at start, and `uv run` without `--no-dev` downloads `mypy` and `ruff` first. +Fixed by `uv sync --no-dev --package ` and calling the venv binaries +directly; `uv run` is now gone from `infra/` entirely. `apps/embedding` had +already done this, with a comment saying why. + +**2. A volume cannot be mounted under `/tmp`.** Railway refuses and fails the +deployment **before emitting a single log line** — no build log, no deploy log, +just `FAILED`, which is indistinguishable from any other cause. This took the +longest to find and the only usable signal was that `agent-worker` was the one +app service with a volume. Moved to `/data/workdirs`. + +**3. `HOST=::` produces an IPv6-only socket.** Measured, after assuming +otherwise: uvicorn given `::` refuses an IPv4 connect even with +`net.ipv6.bindv6only=0`, and given `0.0.0.0` refuses IPv6. There is no +dual-stack option. Railway's health check arrives over **IPv4**, so the probe +never reached the application and the deploy failed with `1/1 replicas never +became healthy` beside a log showing a perfectly started server. R-4's advice to +bind `::` applies to environments created before 2025-10-16; a current one +resolves private DNS to both families, so `0.0.0.0` serves both purposes. + +**4. `PORT` must be set explicitly.** Railway injects `8080`, the services +correctly honour it, and every private-network URL in this plan said `:8000` — so +nginx returned `502` for `/api/*` while both services were healthy. Pinning +`8000` and `8001` makes the other services' URLs right by construction, and +matches compose, which is what anyone debugging will compare against. + +**The pattern worth taking from this**: every one of these presented as a +different problem than it was. A twelve-minute build that was a deploy. A build +failure with no build. A healthy server that was unreachable. A 502 from a +service that was up. In each case the log that mattered was one layer away from +the one being read. + +--- + ## 11. Open Questions Carried here rather than answered by assumption. diff --git a/README.md b/README.md index 4734539..e2dd1bb 100644 --- a/README.md +++ b/README.md @@ -335,10 +335,18 @@ embedding and reranker configuration, and they must agree. `EMBEDDING_DIM` has no production default on purpose: the migration bakes it into the pgvector column, so a wrong value is a re-index rather than a slow query. -The production stack is validated locally only — `dcode.odieyang.com` does **not** -resolve publicly, so the external-demo exit criterion is still open. The plan for -closing it, including the platform constraints that shape it, is -[`Deploy.md`](Deploy.md). +**A gated deployment is live** at [`frontend-production-55c4.up.railway.app`](https://frontend-production-55c4.up.railway.app) — +Railway, HTTPS, one shared account, running the same three models the recorded +evaluation measured. The landing page and `/methodology` are public; the +workbench is not. Access is by request. + +The deployment indexes `psf/requests` at current `HEAD`, which is **not** the +commit the H1 verdict was measured on — the verdict is a statement about the +committed runs in [`results/`](results/), not about what the demo is serving. + +Service configuration, the variable sheet, and the four things that failed on +the first attempt: [`Deploy.md`](Deploy.md) and +[`infra/railway/`](infra/railway/). --- diff --git a/apps/api/tests/test_auth.py b/apps/api/tests/test_auth.py index 473b434..8d7d4c3 100644 --- a/apps/api/tests/test_auth.py +++ b/apps/api/tests/test_auth.py @@ -20,7 +20,7 @@ read_session, verify_password, ) -from dcode_api.deps import get_agent_client, get_redis +from dcode_api.deps import get_agent_client, get_db, get_redis from dcode_api.main import app, create_app from dcode_api.settings import api_settings from fastapi.testclient import TestClient @@ -254,10 +254,18 @@ def test_login_sets_a_hardened_cookie_and_opens_the_gate(gated: TestClient) -> N # The client keeps the cookie, so a previously-401 route now resolves. assert gated.get("/api/v1/auth/me").json()["authenticated"] is True - assert ( - gated.get("/api/v1/repos/00000000-0000-0000-0000-000000000000/status").status_code - != 401 - ) + + # A protected route, with the database stubbed. It used to run against a + # real Postgres, which passed locally and failed in CI where there is none — + # and "the gate opened" is a claim about the dependency, not about the + # database. 404 for an unknown repo is the proof: the request got past + # require_session and into the handler. + app.dependency_overrides[get_db] = lambda: _NoSuchRepo() + try: + response = gated.get("/api/v1/repos/00000000-0000-0000-0000-000000000000/status") + finally: + app.dependency_overrides.clear() + assert response.status_code == 404 def test_logout_clears_the_session(gated: TestClient) -> None: @@ -358,6 +366,13 @@ def test_quota_failure_lets_the_request_through( app.dependency_overrides.clear() +class _NoSuchRepo: + """A session that knows of no repositories, and needs no server.""" + + async def get(self, _model: object, _pk: object) -> None: + return None + + class _NeverCalledAgent: """The quota check runs before the agent, so this only has to exist.""" diff --git a/conftest.py b/conftest.py index 26eb1f5..489215a 100644 --- a/conftest.py +++ b/conftest.py @@ -20,9 +20,18 @@ # Auth defaults off; tests that want the gate turn it on by patching the # settings object directly, which is explicit and machine-independent. +# +# INTERNAL_API_KEY is pinned for a sharper reason. The API and the agent refuse +# to start on the placeholder the settings module defaults to, so on a machine +# with no `.env` — which is exactly what CI is — every test that boots a +# lifespan died with "INTERNAL_API_KEY is still the placeholder". Locally it +# passed, because a developer's `.env` supplies a real value. That asymmetry is +# the whole reason this file exists, and pinning two settings and not the third +# left it in place. os.environ.setdefault("_DCODE_TEST_ENV_PINNED", "1") for name, value in ( ("AUTH_ENABLED", "false"), ("DOCS_ENABLED", "true"), + ("INTERNAL_API_KEY", "test-internal-key-not-the-published-placeholder"), ): os.environ[name] = value diff --git a/docs/en/Final_Report.md b/docs/en/Final_Report.md index 3f157c7..9e300cd 100644 --- a/docs/en/Final_Report.md +++ b/docs/en/Final_Report.md @@ -638,10 +638,24 @@ graph-sensitive H1 re-run; everything else is independent of it. ### Deployment -- DNS for `dcode.odieyang.com` is unresolved; production Compose has only been - smoke-tested locally. -- Decide whether production runs the model sidecars or depends on external model - services. +- ~~DNS for `dcode.odieyang.com` is unresolved; production Compose has only been + smoke-tested locally.~~ **— closed 2026-08-03.** The stack is deployed and + reachable over HTTPS, gated by a single shared account, with a live index and + a query returning verified citations. It runs on Railway under a + platform-provided hostname rather than `dcode.odieyang.com`; a custom domain + is a DNS record away and is not what the criterion was about. + [`Deploy.md`](../../Deploy.md) records the four things that failed first. +- ~~Decide whether production runs the model sidecars or depends on external + model services.~~ **— decided: external.** The deployment calls the same three + models the recorded run measured — Jina v2-base-code at 768 dimensions, BGE + reranker v2-m3, `gpt-4o-mini` — through hosted APIs rather than self-hosted + sidecars, so no figure on `/methodology` describes a configuration other than + the one serving it. `Deploy.md` §2. +- **The deployed index is not the evaluated index.** Both are `psf/requests`, but + the recorded run is pinned to commit `414f0513`, and the deployment indexed + whatever `HEAD` was on 2026-08-03 — 773 chunks against the recorded 726. The + verdict remains a statement about the committed artifacts in `results/`, not + about whatever the demo happens to be serving. ## Known Limits diff --git a/infra/docker/agent.Dockerfile b/infra/docker/agent.Dockerfile index 6271266..fba8ac0 100644 --- a/infra/docker/agent.Dockerfile +++ b/infra/docker/agent.Dockerfile @@ -31,4 +31,15 @@ EXPOSE 8001 # deploy it also could not find alembic at all, because the build-time sync had # installed nothing (see the RUN above). apps/embedding already did it this way # with a comment saying why; api, worker and agent had not caught up. -CMD ["sh", "-c", "exec /app/.venv/bin/uvicorn dcode_agent.main:app --host ${HOST:-::} --port ${PORT:-8001}"] +# 0.0.0.0, not `::`. Measured, not assumed: uvicorn given `::` produces an +# IPv6-ONLY socket — an IPv4 connect is refused even though the container has +# net.ipv6.bindv6only=0 — and given 0.0.0.0 the reverse. There is no dual-stack +# option, so this is a choice between them. +# +# Railway's health check arrives over IPv4: with `::` the request never reached +# the application at all and the deploy failed with "1/1 replicas never became +# healthy" while the log showed a perfectly started server. Private networking +# in environments created after 2025-10-16 resolves to both families, so IPv4 +# serves both purposes. A legacy IPv6-only environment would need `HOST=::` and +# would then need the health check removed. +CMD ["sh", "-c", "exec /app/.venv/bin/uvicorn dcode_agent.main:app --host ${HOST:-0.0.0.0} --port ${PORT:-8001}"] diff --git a/infra/docker/api.Dockerfile b/infra/docker/api.Dockerfile index 6bd47c5..bfc45f1 100644 --- a/infra/docker/api.Dockerfile +++ b/infra/docker/api.Dockerfile @@ -46,4 +46,15 @@ EXPOSE 8000 # deploy it also could not find alembic at all, because the build-time sync had # installed nothing (see the RUN above). apps/embedding already did it this way # with a comment saying why; api, worker and agent had not caught up. +# 0.0.0.0, not `::`. Measured, not assumed: uvicorn given `::` produces an +# IPv6-ONLY socket — an IPv4 connect is refused even though the container has +# net.ipv6.bindv6only=0 — and given 0.0.0.0 the reverse. There is no dual-stack +# option, so this is a choice between them. +# +# Railway's health check arrives over IPv4: with `::` the request never reached +# the application at all and the deploy failed with "1/1 replicas never became +# healthy" while the log showed a perfectly started server. Private networking +# in environments created after 2025-10-16 resolves to both families, so IPv4 +# serves both purposes. A legacy IPv6-only environment would need `HOST=::` and +# would then need the health check removed. CMD ["sh", "-c", "exec /app/.venv/bin/uvicorn dcode_api.main:app --host ${HOST:-0.0.0.0} --port ${PORT:-8000}"] diff --git a/infra/docker/railway-agent-worker-entrypoint.sh b/infra/docker/railway-agent-worker-entrypoint.sh index 84136ab..8103995 100755 --- a/infra/docker/railway-agent-worker-entrypoint.sh +++ b/infra/docker/railway-agent-worker-entrypoint.sh @@ -23,9 +23,10 @@ term() { } trap term TERM INT +# See api.Dockerfile for why this is 0.0.0.0 and not `::`. echo "[entrypoint] starting agent on :${PORT:-8001}" /app/.venv/bin/uvicorn dcode_agent.main:app \ - --host "${HOST:-::}" --port "${PORT:-8001}" & + --host "${HOST:-0.0.0.0}" --port "${PORT:-8001}" & agent_pid=$! echo "[entrypoint] starting index worker" diff --git a/infra/railway/README.md b/infra/railway/README.md index 4a0d327..572d10d 100644 --- a/infra/railway/README.md +++ b/infra/railway/README.md @@ -4,9 +4,10 @@ Three `*.toml` files here, one per service that builds from this repository. The three data services (Postgres, Redis, RabbitMQ) come from Railway templates and need no file. -**Nothing here has been deployed.** These were written from the code, and the -sequence in [`Deploy.md` §10.2](../../Deploy.md) is the intended one rather than -a recorded one. Correct both against what actually happens. +**Deployed and reachable as of 2026-08-03.** What follows has been corrected +against what actually happened rather than left as intent — four things failed +first, and each one is now called out where it bit: the volume under `/tmp`, +`HOST=::`, the injected `PORT`, and `uv run` at container start. ## Wiring each service @@ -21,8 +22,11 @@ from the repository root. Leave Root Directory empty: every image here builds from the repository root, including the frontend — it used to build from `apps/frontend` and was changed precisely so this table has no exception in it. -`agent-worker` needs **one volume**, mounted at whatever `WORKDIR_BASE` says -(`/tmp/dcode-workdirs` by default). Without it the cloned checkout is lost on +`agent-worker` needs **one volume**, and **not mounted under `/tmp`**. Railway +refuses that and fails the deployment *before producing a single log line* — no +build log, no deploy log, just FAILED, which is indistinguishable from a dozen +other causes. Mount it at `/data/workdirs` and set `WORKDIR_BASE` to match. The +`/tmp/dcode-workdirs` default is right for compose and wrong here. Without it the cloned checkout is lost on every redeploy, and the agent's `read_file` / `grep` / `list_directory` fail for every repository until it is re-indexed. Search and the call graph keep working, because they read Postgres — which is exactly the confusing half-failure the @@ -48,7 +52,8 @@ avoids `$` now; a hand-picked Postgres password still can. | Variable | Value | |---|---| -| `HOST` | `::` — **required.** Railway's private network is IPv6; a service bound to `0.0.0.0` there is reachable by nothing | +| `PORT` | `8000` — **set it explicitly.** Railway injects one (8080) otherwise, and then every private URL pointing at `:8000` is wrong. Pinning it makes the other services' URLs correct by construction and matches compose | +| `HOST` | `0.0.0.0`. **Not `::`** — measured: uvicorn given `::` produces an IPv6-only socket and Railway's health check arrives over IPv4, so the probe never reaches the app and the deploy fails with "1/1 replicas never became healthy" beside a log showing a healthy server | | `DATABASE_URL` | `${{Postgres.DATABASE_URL}}`, rewritten to the `postgresql+asyncpg://` scheme | | `REDIS_URL` | `${{Redis.REDIS_URL}}` | | `RABBITMQ_URL` | from the RabbitMQ service | @@ -72,9 +77,10 @@ Same `EMBEDDING_*` / `RERANKER_*` / `INTERNAL_API_KEY` / `DATABASE_URL` / | Variable | Value | |---|---| -| `HOST` | `::` | +| `PORT` | `8001` | +| `HOST` | `0.0.0.0` | | `RETRIEVAL_BASE_URL` | `http://api.railway.internal:8000` | -| `WORKDIR_BASE` | the volume's mount path | +| `WORKDIR_BASE` | `/data/workdirs` — the volume's mount path, and not under `/tmp` | | `WORKDIR_MAX_REPOS` | a number the volume can hold. `0` never evicts, which is how the volume fills; read `Deploy.md` §6 PR 5 for what eviction costs | | `SYNTHESIS_MODEL`, `OPENAI_API_KEY` | `gpt-4o-mini` and your key, to get prose answers rather than the rule-based template | | `AGENT_REQUEST_BUDGET_SECONDS`, `SSE_HEARTBEAT_SECONDS` | `240`, `20`. Both sized against the platform's request limits (`Deploy.md` R-2) |