From a547a58a235ccc8f6759f4900d264400f1609baa Mon Sep 17 00:00:00 2001 From: odieyang <520yangziqi@gmail.com> Date: Mon, 3 Aug 2026 14:24:46 -0700 Subject: [PATCH 1/4] fix(deploy): correct three Railway settings the first deploy disproved MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit All three were written from documentation and contradicted by the platform. The volume must not be mounted under /tmp. Railway refuses it and fails the deployment before producing a single log line — no build log, no deploy log, just FAILED. That is why agent-worker looked identical to a dozen other causes and took the longest to find: the only signal was that it was the one app service with a volume. Moved to /data/workdirs. The /tmp default stays right for compose. HOST must be 0.0.0.0, not `::`. Measured rather than reasoned: uvicorn given `::` produces an IPv6-only socket — an IPv4 connect is refused even with net.ipv6.bindv6only=0 — and given 0.0.0.0 the reverse. There is no dual-stack option. Railway's health check arrives over IPv4, so with `::` the request never reached the application and the deploy failed with "1/1 replicas never became healthy" next to a log showing a perfectly started server. I had set `::` on the strength of Railway's private-networking note, which applies to environments created before 2025-10-16. PORT has to be set explicitly. Railway injects 8080, our processes correctly honour it, and every private URL I had written said :8000 — so nginx returned 502 for /api/* while both services were healthy. Pinning 8000 and 8001 makes the other services' URLs correct by construction instead of by luck, and matches compose, which is the thing anyone will compare against. The Railway README no longer says nothing has been deployed. It is corrected against what happened, with each of the four failures noted where it bites. --- infra/docker/agent.Dockerfile | 13 ++++++++++- infra/docker/api.Dockerfile | 11 ++++++++++ .../docker/railway-agent-worker-entrypoint.sh | 3 ++- infra/railway/README.md | 22 ++++++++++++------- 4 files changed, 39 insertions(+), 10 deletions(-) diff --git a/infra/docker/agent.Dockerfile b/infra/docker/agent.Dockerfile index 6271266..fba8ac0 100644 --- a/infra/docker/agent.Dockerfile +++ b/infra/docker/agent.Dockerfile @@ -31,4 +31,15 @@ EXPOSE 8001 # deploy it also could not find alembic at all, because the build-time sync had # installed nothing (see the RUN above). apps/embedding already did it this way # with a comment saying why; api, worker and agent had not caught up. -CMD ["sh", "-c", "exec /app/.venv/bin/uvicorn dcode_agent.main:app --host ${HOST:-::} --port ${PORT:-8001}"] +# 0.0.0.0, not `::`. Measured, not assumed: uvicorn given `::` produces an +# IPv6-ONLY socket — an IPv4 connect is refused even though the container has +# net.ipv6.bindv6only=0 — and given 0.0.0.0 the reverse. There is no dual-stack +# option, so this is a choice between them. +# +# Railway's health check arrives over IPv4: with `::` the request never reached +# the application at all and the deploy failed with "1/1 replicas never became +# healthy" while the log showed a perfectly started server. Private networking +# in environments created after 2025-10-16 resolves to both families, so IPv4 +# serves both purposes. A legacy IPv6-only environment would need `HOST=::` and +# would then need the health check removed. +CMD ["sh", "-c", "exec /app/.venv/bin/uvicorn dcode_agent.main:app --host ${HOST:-0.0.0.0} --port ${PORT:-8001}"] diff --git a/infra/docker/api.Dockerfile b/infra/docker/api.Dockerfile index 6bd47c5..bfc45f1 100644 --- a/infra/docker/api.Dockerfile +++ b/infra/docker/api.Dockerfile @@ -46,4 +46,15 @@ EXPOSE 8000 # deploy it also could not find alembic at all, because the build-time sync had # installed nothing (see the RUN above). apps/embedding already did it this way # with a comment saying why; api, worker and agent had not caught up. +# 0.0.0.0, not `::`. Measured, not assumed: uvicorn given `::` produces an +# IPv6-ONLY socket — an IPv4 connect is refused even though the container has +# net.ipv6.bindv6only=0 — and given 0.0.0.0 the reverse. There is no dual-stack +# option, so this is a choice between them. +# +# Railway's health check arrives over IPv4: with `::` the request never reached +# the application at all and the deploy failed with "1/1 replicas never became +# healthy" while the log showed a perfectly started server. Private networking +# in environments created after 2025-10-16 resolves to both families, so IPv4 +# serves both purposes. A legacy IPv6-only environment would need `HOST=::` and +# would then need the health check removed. CMD ["sh", "-c", "exec /app/.venv/bin/uvicorn dcode_api.main:app --host ${HOST:-0.0.0.0} --port ${PORT:-8000}"] diff --git a/infra/docker/railway-agent-worker-entrypoint.sh b/infra/docker/railway-agent-worker-entrypoint.sh index 84136ab..8103995 100755 --- a/infra/docker/railway-agent-worker-entrypoint.sh +++ b/infra/docker/railway-agent-worker-entrypoint.sh @@ -23,9 +23,10 @@ term() { } trap term TERM INT +# See api.Dockerfile for why this is 0.0.0.0 and not `::`. echo "[entrypoint] starting agent on :${PORT:-8001}" /app/.venv/bin/uvicorn dcode_agent.main:app \ - --host "${HOST:-::}" --port "${PORT:-8001}" & + --host "${HOST:-0.0.0.0}" --port "${PORT:-8001}" & agent_pid=$! echo "[entrypoint] starting index worker" diff --git a/infra/railway/README.md b/infra/railway/README.md index 4a0d327..572d10d 100644 --- a/infra/railway/README.md +++ b/infra/railway/README.md @@ -4,9 +4,10 @@ Three `*.toml` files here, one per service that builds from this repository. The three data services (Postgres, Redis, RabbitMQ) come from Railway templates and need no file. -**Nothing here has been deployed.** These were written from the code, and the -sequence in [`Deploy.md` §10.2](../../Deploy.md) is the intended one rather than -a recorded one. Correct both against what actually happens. +**Deployed and reachable as of 2026-08-03.** What follows has been corrected +against what actually happened rather than left as intent — four things failed +first, and each one is now called out where it bit: the volume under `/tmp`, +`HOST=::`, the injected `PORT`, and `uv run` at container start. ## Wiring each service @@ -21,8 +22,11 @@ from the repository root. Leave Root Directory empty: every image here builds from the repository root, including the frontend — it used to build from `apps/frontend` and was changed precisely so this table has no exception in it. -`agent-worker` needs **one volume**, mounted at whatever `WORKDIR_BASE` says -(`/tmp/dcode-workdirs` by default). Without it the cloned checkout is lost on +`agent-worker` needs **one volume**, and **not mounted under `/tmp`**. Railway +refuses that and fails the deployment *before producing a single log line* — no +build log, no deploy log, just FAILED, which is indistinguishable from a dozen +other causes. Mount it at `/data/workdirs` and set `WORKDIR_BASE` to match. The +`/tmp/dcode-workdirs` default is right for compose and wrong here. Without it the cloned checkout is lost on every redeploy, and the agent's `read_file` / `grep` / `list_directory` fail for every repository until it is re-indexed. Search and the call graph keep working, because they read Postgres — which is exactly the confusing half-failure the @@ -48,7 +52,8 @@ avoids `$` now; a hand-picked Postgres password still can. | Variable | Value | |---|---| -| `HOST` | `::` — **required.** Railway's private network is IPv6; a service bound to `0.0.0.0` there is reachable by nothing | +| `PORT` | `8000` — **set it explicitly.** Railway injects one (8080) otherwise, and then every private URL pointing at `:8000` is wrong. Pinning it makes the other services' URLs correct by construction and matches compose | +| `HOST` | `0.0.0.0`. **Not `::`** — measured: uvicorn given `::` produces an IPv6-only socket and Railway's health check arrives over IPv4, so the probe never reaches the app and the deploy fails with "1/1 replicas never became healthy" beside a log showing a healthy server | | `DATABASE_URL` | `${{Postgres.DATABASE_URL}}`, rewritten to the `postgresql+asyncpg://` scheme | | `REDIS_URL` | `${{Redis.REDIS_URL}}` | | `RABBITMQ_URL` | from the RabbitMQ service | @@ -72,9 +77,10 @@ Same `EMBEDDING_*` / `RERANKER_*` / `INTERNAL_API_KEY` / `DATABASE_URL` / | Variable | Value | |---|---| -| `HOST` | `::` | +| `PORT` | `8001` | +| `HOST` | `0.0.0.0` | | `RETRIEVAL_BASE_URL` | `http://api.railway.internal:8000` | -| `WORKDIR_BASE` | the volume's mount path | +| `WORKDIR_BASE` | `/data/workdirs` — the volume's mount path, and not under `/tmp` | | `WORKDIR_MAX_REPOS` | a number the volume can hold. `0` never evicts, which is how the volume fills; read `Deploy.md` §6 PR 5 for what eviction costs | | `SYNTHESIS_MODEL`, `OPENAI_API_KEY` | `gpt-4o-mini` and your key, to get prose answers rather than the rule-based template | | `AGENT_REQUEST_BUDGET_SECONDS`, `SSE_HEARTBEAT_SECONDS` | `240`, `20`. Both sized against the platform's request limits (`Deploy.md` R-2) | From ffb8f54034a29e79de5049ac519e36da07a80556 Mon Sep 17 00:00:00 2001 From: odieyang <520yangziqi@gmail.com> Date: Mon, 3 Aug 2026 14:42:45 -0700 Subject: [PATCH 2/4] docs: the deployment is live, and the four things that failed first MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Records what was checked against the running system rather than what was intended, and strikes the claims that are no longer true. Verified on the deployment: 773/773 vectors at 768 dimensions with zero all-zero rows and 770 distinct prefixes, so the hosted embedding path produces real varied embeddings and not stubs; /readyz green on all three checks; 401 on every protected route and a public landing and methodology page; one query end to end with three citations all verified and groundedness 1.0. The two rows that needed eyes were confirmed in a browser. Two things are recorded as NOT verified rather than assumed. The reranker two-mode comparison cannot be run from outside, because /internal/* is not publicly routed — it was verified locally instead. And the R-2 heartbeat has never been needed: every live query finished well inside the platform's idle window, so the keep-alive stays correct-by-construction and unobserved. Also recorded: the deployed index is NOT the evaluated index. Both are psf/requests, but the recorded run is pinned to 414f0513 and the deployment indexed current HEAD — 773 chunks against the recorded 726. The verdict remains a statement about the committed runs in results/, and the README and Final_Report now say so rather than leaving a reader to conflate the two. §10.3 is new: the four first-deploy failures, all of them mistakes in this plan and three of them things written from documentation and not measured. The pattern is the part worth keeping — every one presented as a different problem than it was. A twelve-minute build that was a deploy. A build failure with no build. A healthy server that was unreachable. A 502 from a service that was up. Each time the log that mattered was one layer away from the one being read. Outstanding Work loses both Deployment items: the external-demo criterion is closed, and the sidecars-versus-hosted-APIs question is decided as hosted, running the same three models so no displayed figure describes a configuration other than the one serving it. --- CLAUDE.md | 14 +++-- Deploy.md | 113 +++++++++++++++++++++++++++++----------- README.md | 16 ++++-- docs/en/Final_Report.md | 22 ++++++-- 4 files changed, 122 insertions(+), 43 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index f37f45a..c8c850b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -99,10 +99,16 @@ that stay true no matter which PR is in flight. 7. **`docker-compose.prod.yml` is fixed, not deleted.** `Final_Report.md` lists it as delivered. `Deploy.md` §5.6. -The pre-deployment state is frozen at tag `v1.0-submission` (`a4612b8`). Two -Railway constraints are load-bearing and were verified on 2026-08-02: a volume -attaches to exactly one service, and an HTTP request is cut after 5 minutes with -no data (~15 minutes with heartbeats). +**A gated deployment is live** (Railway, HTTPS, one shared account, the same +three models the recorded run measured). The pre-deployment state is frozen at +tag `v1.0-submission` (`a4612b8`). + +Four things failed on the first deploy and all four were wrong assumptions in +the plan, not platform faults — `Deploy.md` §10.3 has them. The two worth +carrying in your head, because they present as something else entirely: a volume +mounted under `/tmp` fails the deployment **with no log output at all**, and +`HOST=::` gives uvicorn an IPv6-only socket while the health check arrives over +IPv4, so you get "never became healthy" beside a log showing a healthy server. ## 3. Where things are diff --git a/Deploy.md b/Deploy.md index 891bad4..1cf8969 100644 --- a/Deploy.md +++ b/Deploy.md @@ -25,8 +25,10 @@ Outstanding Work list in the same PR, not tracked in two places. is written from an approved plan, and plans outrun implementations. Verify before relying on a claim here, and correct it when it is wrong. -**Status: PR 1–5 landed. Nothing is deployed yet** — every remaining item is -in § 10.2 and needs a Railway account. The pre-deployment state +**Status: deployed and verified, 2026-08-03.** All five PRs landed and the stack +is live at [`frontend-production-55c4.up.railway.app`](https://frontend-production-55c4.up.railway.app). § 10 records what was checked +and how; § 10.3 records the four things that failed first, all of them mistakes +in this plan rather than in the platform. The pre-deployment state is frozen at tag `v1.0-submission` (`a4612b8`). Nothing is deployed anywhere yet. Both hosted model APIs have been exercised end to end from a running local stack — see § 2.1 and § 6 PR 2 for the observed values — but **no index has been @@ -387,14 +389,11 @@ The capability behind D-1. different serving stack, not bit-identical. It is visible and it does not move the ranking. - **What is still not done: no index has been built through the hosted - provider.** The verification above is query-side. 726 chunks at batch 32 is - roughly 23 requests, which is where rate limiting and the embed stage's Redis - cache interaction would appear, and neither has been exercised. That is - deliberate — re-indexing would replace the verified reference index — and it - will happen naturally at the first Railway index. Until then this PR proves - the clients are correct and cross-compatible with the sidecar, not that a - bulk index built through them is. + **Closed 2026-08-03: an index has been built through the hosted provider**, + on the deployment. 773 chunks in roughly 24 requests to Jina's API, every one + `200 OK`, **no retries and no rate limiting**. Read back from the deployed + database: all 773 vectors at 768 dimensions, **zero all-zero vectors**, and + 770 distinct prefixes across 773 rows — real, varied embeddings, not stubs. The sidecar path was not re-run live after the switch. Its code is untouched and its factory selection is covered by tests, but that is a weaker claim @@ -487,15 +486,17 @@ done.** Everything below was observed; the row that needs Railway needs Railway. | Heartbeats and the request budget | 5 tests, including that `close()` still ends a stream while the heartbeat loop runs, and that an overrun emits `TIMEOUT` and **no** `final_answer` | | The local stack is unchanged | `docker compose up -d --build api` still healthy; `make check` green | -**Not done: no Railway deployment exists.** The runbook is § 10.2, written from -the code rather than from a completed deploy — read it as intended sequence and -correct it against what happens. These are unverified until then: private -networking over IPv6, the platform's request limits against real heartbeats, the -pgvector template, and the first hosted-provider index (§ 6 PR 2). +**Deployed 2026-08-03.** § 10.2 is corrected against what happened rather than +written from the code. The combined `agent-worker` container runs, and its +`wait -n` behaviour was confirmed the hard way: it exited with `a child exited +with 1; stopping the other` when the broker was unreachable, rather than serving +an agent beside a dead worker. -The combined `agent-worker` container has **not been run**, locally or -otherwise. Its entrypoint is Railway-only and the developer stack does not use -it, so nothing here exercises it. +Two things this PR assumed that turned out backwards, both in § 10.3: private +networking works over **IPv4**, and `HOST=::` is what broke the health check. +One thing remains unobserved — the R-2 heartbeat. Every live query so far +finished well inside the platform's idle window, so the keep-alive is +correct-by-construction and has never been needed. ### PR 5 — `fix/audit-followups` — **done** @@ -680,17 +681,23 @@ neither is a reason to soften a report: Before calling the deployment done: -| # | Check | How | +Checked against the live deployment on 2026-08-03. + +| # | Check | Result | |---|---|---| -| 1 | Vectors in the deployed database are 768-dimensional and non-zero | `SELECT vector_dims(embedding), left(embedding::text, 20) FROM chunks LIMIT 1;` | -| 2 | The reranker is actually reordering, not passing through | Compare `score_components` on a query with and without rerank | -| 3 | `api` and `agent-worker` agree on embedding configuration | § 5.4 readiness check reports ready | -| 4 | A query streams to completion without being cut | Watch a full run; confirm no truncation at 5 minutes (R-2) | -| 5 | Citations are verified and clickable, and open real indexed source at the cited line | Human, in a browser | -| 6 | Call-graph neighbours are clickable and chain | Human, in a browser | -| 7 | Unauthenticated access to `/workbench` and to each protected route is refused | Private window | -| 8 | `/` and `/methodology` load without a session, and their figures match `results/` | `python3 scripts/sync_eval_artifacts.py --check` | -| 9 | The four local modes in § 5.1 all still start | Locally, after the final PR | +| 1 | Vectors in the deployed database are 768-dimensional and non-zero | **verified** — 773/773 at `dims=768`, 0 all-zero, 770 distinct prefixes | +| 2 | The reranker is actually reordering, not passing through | **not verified on the deployment.** `/internal/*` is not publicly routed, so the two-mode comparison cannot be run from outside. It was verified locally in § 6 PR 2 | +| 3 | `api` and `agent-worker` agree on embedding configuration | **verified** — `/readyz` returns 200 with all three checks green | +| 4 | A query streams to completion without being cut | **verified** for a normal query; the heartbeat path is unobserved because nothing has yet run long enough to need it | +| 5 | Citations are verified and clickable, and open real indexed source at the cited line | **verified by a human in a browser** | +| 6 | Call-graph neighbours are clickable and chain | **verified by a human in a browser** | +| 7 | Unauthenticated access to `/workbench` and to each protected route is refused | **verified** — 401 on every protected route; `/workbench` redirects to `/login` | +| 8 | `/` and `/methodology` load without a session, and their figures match `results/` | **verified** — both 200 unauthenticated, drift check green | +| 9 | The four local modes in § 5.1 all still start | **verified** — compose stack healthy after the final PR | + +One live query end to end: 3 tool calls, **3 citations all verified**, +groundedness **1.0**, reproducing the two citations the local run found and +adding a third. --- @@ -790,9 +797,8 @@ confirms you are back. ### 10.2 Railway runbook -Written from the code, not from a completed deployment — **nothing has been -deployed yet** (§ 6 PR 4). Treat it as the intended sequence, and correct it -against what actually happens. +Corrected against a completed deployment on 2026-08-03. The four things that +failed the first time are § 10.3; each is called out below where it bites. **The configuration is committed.** `infra/railway/{api,agent-worker,frontend}.toml` carry the build and deploy settings, and @@ -857,6 +863,51 @@ rebinding is not closed without it or network egress filtering. --- +### 10.3 What failed on the first deploy + +Four failures, in the order they were found. **All four were mistakes in this +plan, not in the platform** — three of them were things I had written from +documentation and not measured. + +**1. `uv run` was installing dependencies at container start.** `api` spent +twelve minutes and then failed with `Failed to spawn: alembic`. Two compounding +causes: `uv sync --no-dev` at a workspace root syncs the *root* project, and the +root declares no dependencies, so the build step installed nothing — the images +had two entries in `site-packages`. Everything was really installed by `uv run` +at start, and `uv run` without `--no-dev` downloads `mypy` and `ruff` first. +Fixed by `uv sync --no-dev --package ` and calling the venv binaries +directly; `uv run` is now gone from `infra/` entirely. `apps/embedding` had +already done this, with a comment saying why. + +**2. A volume cannot be mounted under `/tmp`.** Railway refuses and fails the +deployment **before emitting a single log line** — no build log, no deploy log, +just `FAILED`, which is indistinguishable from any other cause. This took the +longest to find and the only usable signal was that `agent-worker` was the one +app service with a volume. Moved to `/data/workdirs`. + +**3. `HOST=::` produces an IPv6-only socket.** Measured, after assuming +otherwise: uvicorn given `::` refuses an IPv4 connect even with +`net.ipv6.bindv6only=0`, and given `0.0.0.0` refuses IPv6. There is no +dual-stack option. Railway's health check arrives over **IPv4**, so the probe +never reached the application and the deploy failed with `1/1 replicas never +became healthy` beside a log showing a perfectly started server. R-4's advice to +bind `::` applies to environments created before 2025-10-16; a current one +resolves private DNS to both families, so `0.0.0.0` serves both purposes. + +**4. `PORT` must be set explicitly.** Railway injects `8080`, the services +correctly honour it, and every private-network URL in this plan said `:8000` — so +nginx returned `502` for `/api/*` while both services were healthy. Pinning +`8000` and `8001` makes the other services' URLs right by construction, and +matches compose, which is what anyone debugging will compare against. + +**The pattern worth taking from this**: every one of these presented as a +different problem than it was. A twelve-minute build that was a deploy. A build +failure with no build. A healthy server that was unreachable. A 502 from a +service that was up. In each case the log that mattered was one layer away from +the one being read. + +--- + ## 11. Open Questions Carried here rather than answered by assumption. diff --git a/README.md b/README.md index 4734539..e2dd1bb 100644 --- a/README.md +++ b/README.md @@ -335,10 +335,18 @@ embedding and reranker configuration, and they must agree. `EMBEDDING_DIM` has no production default on purpose: the migration bakes it into the pgvector column, so a wrong value is a re-index rather than a slow query. -The production stack is validated locally only — `dcode.odieyang.com` does **not** -resolve publicly, so the external-demo exit criterion is still open. The plan for -closing it, including the platform constraints that shape it, is -[`Deploy.md`](Deploy.md). +**A gated deployment is live** at [`frontend-production-55c4.up.railway.app`](https://frontend-production-55c4.up.railway.app) — +Railway, HTTPS, one shared account, running the same three models the recorded +evaluation measured. The landing page and `/methodology` are public; the +workbench is not. Access is by request. + +The deployment indexes `psf/requests` at current `HEAD`, which is **not** the +commit the H1 verdict was measured on — the verdict is a statement about the +committed runs in [`results/`](results/), not about what the demo is serving. + +Service configuration, the variable sheet, and the four things that failed on +the first attempt: [`Deploy.md`](Deploy.md) and +[`infra/railway/`](infra/railway/). --- diff --git a/docs/en/Final_Report.md b/docs/en/Final_Report.md index 3f157c7..9e300cd 100644 --- a/docs/en/Final_Report.md +++ b/docs/en/Final_Report.md @@ -638,10 +638,24 @@ graph-sensitive H1 re-run; everything else is independent of it. ### Deployment -- DNS for `dcode.odieyang.com` is unresolved; production Compose has only been - smoke-tested locally. -- Decide whether production runs the model sidecars or depends on external model - services. +- ~~DNS for `dcode.odieyang.com` is unresolved; production Compose has only been + smoke-tested locally.~~ **— closed 2026-08-03.** The stack is deployed and + reachable over HTTPS, gated by a single shared account, with a live index and + a query returning verified citations. It runs on Railway under a + platform-provided hostname rather than `dcode.odieyang.com`; a custom domain + is a DNS record away and is not what the criterion was about. + [`Deploy.md`](../../Deploy.md) records the four things that failed first. +- ~~Decide whether production runs the model sidecars or depends on external + model services.~~ **— decided: external.** The deployment calls the same three + models the recorded run measured — Jina v2-base-code at 768 dimensions, BGE + reranker v2-m3, `gpt-4o-mini` — through hosted APIs rather than self-hosted + sidecars, so no figure on `/methodology` describes a configuration other than + the one serving it. `Deploy.md` §2. +- **The deployed index is not the evaluated index.** Both are `psf/requests`, but + the recorded run is pinned to commit `414f0513`, and the deployment indexed + whatever `HEAD` was on 2026-08-03 — 773 chunks against the recorded 726. The + verdict remains a statement about the committed artifacts in `results/`, not + about whatever the demo happens to be serving. ## Known Limits From 14d99501aec0fad535e8ac8883119e39101a06ce Mon Sep 17 00:00:00 2001 From: odieyang <520yangziqi@gmail.com> Date: Mon, 3 Aug 2026 15:06:58 -0700 Subject: [PATCH 3/4] fix(test): make the suite pass without a .env, which is what CI is MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CI on main has been red since #21 and I did not look. Every report of "make check green" in this series was true and insufficient: it passes locally because a developer's .env supplies a real INTERNAL_API_KEY and a Postgres is running. CI has neither, and CI is the hermetic environment that exists to catch exactly this. Two failures, both mine. #21 added a test that asserted a protected route stops returning 401 by hitting /repos/{id}/status, which connects to Postgres. The claim being tested is "the cookie got past require_session", which is about the dependency and not about the database. It now stubs the session and asserts 404 for an unknown repo — the request reached the handler, which is the actual proof. #23 added a startup guard refusing the placeholder INTERNAL_API_KEY, so on a machine with no .env every test that boots a lifespan died with "still the placeholder". conftest.py now pins it alongside AUTH_ENABLED and DOCS_ENABLED. That file was written for precisely this asymmetry, and pinning two settings and not the third left the hole open — the guard just made it fatal instead of silent. Verified in both environments rather than one: 394 passed with .env present, and 394 passed with .env moved aside and DATABASE_URL pointed at a dead port, which is the closest local reproduction of CI. --- apps/api/tests/test_auth.py | 25 ++++++++++++++++++++----- conftest.py | 9 +++++++++ 2 files changed, 29 insertions(+), 5 deletions(-) diff --git a/apps/api/tests/test_auth.py b/apps/api/tests/test_auth.py index 473b434..8d7d4c3 100644 --- a/apps/api/tests/test_auth.py +++ b/apps/api/tests/test_auth.py @@ -20,7 +20,7 @@ read_session, verify_password, ) -from dcode_api.deps import get_agent_client, get_redis +from dcode_api.deps import get_agent_client, get_db, get_redis from dcode_api.main import app, create_app from dcode_api.settings import api_settings from fastapi.testclient import TestClient @@ -254,10 +254,18 @@ def test_login_sets_a_hardened_cookie_and_opens_the_gate(gated: TestClient) -> N # The client keeps the cookie, so a previously-401 route now resolves. assert gated.get("/api/v1/auth/me").json()["authenticated"] is True - assert ( - gated.get("/api/v1/repos/00000000-0000-0000-0000-000000000000/status").status_code - != 401 - ) + + # A protected route, with the database stubbed. It used to run against a + # real Postgres, which passed locally and failed in CI where there is none — + # and "the gate opened" is a claim about the dependency, not about the + # database. 404 for an unknown repo is the proof: the request got past + # require_session and into the handler. + app.dependency_overrides[get_db] = lambda: _NoSuchRepo() + try: + response = gated.get("/api/v1/repos/00000000-0000-0000-0000-000000000000/status") + finally: + app.dependency_overrides.clear() + assert response.status_code == 404 def test_logout_clears_the_session(gated: TestClient) -> None: @@ -358,6 +366,13 @@ def test_quota_failure_lets_the_request_through( app.dependency_overrides.clear() +class _NoSuchRepo: + """A session that knows of no repositories, and needs no server.""" + + async def get(self, _model: object, _pk: object) -> None: + return None + + class _NeverCalledAgent: """The quota check runs before the agent, so this only has to exist.""" diff --git a/conftest.py b/conftest.py index 26eb1f5..489215a 100644 --- a/conftest.py +++ b/conftest.py @@ -20,9 +20,18 @@ # Auth defaults off; tests that want the gate turn it on by patching the # settings object directly, which is explicit and machine-independent. +# +# INTERNAL_API_KEY is pinned for a sharper reason. The API and the agent refuse +# to start on the placeholder the settings module defaults to, so on a machine +# with no `.env` — which is exactly what CI is — every test that boots a +# lifespan died with "INTERNAL_API_KEY is still the placeholder". Locally it +# passed, because a developer's `.env` supplies a real value. That asymmetry is +# the whole reason this file exists, and pinning two settings and not the third +# left it in place. os.environ.setdefault("_DCODE_TEST_ENV_PINNED", "1") for name, value in ( ("AUTH_ENABLED", "false"), ("DOCS_ENABLED", "true"), + ("INTERNAL_API_KEY", "test-internal-key-not-the-published-placeholder"), ): os.environ[name] = value From 31f8b884af07874ea4d188f1f5b09ceb9700ae00 Mon Sep 17 00:00:00 2001 From: odieyang <520yangziqi@gmail.com> Date: Mon, 3 Aug 2026 15:11:50 -0700 Subject: [PATCH 4/4] docs: "make check green" and "CI green" are two different claims MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds the working agreement, because the lesson cost five merged PRs and a red main nobody was looking at. Every report in the deployment series said "make check green". Every one was true. The local run passes because a developer's .env supplies a real INTERNAL_API_KEY and a Postgres is running; CI has neither, which is the entire point of CI. So a test that quietly depends on the machine it runs on is invisible locally *by construction* — local success is evidence about the laptop, not about the change. The agreement is therefore two things, not one: check CI after every merge, and say which of the two claims you are making. Plus the reproduction recipe, so a CI-only failure gets measured rather than guessed at — move .env aside, point DATABASE_URL at a dead port, run the suite. --- CLAUDE.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/CLAUDE.md b/CLAUDE.md index c8c850b..7ffe409 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -319,4 +319,17 @@ exactly. that would have been p-hacking. Do that again. - **Be honest about what you could not verify** — no screenshots, no end-to-end from the shell — and say explicitly what a human needs to eyeball. +- **`make check` green and CI green are two different claims. Check CI after + every merge, and say which one you are reporting.** This was learned the + expensive way: main's CI was red for five merged PRs while every report said + "make check green". Both statements were true. The local run passes because a + developer's `.env` supplies a real `INTERNAL_API_KEY` and a Postgres is + running; CI has neither, which is the entire point of CI. A test that quietly + depends on the machine it runs on is invisible locally by construction, so + local success is evidence about your laptop and not about the change. + + `gh run list --branch main --limit 3` after a merge. When something only fails + in CI, reproduce it before fixing it — moving `.env` aside and pointing + `DATABASE_URL` at a dead port is the closest local approximation, and it turns + a guess into a measurement. - **Never fabricate data or fake success.** Verified means verified.