Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 23 additions & 4 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,10 +99,16 @@ that stay true no matter which PR is in flight.
7. **`docker-compose.prod.yml` is fixed, not deleted.** `Final_Report.md` lists it
as delivered. `Deploy.md` §5.6.

The pre-deployment state is frozen at tag `v1.0-submission` (`a4612b8`). Two
Railway constraints are load-bearing and were verified on 2026-08-02: a volume
attaches to exactly one service, and an HTTP request is cut after 5 minutes with
no data (~15 minutes with heartbeats).
**A gated deployment is live** (Railway, HTTPS, one shared account, the same
three models the recorded run measured). The pre-deployment state is frozen at
tag `v1.0-submission` (`a4612b8`).

Four things failed on the first deploy and all four were wrong assumptions in
the plan, not platform faults — `Deploy.md` §10.3 has them. The two worth
carrying in your head, because they present as something else entirely: a volume
mounted under `/tmp` fails the deployment **with no log output at all**, and
`HOST=::` gives uvicorn an IPv6-only socket while the health check arrives over
IPv4, so you get "never became healthy" beside a log showing a healthy server.

## 3. Where things are

Expand Down Expand Up @@ -313,4 +319,17 @@ exactly.
that would have been p-hacking. Do that again.
- **Be honest about what you could not verify** — no screenshots, no end-to-end
from the shell — and say explicitly what a human needs to eyeball.
- **`make check` green and CI green are two different claims. Check CI after
every merge, and say which one you are reporting.** This was learned the
expensive way: main's CI was red for five merged PRs while every report said
"make check green". Both statements were true. The local run passes because a
developer's `.env` supplies a real `INTERNAL_API_KEY` and a Postgres is
running; CI has neither, which is the entire point of CI. A test that quietly
depends on the machine it runs on is invisible locally by construction, so
local success is evidence about your laptop and not about the change.

`gh run list --branch main --limit 3` after a merge. When something only fails
in CI, reproduce it before fixing it — moving `.env` aside and pointing
`DATABASE_URL` at a dead port is the closest local approximation, and it turns
a guess into a measurement.
- **Never fabricate data or fake success.** Verified means verified.
113 changes: 82 additions & 31 deletions Deploy.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,10 @@ Outstanding Work list in the same PR, not tracked in two places.
is written from an approved plan, and plans outrun implementations. Verify before
relying on a claim here, and correct it when it is wrong.

**Status: PR 1–5 landed. Nothing is deployed yet** — every remaining item is
in § 10.2 and needs a Railway account. The pre-deployment state
**Status: deployed and verified, 2026-08-03.** All five PRs landed and the stack
is live at [`frontend-production-55c4.up.railway.app`](https://frontend-production-55c4.up.railway.app). § 10 records what was checked
and how; § 10.3 records the four things that failed first, all of them mistakes
in this plan rather than in the platform. The pre-deployment state
is frozen at tag `v1.0-submission` (`a4612b8`). Nothing is deployed anywhere
yet. Both hosted model APIs have been exercised end to end from a running local
stack — see § 2.1 and § 6 PR 2 for the observed values — but **no index has been
Expand Down Expand Up @@ -387,14 +389,11 @@ The capability behind D-1.
different serving stack, not bit-identical. It is visible and it does not
move the ranking.

**What is still not done: no index has been built through the hosted
provider.** The verification above is query-side. 726 chunks at batch 32 is
roughly 23 requests, which is where rate limiting and the embed stage's Redis
cache interaction would appear, and neither has been exercised. That is
deliberate — re-indexing would replace the verified reference index — and it
will happen naturally at the first Railway index. Until then this PR proves
the clients are correct and cross-compatible with the sidecar, not that a
bulk index built through them is.
**Closed 2026-08-03: an index has been built through the hosted provider**,
on the deployment. 773 chunks in roughly 24 requests to Jina's API, every one
`200 OK`, **no retries and no rate limiting**. Read back from the deployed
database: all 773 vectors at 768 dimensions, **zero all-zero vectors**, and
770 distinct prefixes across 773 rows — real, varied embeddings, not stubs.

The sidecar path was not re-run live after the switch. Its code is untouched
and its factory selection is covered by tests, but that is a weaker claim
Expand Down Expand Up @@ -487,15 +486,17 @@ done.** Everything below was observed; the row that needs Railway needs Railway.
| Heartbeats and the request budget | 5 tests, including that `close()` still ends a stream while the heartbeat loop runs, and that an overrun emits `TIMEOUT` and **no** `final_answer` |
| The local stack is unchanged | `docker compose up -d --build api` still healthy; `make check` green |

**Not done: no Railway deployment exists.** The runbook is § 10.2, written from
the code rather than from a completed deploy — read it as intended sequence and
correct it against what happens. These are unverified until then: private
networking over IPv6, the platform's request limits against real heartbeats, the
pgvector template, and the first hosted-provider index (§ 6 PR 2).
**Deployed 2026-08-03.** § 10.2 is corrected against what happened rather than
written from the code. The combined `agent-worker` container runs, and its
`wait -n` behaviour was confirmed the hard way: it exited with `a child exited
with 1; stopping the other` when the broker was unreachable, rather than serving
an agent beside a dead worker.

The combined `agent-worker` container has **not been run**, locally or
otherwise. Its entrypoint is Railway-only and the developer stack does not use
it, so nothing here exercises it.
Two things this PR assumed that turned out backwards, both in § 10.3: private
networking works over **IPv4**, and `HOST=::` is what broke the health check.
One thing remains unobserved — the R-2 heartbeat. Every live query so far
finished well inside the platform's idle window, so the keep-alive is
correct-by-construction and has never been needed.

### PR 5 — `fix/audit-followups` — **done**

Expand Down Expand Up @@ -680,17 +681,23 @@ neither is a reason to soften a report:

Before calling the deployment done:

| # | Check | How |
Checked against the live deployment on 2026-08-03.

| # | Check | Result |
|---|---|---|
| 1 | Vectors in the deployed database are 768-dimensional and non-zero | `SELECT vector_dims(embedding), left(embedding::text, 20) FROM chunks LIMIT 1;` |
| 2 | The reranker is actually reordering, not passing through | Compare `score_components` on a query with and without rerank |
| 3 | `api` and `agent-worker` agree on embedding configuration | § 5.4 readiness check reports ready |
| 4 | A query streams to completion without being cut | Watch a full run; confirm no truncation at 5 minutes (R-2) |
| 5 | Citations are verified and clickable, and open real indexed source at the cited line | Human, in a browser |
| 6 | Call-graph neighbours are clickable and chain | Human, in a browser |
| 7 | Unauthenticated access to `/workbench` and to each protected route is refused | Private window |
| 8 | `/` and `/methodology` load without a session, and their figures match `results/` | `python3 scripts/sync_eval_artifacts.py --check` |
| 9 | The four local modes in § 5.1 all still start | Locally, after the final PR |
| 1 | Vectors in the deployed database are 768-dimensional and non-zero | **verified** — 773/773 at `dims=768`, 0 all-zero, 770 distinct prefixes |
| 2 | The reranker is actually reordering, not passing through | **not verified on the deployment.** `/internal/*` is not publicly routed, so the two-mode comparison cannot be run from outside. It was verified locally in § 6 PR 2 |
| 3 | `api` and `agent-worker` agree on embedding configuration | **verified** — `/readyz` returns 200 with all three checks green |
| 4 | A query streams to completion without being cut | **verified** for a normal query; the heartbeat path is unobserved because nothing has yet run long enough to need it |
| 5 | Citations are verified and clickable, and open real indexed source at the cited line | **verified by a human in a browser** |
| 6 | Call-graph neighbours are clickable and chain | **verified by a human in a browser** |
| 7 | Unauthenticated access to `/workbench` and to each protected route is refused | **verified** — 401 on every protected route; `/workbench` redirects to `/login` |
| 8 | `/` and `/methodology` load without a session, and their figures match `results/` | **verified** — both 200 unauthenticated, drift check green |
| 9 | The four local modes in § 5.1 all still start | **verified** — compose stack healthy after the final PR |

One live query end to end: 3 tool calls, **3 citations all verified**,
groundedness **1.0**, reproducing the two citations the local run found and
adding a third.

---

Expand Down Expand Up @@ -790,9 +797,8 @@ confirms you are back.

### 10.2 Railway runbook

Written from the code, not from a completed deployment — **nothing has been
deployed yet** (§ 6 PR 4). Treat it as the intended sequence, and correct it
against what actually happens.
Corrected against a completed deployment on 2026-08-03. The four things that
failed the first time are § 10.3; each is called out below where it bites.

**The configuration is committed.** `infra/railway/{api,agent-worker,frontend}.toml`
carry the build and deploy settings, and
Expand Down Expand Up @@ -857,6 +863,51 @@ rebinding is not closed without it or network egress filtering.

---

### 10.3 What failed on the first deploy

Four failures, in the order they were found. **All four were mistakes in this
plan, not in the platform** — three of them were things I had written from
documentation and not measured.

**1. `uv run` was installing dependencies at container start.** `api` spent
twelve minutes and then failed with `Failed to spawn: alembic`. Two compounding
causes: `uv sync --no-dev` at a workspace root syncs the *root* project, and the
root declares no dependencies, so the build step installed nothing — the images
had two entries in `site-packages`. Everything was really installed by `uv run`
at start, and `uv run` without `--no-dev` downloads `mypy` and `ruff` first.
Fixed by `uv sync --no-dev --package <pkg>` and calling the venv binaries
directly; `uv run` is now gone from `infra/` entirely. `apps/embedding` had
already done this, with a comment saying why.

**2. A volume cannot be mounted under `/tmp`.** Railway refuses and fails the
deployment **before emitting a single log line** — no build log, no deploy log,
just `FAILED`, which is indistinguishable from any other cause. This took the
longest to find and the only usable signal was that `agent-worker` was the one
app service with a volume. Moved to `/data/workdirs`.

**3. `HOST=::` produces an IPv6-only socket.** Measured, after assuming
otherwise: uvicorn given `::` refuses an IPv4 connect even with
`net.ipv6.bindv6only=0`, and given `0.0.0.0` refuses IPv6. There is no
dual-stack option. Railway's health check arrives over **IPv4**, so the probe
never reached the application and the deploy failed with `1/1 replicas never
became healthy` beside a log showing a perfectly started server. R-4's advice to
bind `::` applies to environments created before 2025-10-16; a current one
resolves private DNS to both families, so `0.0.0.0` serves both purposes.

**4. `PORT` must be set explicitly.** Railway injects `8080`, the services
correctly honour it, and every private-network URL in this plan said `:8000` — so
nginx returned `502` for `/api/*` while both services were healthy. Pinning
`8000` and `8001` makes the other services' URLs right by construction, and
matches compose, which is what anyone debugging will compare against.

**The pattern worth taking from this**: every one of these presented as a
different problem than it was. A twelve-minute build that was a deploy. A build
failure with no build. A healthy server that was unreachable. A 502 from a
service that was up. In each case the log that mattered was one layer away from
the one being read.

---

## 11. Open Questions

Carried here rather than answered by assumption.
Expand Down
16 changes: 12 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -335,10 +335,18 @@ embedding and reranker configuration, and they must agree. `EMBEDDING_DIM` has
no production default on purpose: the migration bakes it into the pgvector
column, so a wrong value is a re-index rather than a slow query.

The production stack is validated locally only — `dcode.odieyang.com` does **not**
resolve publicly, so the external-demo exit criterion is still open. The plan for
closing it, including the platform constraints that shape it, is
[`Deploy.md`](Deploy.md).
**A gated deployment is live** at [`frontend-production-55c4.up.railway.app`](https://frontend-production-55c4.up.railway.app) —
Railway, HTTPS, one shared account, running the same three models the recorded
evaluation measured. The landing page and `/methodology` are public; the
workbench is not. Access is by request.

The deployment indexes `psf/requests` at current `HEAD`, which is **not** the
commit the H1 verdict was measured on — the verdict is a statement about the
committed runs in [`results/`](results/), not about what the demo is serving.

Service configuration, the variable sheet, and the four things that failed on
the first attempt: [`Deploy.md`](Deploy.md) and
[`infra/railway/`](infra/railway/).

---

Expand Down
25 changes: 20 additions & 5 deletions apps/api/tests/test_auth.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
read_session,
verify_password,
)
from dcode_api.deps import get_agent_client, get_redis
from dcode_api.deps import get_agent_client, get_db, get_redis
from dcode_api.main import app, create_app
from dcode_api.settings import api_settings
from fastapi.testclient import TestClient
Expand Down Expand Up @@ -254,10 +254,18 @@ def test_login_sets_a_hardened_cookie_and_opens_the_gate(gated: TestClient) -> N

# The client keeps the cookie, so a previously-401 route now resolves.
assert gated.get("/api/v1/auth/me").json()["authenticated"] is True
assert (
gated.get("/api/v1/repos/00000000-0000-0000-0000-000000000000/status").status_code
!= 401
)

# A protected route, with the database stubbed. It used to run against a
# real Postgres, which passed locally and failed in CI where there is none —
# and "the gate opened" is a claim about the dependency, not about the
# database. 404 for an unknown repo is the proof: the request got past
# require_session and into the handler.
app.dependency_overrides[get_db] = lambda: _NoSuchRepo()
try:
response = gated.get("/api/v1/repos/00000000-0000-0000-0000-000000000000/status")
finally:
app.dependency_overrides.clear()
assert response.status_code == 404


def test_logout_clears_the_session(gated: TestClient) -> None:
Expand Down Expand Up @@ -358,6 +366,13 @@ def test_quota_failure_lets_the_request_through(
app.dependency_overrides.clear()


class _NoSuchRepo:
"""A session that knows of no repositories, and needs no server."""

async def get(self, _model: object, _pk: object) -> None:
return None


class _NeverCalledAgent:
"""The quota check runs before the agent, so this only has to exist."""

Expand Down
9 changes: 9 additions & 0 deletions conftest.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,18 @@

# Auth defaults off; tests that want the gate turn it on by patching the
# settings object directly, which is explicit and machine-independent.
#
# INTERNAL_API_KEY is pinned for a sharper reason. The API and the agent refuse
# to start on the placeholder the settings module defaults to, so on a machine
# with no `.env` — which is exactly what CI is — every test that boots a
# lifespan died with "INTERNAL_API_KEY is still the placeholder". Locally it
# passed, because a developer's `.env` supplies a real value. That asymmetry is
# the whole reason this file exists, and pinning two settings and not the third
# left it in place.
os.environ.setdefault("_DCODE_TEST_ENV_PINNED", "1")
for name, value in (
("AUTH_ENABLED", "false"),
("DOCS_ENABLED", "true"),
("INTERNAL_API_KEY", "test-internal-key-not-the-published-placeholder"),
):
os.environ[name] = value
22 changes: 18 additions & 4 deletions docs/en/Final_Report.md
Original file line number Diff line number Diff line change
Expand Up @@ -638,10 +638,24 @@ graph-sensitive H1 re-run; everything else is independent of it.

### Deployment

- DNS for `dcode.odieyang.com` is unresolved; production Compose has only been
smoke-tested locally.
- Decide whether production runs the model sidecars or depends on external model
services.
- ~~DNS for `dcode.odieyang.com` is unresolved; production Compose has only been
smoke-tested locally.~~ **— closed 2026-08-03.** The stack is deployed and
reachable over HTTPS, gated by a single shared account, with a live index and
a query returning verified citations. It runs on Railway under a
platform-provided hostname rather than `dcode.odieyang.com`; a custom domain
is a DNS record away and is not what the criterion was about.
[`Deploy.md`](../../Deploy.md) records the four things that failed first.
- ~~Decide whether production runs the model sidecars or depends on external
model services.~~ **— decided: external.** The deployment calls the same three
models the recorded run measured — Jina v2-base-code at 768 dimensions, BGE
reranker v2-m3, `gpt-4o-mini` — through hosted APIs rather than self-hosted
sidecars, so no figure on `/methodology` describes a configuration other than
the one serving it. `Deploy.md` §2.
- **The deployed index is not the evaluated index.** Both are `psf/requests`, but
the recorded run is pinned to commit `414f0513`, and the deployment indexed
whatever `HEAD` was on 2026-08-03 — 773 chunks against the recorded 726. The
verdict remains a statement about the committed artifacts in `results/`, not
about whatever the demo happens to be serving.

## Known Limits

Expand Down
13 changes: 12 additions & 1 deletion infra/docker/agent.Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -31,4 +31,15 @@ EXPOSE 8001
# deploy it also could not find alembic at all, because the build-time sync had
# installed nothing (see the RUN above). apps/embedding already did it this way
# with a comment saying why; api, worker and agent had not caught up.
CMD ["sh", "-c", "exec /app/.venv/bin/uvicorn dcode_agent.main:app --host ${HOST:-::} --port ${PORT:-8001}"]
# 0.0.0.0, not `::`. Measured, not assumed: uvicorn given `::` produces an
# IPv6-ONLY socket — an IPv4 connect is refused even though the container has
# net.ipv6.bindv6only=0 — and given 0.0.0.0 the reverse. There is no dual-stack
# option, so this is a choice between them.
#
# Railway's health check arrives over IPv4: with `::` the request never reached
# the application at all and the deploy failed with "1/1 replicas never became
# healthy" while the log showed a perfectly started server. Private networking
# in environments created after 2025-10-16 resolves to both families, so IPv4
# serves both purposes. A legacy IPv6-only environment would need `HOST=::` and
# would then need the health check removed.
CMD ["sh", "-c", "exec /app/.venv/bin/uvicorn dcode_agent.main:app --host ${HOST:-0.0.0.0} --port ${PORT:-8001}"]
Loading
Loading