Repository navigation
fix(deploy): correct what the first Railway deploy disproved, and record it - #25
Merged
Merged
Conversation
All three were written from documentation and contradicted by the platform. The volume must not be mounted under /tmp. Railway refuses it and fails the deployment before producing a single log line — no build log, no deploy log, just FAILED. That is why agent-worker looked identical to a dozen other causes and took the longest to find: the only signal was that it was the one app service with a volume. Moved to /data/workdirs. The /tmp default stays right for compose. HOST must be 0.0.0.0, not `::`. Measured rather than reasoned: uvicorn given `::` produces an IPv6-only socket — an IPv4 connect is refused even with net.ipv6.bindv6only=0 — and given 0.0.0.0 the reverse. There is no dual-stack option. Railway's health check arrives over IPv4, so with `::` the request never reached the application and the deploy failed with "1/1 replicas never became healthy" next to a log showing a perfectly started server. I had set `::` on the strength of Railway's private-networking note, which applies to environments created before 2025-10-16. PORT has to be set explicitly. Railway injects 8080, our processes correctly honour it, and every private URL I had written said :8000 — so nginx returned 502 for /api/* while both services were healthy. Pinning 8000 and 8001 makes the other services' URLs correct by construction instead of by luck, and matches compose, which is the thing anyone will compare against. The Railway README no longer says nothing has been deployed. It is corrected against what happened, with each of the four failures noted where it bites.
Records what was checked against the running system rather than what was intended, and strikes the claims that are no longer true. Verified on the deployment: 773/773 vectors at 768 dimensions with zero all-zero rows and 770 distinct prefixes, so the hosted embedding path produces real varied embeddings and not stubs; /readyz green on all three checks; 401 on every protected route and a public landing and methodology page; one query end to end with three citations all verified and groundedness 1.0. The two rows that needed eyes were confirmed in a browser. Two things are recorded as NOT verified rather than assumed. The reranker two-mode comparison cannot be run from outside, because /internal/* is not publicly routed — it was verified locally instead. And the R-2 heartbeat has never been needed: every live query finished well inside the platform's idle window, so the keep-alive stays correct-by-construction and unobserved. Also recorded: the deployed index is NOT the evaluated index. Both are psf/requests, but the recorded run is pinned to 414f0513 and the deployment indexed current HEAD — 773 chunks against the recorded 726. The verdict remains a statement about the committed runs in results/, and the README and Final_Report now say so rather than leaving a reader to conflate the two. §10.3 is new: the four first-deploy failures, all of them mistakes in this plan and three of them things written from documentation and not measured. The pattern is the part worth keeping — every one presented as a different problem than it was. A twelve-minute build that was a deploy. A build failure with no build. A healthy server that was unreachable. A 502 from a service that was up. Each time the log that mattered was one layer away from the one being read. Outstanding Work loses both Deployment items: the external-demo criterion is closed, and the sidecars-versus-hosted-APIs question is decided as hosted, running the same three models so no displayed figure describes a configuration other than the one serving it.
CI on main has been red since #21 and I did not look. Every report of "make check green" in this series was true and insufficient: it passes locally because a developer's .env supplies a real INTERNAL_API_KEY and a Postgres is running. CI has neither, and CI is the hermetic environment that exists to catch exactly this. Two failures, both mine. #21 added a test that asserted a protected route stops returning 401 by hitting /repos/{id}/status, which connects to Postgres. The claim being tested is "the cookie got past require_session", which is about the dependency and not about the database. It now stubs the session and asserts 404 for an unknown repo — the request reached the handler, which is the actual proof. #23 added a startup guard refusing the placeholder INTERNAL_API_KEY, so on a machine with no .env every test that boots a lifespan died with "still the placeholder". conftest.py now pins it alongside AUTH_ENABLED and DOCS_ENABLED. That file was written for precisely this asymmetry, and pinning two settings and not the third left the hole open — the guard just made it fatal instead of silent. Verified in both environments rather than one: 394 passed with .env present, and 394 passed with .env moved aside and DATABASE_URL pointed at a dead port, which is the closest local reproduction of CI.
Adds the working agreement, because the lesson cost five merged PRs and a red main nobody was looking at. Every report in the deployment series said "make check green". Every one was true. The local run passes because a developer's .env supplies a real INTERNAL_API_KEY and a Postgres is running; CI has neither, which is the entire point of CI. So a test that quietly depends on the machine it runs on is invisible locally *by construction* — local success is evidence about the laptop, not about the change. The agreement is therefore two things, not one: check CI after every merge, and say which of the two claims you are making. Plus the reproduction recipe, so a CI-only failure gets measured rather than guessed at — move .env aside, point DATABASE_URL at a dead port, run the suite.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The stack is live and verified: https://frontend-production-55c4.up.railway.app
Four things failed on the first attempt. All four were mistakes in this plan, not platform faults — three were written from documentation and never measured.
Deploy.md§10.3 has the full account.The four
1.
uv runwas installing dependencies at container start.apispent twelve minutes and failed withFailed to spawn: alembic.uv sync --no-devat a workspace root syncs the root project, which declares no dependencies — so the build installed nothing anduv rundid the real work at start, downloadingmypyandrufffirst. (Fixed in #24.)2. A volume cannot be mounted under
/tmp. Railway refuses and fails the deployment before emitting a single log line — no build log, no deploy log, justFAILED. This took the longest to find; the only usable signal was thatagent-workerwas the one app service with a volume.3.
HOST=::produces an IPv6-only socket. Measured after assuming otherwise: uvicorn given::refuses IPv4 even withnet.ipv6.bindv6only=0, and given0.0.0.0refuses IPv6. No dual-stack option. Railway's health check arrives over IPv4, so the probe never reached the app and the deploy failed with1/1 replicas never became healthybeside a log showing a perfectly started server. R-4's advice to bind::applies to environments created before 2025-10-16.4.
PORTmust be set explicitly. Railway injects8080, the services correctly honour it, and every private URL here said:8000— nginx returned502for/api/*while both services were healthy.The pattern is the part worth keeping. Every one presented as a different problem than it was: a twelve-minute build that was a deploy, a build failure with no build, a healthy server that was unreachable, a 502 from a service that was up. Each time the log that mattered was one layer away from the one being read.
Verified on the deployment
200 OK, no retries, no rate limiting/readyz/and/methodologypublic; cookieHttpOnly; Secure; SameSite=laxThis closes the last open item in
Deploy.md§6 PR 2 — an index built through the hosted provider — and both Deployment entries inFinal_Report.mdOutstanding Work.Recorded as not verified
/internal/*is not publicly routed. Verified locally instead.One thing a reader could get wrong
The deployed index is not the evaluated index. Both are
psf/requests, but the recorded run is pinned to414f0513and the deployment indexed currentHEAD— 773 chunks against the recorded 726. The verdict is a statement about the committed runs inresults/, not about what the demo serves. The README andFinal_Report.mdnow say so.make checkgreen: 394 passed / 5 skipped, 86 frontend tests, mypy strict over 88 files.