Repository navigation
fix(deploy): make the images self-contained instead of installing at start - #24
Merged
Merged
Conversation
…start The first Railway deploy of `api` spent twelve minutes and then failed with `error: Failed to spawn: alembic`. Two causes, both invisible locally. `uv sync --no-dev` at the workspace root syncs the ROOT project, not the members. The root pyproject declares no dependencies, so that build step installed nothing — the local images have two entries in site-packages and no alembic. Everything was really being installed by `uv run` at container start, every start. `--package dcode-api` (and the equivalent for worker and agent) makes the build actually produce a usable environment. Then `uv run` re-syncs at start, and without --no-dev it pulls the dev group: the deploy log shows it downloading mypy, ruff and pygments before doing anything. That is where the twelve minutes went, on a loop, and it still could not find alembic because the preDeployCommand ran before any of this had a chance to help. So every entrypoint now calls the venv binary directly and `uv run` is gone from infra/ entirely. apps/embedding already did exactly this, with a comment saying "avoid `uv run` reinstalling packages on restart" — the lesson was learned in one image and not the other four. Verified rather than assumed. My first diagnosis was that the workspace sync was dragging in torch from the two model sidecars; that was wrong — the old images contain no torch, because the sync was a no-op. The build log settled it: `Resolved 109 packages in 774ms`, `Checked in 1ms`, and the twelve minutes were in the deploy phase. What is now verified: the api image runs alembic and uvicorn with `--network none` in under a second, where the old one could not run at all without downloading first; all four images build; the local compose stack still comes up healthy with uvicorn as PID 1, and /readyz reports all three checks green. Also the first actual run of the combined agent-worker entrypoint, which had never been executed anywhere. Both children start and the worker retries its broker. On Railway it exited with "a child exited with 1; stopping the other" when RabbitMQ was unreachable — which is the designed behaviour working, not a bug: a dead worker takes the container down instead of leaving one that answers questions while every submitted repository sits in `queued` forever.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found by the first real Railway deploy.
apispent twelve minutes and then failed witherror: Failed to spawn: alembic.Two causes, both invisible locally
1. The build-time sync installed nothing.
uv sync --no-devat a workspace root syncs the root project, not the members — and the rootpyproject.tomldeclares no dependencies. The local images have two entries insite-packagesand no alembic. Everything was really installed byuv runat container start, on every start.2.
uv runpulls the dev group. The deploy log shows it downloadingmypy,ruffandpygmentsbefore doing anything useful. That is where the twelve minutes went, on a retry loop — and it still could not find alembic, becausepreDeployCommandran before any of it could help.The fix
uv sync --no-dev --package <pkg>so the build produces a usable environment (--package dcode-agent --package dcode-workerfor the combined image).uv runis now gone frominfra/entirely.apps/embeddingalready did exactly this, with a comment reading "avoiduv runreinstalling packages on restart". The lesson had been learned in one image and not the other four.A correction
My first diagnosis was that the root sync was dragging in torch from the two model sidecars. That was wrong — the old images contain no torch, precisely because the sync was a no-op. The build log settled it:
Resolved 109 packages in 774ms,Checked in 1ms, and the twelve minutes were in the deploy phase, not the build.Verified
alembicanduvicornunder--network nonein <1s. The old image could not run at all without downloading first/readyzall three checks greenagent-workerentrypointOn Railway that entrypoint exited with
a child exited with 1; stopping the otherbecause RabbitMQ was unreachable. That is the designed behaviour working: a dead worker takes the container down rather than leaving one that answers questions while every submitted repository sits inqueuedforever.make checkgreen: 394 passed / 5 skipped, 86 frontend tests, mypy strict over 88 files.