Skip to content

fix(deploy): make the images self-contained instead of installing at start - #24

Merged
Odiethebest merged 1 commit into
mainfrom
fix/deploy-startup-cost
Aug 3, 2026
Merged

Odiethebest merged 1 commit into
mainfrom
fix/deploy-startup-cost

Conversation

@Odiethebest

Copy link
Copy Markdown
Owner

Found by the first real Railway deploy. api spent twelve minutes and then failed with error: Failed to spawn: alembic.

Two causes, both invisible locally

1. The build-time sync installed nothing. uv sync --no-dev at a workspace root syncs the root project, not the members — and the root pyproject.toml declares no dependencies. The local images have two entries in site-packages and no alembic. Everything was really installed by uv run at container start, on every start.

2. uv run pulls the dev group. The deploy log shows it downloading mypy, ruff and pygments before doing anything useful. That is where the twelve minutes went, on a retry loop — and it still could not find alembic, because preDeployCommand ran before any of it could help.

The fix

  • uv sync --no-dev --package <pkg> so the build produces a usable environment (--package dcode-agent --package dcode-worker for the combined image).
  • Every entrypoint calls the venv binary directly. uv run is now gone from infra/ entirely.

apps/embedding already did exactly this, with a comment reading "avoid uv run reinstalling packages on restart". The lesson had been learned in one image and not the other four.

A correction

My first diagnosis was that the root sync was dragging in torch from the two model sidecars. That was wrong — the old images contain no torch, precisely because the sync was a no-op. The build log settled it: Resolved 109 packages in 774ms, Checked in 1ms, and the twelve minutes were in the deploy phase, not the build.

Verified

Check Result
api image offline runs alembic and uvicorn under --network none in <1s. The old image could not run at all without downloading first
All four images build
Local compose comes up healthy, uvicorn as PID 1 (exec semantics kept), /readyz all three checks green
Combined agent-worker entrypoint first actual execution anywhere. Both children start; the worker retries its broker

On Railway that entrypoint exited with a child exited with 1; stopping the other because RabbitMQ was unreachable. That is the designed behaviour working: a dead worker takes the container down rather than leaving one that answers questions while every submitted repository sits in queued forever.

make check green: 394 passed / 5 skipped, 86 frontend tests, mypy strict over 88 files.

…start

The first Railway deploy of `api` spent twelve minutes and then failed with
`error: Failed to spawn: alembic`. Two causes, both invisible locally.

`uv sync --no-dev` at the workspace root syncs the ROOT project, not the
members. The root pyproject declares no dependencies, so that build step
installed nothing — the local images have two entries in site-packages and
no alembic. Everything was really being installed by `uv run` at container
start, every start. `--package dcode-api` (and the equivalent for worker and
agent) makes the build actually produce a usable environment.

Then `uv run` re-syncs at start, and without --no-dev it pulls the dev
group: the deploy log shows it downloading mypy, ruff and pygments before
doing anything. That is where the twelve minutes went, on a loop, and it
still could not find alembic because the preDeployCommand ran before any of
this had a chance to help.

So every entrypoint now calls the venv binary directly and `uv run` is gone
from infra/ entirely. apps/embedding already did exactly this, with a
comment saying "avoid `uv run` reinstalling packages on restart" — the
lesson was learned in one image and not the other four.

Verified rather than assumed. My first diagnosis was that the workspace sync
was dragging in torch from the two model sidecars; that was wrong — the old
images contain no torch, because the sync was a no-op. The build log settled
it: `Resolved 109 packages in 774ms`, `Checked in 1ms`, and the twelve
minutes were in the deploy phase.

What is now verified: the api image runs alembic and uvicorn with
`--network none` in under a second, where the old one could not run at all
without downloading first; all four images build; the local compose stack
still comes up healthy with uvicorn as PID 1, and /readyz reports all three
checks green.

Also the first actual run of the combined agent-worker entrypoint, which had
never been executed anywhere. Both children start and the worker retries its
broker. On Railway it exited with "a child exited with 1; stopping the
other" when RabbitMQ was unreachable — which is the designed behaviour
working, not a bug: a dead worker takes the container down instead of
leaving one that answers questions while every submitted repository sits in
`queued` forever.
@Odiethebest
Odiethebest merged commit 6875101 into main Aug 3, 2026
1 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant