Repository navigation
Modernize the Docker deployment - #13
Conversation
This repository releases by moving the pins in .env and writing a dated CHANGELOG.md entry, and since the Keep a Changelog adoption its own changes have somewhere to go - but nothing made anyone write them down. A pull request that changes what a deployment is built from now has to touch CHANGELOG.md or carry the no-changelog label. RELEASING.md says how a bump is done, what a release is here, and where this repository sits in the ecosystem's release order. CI pins actions/checkout to a commit SHA, drops the checkout's credentials from the runner, and runs with a read-only token. Dependabot watches the pinned actions and the Ubuntu base images of both Dockerfiles, so a base image bump arrives as a reviewable change rather than as whatever the tag resolved to on rebuild day.
…steps at real documentation CHANGELOG.md already says CI changes belong under [Unreleased], but the changelog check exempted anything under .github/. Add it to the pattern. RELEASING.md told the bumper to preserve deviations "the README documents"; the README documents none. Point at the rationale comments in config/ and the changelog instead, and give the exact command for the no-changelog label the check depends on, which the repository does not yet have. The dependabot comment promised an Ubuntu LTS bump; the docker updater offers any newer tag, so say base-image bump.
Images
- Build in one apt layer with the lists removed, without apt-get upgrade,
with shallow clones and a pip cache mount. Drop gcc-multilib: nothing needs
a 32-bit toolchain and the package does not exist on arm64, so the images
now build on Apple Silicon. Replace locales-all with locales plus
locale-gen for en_US.UTF-8. Add .dockerignore files.
- mcritweb no longer bakes FLASK_DEBUG=1 into the image and is served by
gunicorn (installed explicitly, requirements.txt does not list it) instead
of the Flask development server.
- The mcrit image is built once and shared by server and worker.
Runtime
- Entry scripts run under set -eu and exec their process; the mcrit services
get init: true so SIGTERM reaches Python, and docker compose stop returns
in well under a second instead of the ten-second timeout.
- Healthchecks on mongodb, mcrit-server and mcritweb, depends_on conditions
instead of sleep 1, restart: unless-stopped in production, ${VAR:?} guards
on every interpolated tag, config and nginx mounts read-only.
- MongoDB 5.0 (end of life) moves to 8.0; the README documents the stepped
upgrade an existing storage/mongodb needs. mongod logs to stdout.
- nginx pinned through NGINX_TAG instead of nginx:latest; the TLS config
serves TLS 1.2 and 1.3 with the Mozilla intermediate ciphers, drops DHE and
ssl_dhparam, and no longer requests HSTS preload.
- nginx/ssl/*.pem are untracked (placeholders kept as .example) so a filled-in
private key cannot be committed by accident.
Scripts and CI
- build.sh uses docker compose v2; reset.sh and clone_repositories.sh get
shebangs, set -euo pipefail and no cd juggling; test_build.sh drops
--no-cache and builds the compose image names.
- CI builds both images with Buildx and a GitHub Actions layer cache, reads
the tags from .env, and has a timeout.
Verified locally: both images build, the stack comes up healthy, the web UI
answers through nginx and gunicorn, the mcrit API reports 1.9.0, and the
test entrypoint passes the 191 upstream unit tests.
Both images carried their build toolchain into the runtime and ran everything as root: build-essential, python3-dev, git and the pip cache stayed in the final layer, and a compromise of a web request had uid 0 inside the container. Split each Dockerfile into a builder stage that installs into /opt/venv and a runtime stage that copies the venv and the source across and adds only what running needs. The MCRIT image goes from 1.19 GB to 714 MB and the MCRITweb image from 1.28 GB to 787 MB, and both end on uid 10001. Running unprivileged makes ./storage/mcritweb, a host bind mount, something the operator has to hand to that uid once; both MCRITweb entry scripts now check the directory and exit with the chown command in the message instead of failing later inside Flask. entry_test.sh no longer pip-installs pytest at startup, which an unprivileged user cannot do: the image carries pytest and the script runs it directly, so it needs no make either. The base images are pinned by digest rather than by the 24.04 tag, so a rebuild cannot silently pick up a different Ubuntu, and both images carry OCI source, version and licence labels plus a .git-revision file recording the upstream commit they were built from - a label cannot hold a value resolved during the build. On the compose side, every service gets no-new-privileges, a json-file log cap of 5 files of 50 MB through one x-logging anchor so a chatty container cannot fill the host's disk, and MCRIT_AUTH_TOKEN passed through to the server and the worker. CI gains a lint job running hadolint over both Dockerfiles, docker compose config over both compose files and shellcheck over every entry script; .hadolint.yaml records why apt and pip version pinning are not enforced here. The README gains a production checklist covering TLS, the API token pairing with MCRITweb, the instance directory ownership and backups.
The README had grown into a mix of a project pitch, a partial setup guide and three version-specific maintenance notes, while the things a first deployment actually needs - the uid the containers run as, where the version pins live, what to change for TLS - were scattered or absent. It also still used docker-compose and duplicated release information that CHANGELOG.md owns. Restructure it in the order an operator meets the deployment: what is run and where the pins are, requirements, first deployment, configuration, production, upgrading, development mode, and a pointer to CHANGELOG.md for the version history. The MongoDB 5.0 stepping procedure stays because it is a one-directional host-level operation; the MCRIT 1.7.0 migration note and the SMDA fingerprint note are dropped in favour of the changelog entry and the upstream migration guide, which are where a release states what it requires. Every command is a fenced block using docker compose, and every path, variable and script name is checked against the tree.
danielplohmann
left a comment
There was a problem hiding this comment.
Reviewed by reading the diff and by building and running both images on an x86_64 host with a live deployment on it (you tested arm64). Every claim in the description checks out, several of them better than stated. Two things I would fix first, one of which is a silent-revert risk rather than a bug.
Verified rather than taken on trust
| claim | result |
|---|---|
| 191 unit tests, 91 mongo deselected | exact match — 191 passed, 91 deselected |
| unprivileged runtime | uid 10001 in both images |
| healthcheck | curl is in the runtime stage, /healthcheck is genuinely unrouted (404), and curl -s -o /dev/null exits 0 on a 404 — so it passes exactly when the server serves |
| gunicorn replaces the dev server | present; entry_web_prod.sh execs it |
| image sizes | better than the description: mcrit 1.3 GB → 483 MB, mcritweb 1.35 GB → 528 MB (you quote 714 MB / 787 MB from arm64) |
I also reproduced the MongoDB step independently, since the host runs 5.0.12 at FCV 5.0 on 47 GB:
mongo:8.0 on FCV-5.0 data → exit 62
"UPGRADE PROBLEM: ... Invalid feature compatibility version value '5.0'; expected '7.0' or '7.3' or '8.0'"
and 5.0 → 6.0 → 7.0 → 8.0 preserves the data at every step, needing confirm: true from 7.0 onwards — exactly as your README says. Corroboration, not a finding.
1. The escaper fingerprint section is dropped, and it lands nowhere
The description calls the removed README sections "three version-specific maintenance notes". For the 1.7.0 disassembly split that is right. "Checking the SMDA escaper fingerprint after a rebuild" is not version-specific — it documents a standing property of the deployment:
MCRIT does not pin SMDA, so rebuilding the images can pull a newer SMDA whose instruction escaping differs. [...] previously indexed signatures stop being comparable to newly computed ones. They still look valid and still match each other, while identical code submitted afterwards no longer finds them.
Searched rather than assumed: zero occurrences of "escaper" in this PR's README.md, RELEASING.md, or in #16's README. The one CHANGELOG.md hit is a table cell about repair_minhashes, not the check.
This matters most right here: this PR is itself a rebuild-everything change, which is precisely when the escaper can move. The worked example it cites, SMDA 4.4.5, altered about a fifth of the stored MinHashes on a real corpus, and it is a failure that is silent by construction — recall quietly drops and nothing errors.
It is six lines and a curl. The natural home is the new ## Upgrading section, next to the MongoDB steps, which is where an operator is already being told what a rebuild costs them.
2. This conflicts with #17 exactly where the clone URL lives
Tested in both directions: README.md, docker/mcrit/Dockerfile and docker/mcritweb/Dockerfile conflict either way, and the Dockerfile hunks are the git clone lines that #17 rewrites to familiary/*.
Resolving toward this branch silently reverts the rename, and nothing fails — GitHub's redirect keeps danielplohmann/mcrit cloning perfectly, so the regression stays invisible until the redirect one day does not. Whoever merges second should diff the two clone URLs afterwards rather than trusting the resolution.
Minor, none blocking
- Dev mode disables pytest's cache. The Dockerfile chowns
/opt/mcritto 10001 soentry_test.shcan write.pytest_cache, butdocker-compose-dev.ymlmounts the host checkout over it and restores host ownership. Measured: the suite still passes 191/191, emitting onlyPytestCacheWarning: could not create cache path. Cosmetic, but the Dockerfile comment implies a guarantee the dev file removes. logs/mongodb/.emptyis added while mongod stops writing a logfile (--logpathdropped for stdout plus the json-file cap), so that directory now has no writer.- Folding
mcrit-server/mcrit-workerinto onemcritimage is a real dedup — the worker was building a byte-identical second copy. Existing deployments keep the old tags as orphans; worth a line in the upgrade notes. RUN --mount=type=cachemakes BuildKit mandatory.build.shandtest_build.shboth go throughdocker compose build, which supplies it, so the supported path is fine — a plaindocker buildfails on a daemon without buildx, as it did here.
One note for existing deployments
storage/mcritweb is root:root on this host, so uid 10001 cannot write it and chown -R 10001:10001 is required. This is handled well — entry_web_prod.sh checks writability up front and exits with the exact command, rather than failing obscurely later.
🤖 Generated with Claude Code
The rewrite dropped three maintenance sections as version-specific. Two of them were: the 1.7.0 disassembly split is a one-time migration, and the upstream guide owns it. The escaper fingerprint check is not version-specific. MCRIT does not pin SMDA, so any rebuild can pull a newer one whose instruction escaping differs, and MinHashes are derived from the escaped representation. The failure is silent by construction: old signatures still look valid and still match each other, while identical code submitted afterwards stops finding them. SMDA 4.4.5 altered about a fifth of the stored MinHashes on a real corpus. That matters most in this pull request, which rebuilds both images from scratch - a rebuild is exactly when the escaper can move, and the check was the only thing telling an operator to look. Restored under ## Upgrading, beside the MongoDB steps, where an operator is already being told what an upgrade costs them. Shortened to fit the new structure, and the command is verified against a running instance: smda_version: 4.5.0 escaper_fingerprint: 110c9d152232a2e6 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GwSfzAD1ZwEoWY3eWvbYvX
|
Pushed the escaper fix from my review directly to the branch ( It restores the fingerprint check as a Shortened to fit the new structure, and the command is verified against a running instance rather than copied forward: README only, 21 lines added, nothing else touched. Your other two dropped sections stay dropped — the 1.7.0 disassembly split is genuinely a one-time migration and the upstream guide owns it. 🤖 Generated with Claude Code |
danielplohmann
left a comment
There was a problem hiding this comment.
Approved — CI green (build-test, lint, changelog gate, TUNING.md mirror). Reviewed in detail on #13; the increments here are small and sit on that base.
The merge-base changed after approval.
MCRIT, MCRITweb and this repository now live under github.com/familiary. The old paths redirect, so nothing is broken today - the images build and clone_repositories.sh works. But a redirect is not a guarantee: it stops working the moment a repository of the same name is created under the old owner, and the two places that would break are build-time clones, where the failure lands on whoever is deploying. Rebuilt on top of the modernization rather than merged into it. The original branch predated #13, so merging it produced a conflict in both Dockerfiles between the old single-stage structure carrying familiary URLs and the new multi-stage structure carrying the old ones - a conflict where taking either side alone is wrong, and where taking "theirs" silently restores the old URLs while still building fine. Deliberately unchanged: - danielplohmann/smda and danielplohmann/purepdb in RELEASING.md, which have not moved - checked, not assumed. - docs/TUNING.md, a verbatim mirror of MCRIT's copy that the tuning-doc-mirror job diffs against the canonical version. - CHANGELOG.md history, apart from the new entry this change itself needs. - .github/workflows/test.yml, which still fetches TUNING.md from the old org via raw.githubusercontent.com. That redirect resolves, and changing it needs a token with the workflow scope. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GwSfzAD1ZwEoWY3eWvbYvX
This is stacked on #12 and will be rebased once that merges; its first two commits are #12 itself plus two follow-ups to it, and the change proper starts at 470f4f0. The follow-ups are small but load-bearing: the changelog check exempted .github/ although CHANGELOG.md says CI changes belong in it, and RELEASING.md cited README documentation of the config deviations that does not exist, so it now points at the rationale comments in config/ and at the changelog, and gives the exact gh label create command for the no-changelog label the check depends on. That label has to exist before #12 merges or the check has no escape hatch. The deployment this repository ships was built for a demonstration rather than for a server it stays on: MongoDB 5.0 has been out of support since October 2024, NGINX was pulled as latest, MCRITweb was served by the Flask development server with FLASK_DEBUG baked into the image, services waited for each other with sleep 1, and both images carried their build toolchain into the runtime and ran everything as root.
The pull request rebuilds the deployment around what it takes to leave it running. Each image is split into a builder stage that installs into a virtualenv and a runtime stage that copies the venv and the source across, ending on uid 10001; the base images are pinned by digest rather than by the 24.04 tag, and each carries OCI source, version and licence labels plus a .git-revision file, because a label cannot hold a value resolved during the build. Healthchecks and depends_on conditions replace the sleeps, the entry scripts exec their process under init: true so signals reach it, gunicorn replaces the development server, MongoDB moves to 8.0, and NGINX is pinned through NGINX_TAG with a TLS configuration serving 1.2 and 1.3 on the Mozilla intermediate ciphers. On the compose side every service gets no-new-privileges, a 250 MB json-file log cap through one anchor, and MCRIT_AUTH_TOKEN passed to the server and the worker. The private keys under nginx/ssl are untracked, with the placeholders kept as .example files.
Built and run on arm64: both images build, docker compose up -d --wait reports all five services healthy, the web interface answers through NGINX and gunicorn, and docker run --rm --entrypoint /entry_test.sh mcrit:1.9.0 passes 191 unit tests with 91 mongo tests deselected. The staging work is what the size numbers come from: the MCRIT image drops from 1.19 GB to 714 MB and the MCRITweb image from 1.28 GB to 787 MB. shellcheck, hadolint, actionlint and docker compose config are clean on both compose files, and CI now runs the first three as a lint job of its own. The README is rewritten around the order an operator meets the deployment, and the three version-specific maintenance notes it carried are dropped in favour of CHANGELOG.md and the upstream migration guide, which own that information.
Left out deliberately: the clones are still by tag rather than by commit, and the pip installs are not hash-locked, so a rebuild of an old tag is reproducible only as far as the upstream tag and the digest-pinned base make it. MCRIT stays an editable install because the compose files mount config/ over the package's own config directory. The uid the containers run as is a one-time cost for existing deployments, which have to hand ./storage/mcritweb to it; MCRITweb exits with that command in the message rather than failing later. Three further pull requests are stacked on this one, addressing #9, #3 and #8.