Skip to content

Modernize the Docker deployment - #13

Merged
danielplohmann merged 6 commits into
familiary:mainfrom
r0ny123:docker/modernize
Sep 23, 2026
Merged

danielplohmann merged 6 commits into
familiary:mainfrom
r0ny123:docker/modernize

Conversation

@r0ny123

@r0ny123 r0ny123 commented Sep 15, 2026 •

Copy link
Copy Markdown
Collaborator

This is stacked on #12 and will be rebased once that merges; its first two commits are #12 itself plus two follow-ups to it, and the change proper starts at 470f4f0. The follow-ups are small but load-bearing: the changelog check exempted .github/ although CHANGELOG.md says CI changes belong in it, and RELEASING.md cited README documentation of the config deviations that does not exist, so it now points at the rationale comments in config/ and at the changelog, and gives the exact gh label create command for the no-changelog label the check depends on. That label has to exist before #12 merges or the check has no escape hatch. The deployment this repository ships was built for a demonstration rather than for a server it stays on: MongoDB 5.0 has been out of support since October 2024, NGINX was pulled as latest, MCRITweb was served by the Flask development server with FLASK_DEBUG baked into the image, services waited for each other with sleep 1, and both images carried their build toolchain into the runtime and ran everything as root.

The pull request rebuilds the deployment around what it takes to leave it running. Each image is split into a builder stage that installs into a virtualenv and a runtime stage that copies the venv and the source across, ending on uid 10001; the base images are pinned by digest rather than by the 24.04 tag, and each carries OCI source, version and licence labels plus a .git-revision file, because a label cannot hold a value resolved during the build. Healthchecks and depends_on conditions replace the sleeps, the entry scripts exec their process under init: true so signals reach it, gunicorn replaces the development server, MongoDB moves to 8.0, and NGINX is pinned through NGINX_TAG with a TLS configuration serving 1.2 and 1.3 on the Mozilla intermediate ciphers. On the compose side every service gets no-new-privileges, a 250 MB json-file log cap through one anchor, and MCRIT_AUTH_TOKEN passed to the server and the worker. The private keys under nginx/ssl are untracked, with the placeholders kept as .example files.

Built and run on arm64: both images build, docker compose up -d --wait reports all five services healthy, the web interface answers through NGINX and gunicorn, and docker run --rm --entrypoint /entry_test.sh mcrit:1.9.0 passes 191 unit tests with 91 mongo tests deselected. The staging work is what the size numbers come from: the MCRIT image drops from 1.19 GB to 714 MB and the MCRITweb image from 1.28 GB to 787 MB. shellcheck, hadolint, actionlint and docker compose config are clean on both compose files, and CI now runs the first three as a lint job of its own. The README is rewritten around the order an operator meets the deployment, and the three version-specific maintenance notes it carried are dropped in favour of CHANGELOG.md and the upstream migration guide, which own that information.

Left out deliberately: the clones are still by tag rather than by commit, and the pip installs are not hash-locked, so a rebuild of an old tag is reproducible only as far as the upstream tag and the digest-pinned base make it. MCRIT stays an editable install because the compose files mount config/ over the package's own config directory. The uid the containers run as is a one-time cost for existing deployments, which have to hand ./storage/mcritweb to it; MCRITweb exits with that command in the message rather than failing later. Three further pull requests are stacked on this one, addressing #9, #3 and #8.

This repository releases by moving the pins in .env and writing a dated
CHANGELOG.md entry, and since the Keep a Changelog adoption its own changes
have somewhere to go - but nothing made anyone write them down. A pull
request that changes what a deployment is built from now has to touch
CHANGELOG.md or carry the no-changelog label. RELEASING.md says how a bump
is done, what a release is here, and where this repository sits in the
ecosystem's release order.

CI pins actions/checkout to a commit SHA, drops the checkout's credentials
from the runner, and runs with a read-only token. Dependabot watches the
pinned actions and the Ubuntu base images of both Dockerfiles, so a base
image bump arrives as a reviewable change rather than as whatever the tag
resolved to on rebuild day.
…steps at real documentation

CHANGELOG.md already says CI changes belong under [Unreleased], but the
changelog check exempted anything under .github/. Add it to the pattern.

RELEASING.md told the bumper to preserve deviations "the README documents";
the README documents none. Point at the rationale comments in config/ and the
changelog instead, and give the exact command for the no-changelog label the
check depends on, which the repository does not yet have.

The dependabot comment promised an Ubuntu LTS bump; the docker updater offers
any newer tag, so say base-image bump.
Images
- Build in one apt layer with the lists removed, without apt-get upgrade,
  with shallow clones and a pip cache mount. Drop gcc-multilib: nothing needs
  a 32-bit toolchain and the package does not exist on arm64, so the images
  now build on Apple Silicon. Replace locales-all with locales plus
  locale-gen for en_US.UTF-8. Add .dockerignore files.
- mcritweb no longer bakes FLASK_DEBUG=1 into the image and is served by
  gunicorn (installed explicitly, requirements.txt does not list it) instead
  of the Flask development server.
- The mcrit image is built once and shared by server and worker.

Runtime
- Entry scripts run under set -eu and exec their process; the mcrit services
  get init: true so SIGTERM reaches Python, and docker compose stop returns
  in well under a second instead of the ten-second timeout.
- Healthchecks on mongodb, mcrit-server and mcritweb, depends_on conditions
  instead of sleep 1, restart: unless-stopped in production, ${VAR:?} guards
  on every interpolated tag, config and nginx mounts read-only.
- MongoDB 5.0 (end of life) moves to 8.0; the README documents the stepped
  upgrade an existing storage/mongodb needs. mongod logs to stdout.
- nginx pinned through NGINX_TAG instead of nginx:latest; the TLS config
  serves TLS 1.2 and 1.3 with the Mozilla intermediate ciphers, drops DHE and
  ssl_dhparam, and no longer requests HSTS preload.
- nginx/ssl/*.pem are untracked (placeholders kept as .example) so a filled-in
  private key cannot be committed by accident.

Scripts and CI
- build.sh uses docker compose v2; reset.sh and clone_repositories.sh get
  shebangs, set -euo pipefail and no cd juggling; test_build.sh drops
  --no-cache and builds the compose image names.
- CI builds both images with Buildx and a GitHub Actions layer cache, reads
  the tags from .env, and has a timeout.

Verified locally: both images build, the stack comes up healthy, the web UI
answers through nginx and gunicorn, the mcrit API reports 1.9.0, and the
test entrypoint passes the 191 upstream unit tests.
Both images carried their build toolchain into the runtime and ran everything
as root: build-essential, python3-dev, git and the pip cache stayed in the
final layer, and a compromise of a web request had uid 0 inside the container.
Split each Dockerfile into a builder stage that installs into /opt/venv and a
runtime stage that copies the venv and the source across and adds only what
running needs. The MCRIT image goes from 1.19 GB to 714 MB and the MCRITweb
image from 1.28 GB to 787 MB, and both end on uid 10001.

Running unprivileged makes ./storage/mcritweb, a host bind mount, something
the operator has to hand to that uid once; both MCRITweb entry scripts now
check the directory and exit with the chown command in the message instead of
failing later inside Flask. entry_test.sh no longer pip-installs pytest at
startup, which an unprivileged user cannot do: the image carries pytest and
the script runs it directly, so it needs no make either.

The base images are pinned by digest rather than by the 24.04 tag, so a
rebuild cannot silently pick up a different Ubuntu, and both images carry OCI
source, version and licence labels plus a .git-revision file recording the
upstream commit they were built from - a label cannot hold a value resolved
during the build.

On the compose side, every service gets no-new-privileges, a json-file log cap
of 5 files of 50 MB through one x-logging anchor so a chatty container cannot
fill the host's disk, and MCRIT_AUTH_TOKEN passed through to the server and
the worker. CI gains a lint job running hadolint over both Dockerfiles,
docker compose config over both compose files and shellcheck over every entry
script; .hadolint.yaml records why apt and pip version pinning are not
enforced here. The README gains a production checklist covering TLS, the API
token pairing with MCRITweb, the instance directory ownership and backups.
The README had grown into a mix of a project pitch, a partial setup guide and
three version-specific maintenance notes, while the things a first deployment
actually needs - the uid the containers run as, where the version pins live,
what to change for TLS - were scattered or absent. It also still used
docker-compose and duplicated release information that CHANGELOG.md owns.

Restructure it in the order an operator meets the deployment: what is run and
where the pins are, requirements, first deployment, configuration, production,
upgrading, development mode, and a pointer to CHANGELOG.md for the version
history. The MongoDB 5.0 stepping procedure stays because it is a
one-directional host-level operation; the MCRIT 1.7.0 migration note and the
SMDA fingerprint note are dropped in favour of the changelog entry and the
upstream migration guide, which are where a release states what it requires.
Every command is a fenced block using docker compose, and every path,
variable and script name is checked against the tree.

@danielplohmann danielplohmann left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed by reading the diff and by building and running both images on an x86_64 host with a live deployment on it (you tested arm64). Every claim in the description checks out, several of them better than stated. Two things I would fix first, one of which is a silent-revert risk rather than a bug.

Verified rather than taken on trust

claim result
191 unit tests, 91 mongo deselected exact match — 191 passed, 91 deselected
unprivileged runtime uid 10001 in both images
healthcheck curl is in the runtime stage, /healthcheck is genuinely unrouted (404), and curl -s -o /dev/null exits 0 on a 404 — so it passes exactly when the server serves
gunicorn replaces the dev server present; entry_web_prod.sh execs it
image sizes better than the description: mcrit 1.3 GB → 483 MB, mcritweb 1.35 GB → 528 MB (you quote 714 MB / 787 MB from arm64)

I also reproduced the MongoDB step independently, since the host runs 5.0.12 at FCV 5.0 on 47 GB:

mongo:8.0 on FCV-5.0 data → exit 62
"UPGRADE PROBLEM: ... Invalid feature compatibility version value '5.0'; expected '7.0' or '7.3' or '8.0'"

and 5.0 → 6.0 → 7.0 → 8.0 preserves the data at every step, needing confirm: true from 7.0 onwards — exactly as your README says. Corroboration, not a finding.

1. The escaper fingerprint section is dropped, and it lands nowhere

The description calls the removed README sections "three version-specific maintenance notes". For the 1.7.0 disassembly split that is right. "Checking the SMDA escaper fingerprint after a rebuild" is not version-specific — it documents a standing property of the deployment:

MCRIT does not pin SMDA, so rebuilding the images can pull a newer SMDA whose instruction escaping differs. [...] previously indexed signatures stop being comparable to newly computed ones. They still look valid and still match each other, while identical code submitted afterwards no longer finds them.

Searched rather than assumed: zero occurrences of "escaper" in this PR's README.md, RELEASING.md, or in #16's README. The one CHANGELOG.md hit is a table cell about repair_minhashes, not the check.

This matters most right here: this PR is itself a rebuild-everything change, which is precisely when the escaper can move. The worked example it cites, SMDA 4.4.5, altered about a fifth of the stored MinHashes on a real corpus, and it is a failure that is silent by construction — recall quietly drops and nothing errors.

It is six lines and a curl. The natural home is the new ## Upgrading section, next to the MongoDB steps, which is where an operator is already being told what a rebuild costs them.

2. This conflicts with #17 exactly where the clone URL lives

Tested in both directions: README.md, docker/mcrit/Dockerfile and docker/mcritweb/Dockerfile conflict either way, and the Dockerfile hunks are the git clone lines that #17 rewrites to familiary/*.

Resolving toward this branch silently reverts the rename, and nothing fails — GitHub's redirect keeps danielplohmann/mcrit cloning perfectly, so the regression stays invisible until the redirect one day does not. Whoever merges second should diff the two clone URLs afterwards rather than trusting the resolution.

Minor, none blocking

  • Dev mode disables pytest's cache. The Dockerfile chowns /opt/mcrit to 10001 so entry_test.sh can write .pytest_cache, but docker-compose-dev.yml mounts the host checkout over it and restores host ownership. Measured: the suite still passes 191/191, emitting only PytestCacheWarning: could not create cache path. Cosmetic, but the Dockerfile comment implies a guarantee the dev file removes.
  • logs/mongodb/.empty is added while mongod stops writing a logfile (--logpath dropped for stdout plus the json-file cap), so that directory now has no writer.
  • Folding mcrit-server/mcrit-worker into one mcrit image is a real dedup — the worker was building a byte-identical second copy. Existing deployments keep the old tags as orphans; worth a line in the upgrade notes.
  • RUN --mount=type=cache makes BuildKit mandatory. build.sh and test_build.sh both go through docker compose build, which supplies it, so the supported path is fine — a plain docker build fails on a daemon without buildx, as it did here.

One note for existing deployments

storage/mcritweb is root:root on this host, so uid 10001 cannot write it and chown -R 10001:10001 is required. This is handled well — entry_web_prod.sh checks writability up front and exits with the exact command, rather than failing obscurely later.

🤖 Generated with Claude Code

https://claude.ai/code/session_01GwSfzAD1ZwEoWY3eWvbYvX

The rewrite dropped three maintenance sections as version-specific. Two of
them were: the 1.7.0 disassembly split is a one-time migration, and the
upstream guide owns it.

The escaper fingerprint check is not version-specific. MCRIT does not pin
SMDA, so any rebuild can pull a newer one whose instruction escaping differs,
and MinHashes are derived from the escaped representation. The failure is
silent by construction: old signatures still look valid and still match each
other, while identical code submitted afterwards stops finding them. SMDA
4.4.5 altered about a fifth of the stored MinHashes on a real corpus.

That matters most in this pull request, which rebuilds both images from
scratch - a rebuild is exactly when the escaper can move, and the check was
the only thing telling an operator to look.

Restored under ## Upgrading, beside the MongoDB steps, where an operator is
already being told what an upgrade costs them. Shortened to fit the new
structure, and the command is verified against a running instance:

  smda_version: 4.5.0
  escaper_fingerprint: 110c9d152232a2e6

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GwSfzAD1ZwEoWY3eWvbYvX
@danielplohmann

Copy link
Copy Markdown
Collaborator

Pushed the escaper fix from my review directly to the branch (19ee792), since maintainerCanModify is on — revert or reword it freely, it is your PR.

It restores the fingerprint check as a ### under your new ## Upgrading, immediately after the MongoDB steps, rather than reinstating the old ## Maintenance section. That seemed the right home: it is the place an operator is already being told what an upgrade costs them, and the check is the same kind of thing as the FCV stepping — something to do because you are rebuilding.

Shortened to fit the new structure, and the command is verified against a running instance rather than copied forward:

smda_version: 4.5.0
escaper_fingerprint: 110c9d152232a2e6

README only, 21 lines added, nothing else touched. Your other two dropped sections stay dropped — the 1.7.0 disassembly split is genuinely a one-time migration and the upstream guide owns it.

🤖 Generated with Claude Code

https://claude.ai/code/session_01GwSfzAD1ZwEoWY3eWvbYvX

danielplohmann
danielplohmann previously approved these changes Sep 23, 2026

@danielplohmann danielplohmann left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved — CI green (build-test, lint, changelog gate, TUNING.md mirror). Reviewed in detail on #13; the increments here are small and sit on that base.

@r0ny123
r0ny123 dismissed danielplohmann’s stale review September 23, 2026 14:33

The merge-base changed after approval.

@danielplohmann
danielplohmann merged commit dffdb10 into familiary:main Sep 23, 2026
4 checks passed
danielplohmann pushed a commit that referenced this pull request Sep 23, 2026
MCRIT, MCRITweb and this repository now live under github.com/familiary. The
old paths redirect, so nothing is broken today - the images build and
clone_repositories.sh works. But a redirect is not a guarantee: it stops
working the moment a repository of the same name is created under the old
owner, and the two places that would break are build-time clones, where the
failure lands on whoever is deploying.

Rebuilt on top of the modernization rather than merged into it. The original
branch predated #13, so merging it produced a conflict in both Dockerfiles
between the old single-stage structure carrying familiary URLs and the new
multi-stage structure carrying the old ones - a conflict where taking either
side alone is wrong, and where taking "theirs" silently restores the old URLs
while still building fine.

Deliberately unchanged:

- danielplohmann/smda and danielplohmann/purepdb in RELEASING.md, which have
  not moved - checked, not assumed.
- docs/TUNING.md, a verbatim mirror of MCRIT's copy that the tuning-doc-mirror
  job diffs against the canonical version.
- CHANGELOG.md history, apart from the new entry this change itself needs.
- .github/workflows/test.yml, which still fetches TUNING.md from the old org
  via raw.githubusercontent.com. That redirect resolves, and changing it needs
  a token with the workflow scope.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GwSfzAD1ZwEoWY3eWvbYvX
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants