Skip to content

Create the generated secret key atomically across workers (#259) - #263

Closed
r0ny123 wants to merge 2 commits into
fix/sqlite-walfrom
fix/secret-key-race
Closed

r0ny123 wants to merge 2 commits into
fix/sqlite-walfrom
fix/secret-key-race

Conversation

@r0ny123

@r0ny123 r0ny123 commented Oct 3, 2026 •

Copy link
Copy Markdown
Collaborator

load_or_create_secret_key wrote a missing key with O_CREAT | O_TRUNC (secret_key.py:42), and create_app runs it in every gunicorn worker, so two workers starting without the file could each keep a key of their own (#259). A login or CSRF token from one worker was then refused by the other until a restart. The key is now published atomically, and every worker returns the one key that ends up on disk.

When it happens. Not on a fresh docker-mcrit install: entry_web_prod.sh:15-16 runs flask init-db before gunicorn, and that creates the key in one process. It happens when mcritweb.sqlite exists but secret_key doesn't, which is an upgrade from before 1.4.4 (when the file came in), a deleted key file, or any deployment that doesn't run init-db first. An explicit SECRET_KEY in instance/config.py was never affected.

Change.

  • A new key is written in full to a private temp file beside the real one (mkstemp, so O_EXCL and 0o600 from creation), fsynced, and published with os.link, which fails instead of overwriting when another process got there first. The loser reads and returns the winner's key. The temp file is removed either way. Nothing can read a half-written key file any more.
  • An empty key file left by older code or a crash is still replaced rather than used, as before, but only once: workers that find it queue on a flock, the first one through swaps its key in with os.replace after checking the path still names that empty file, and the rest read what replaced it. fcntl is imported only on that path, so the module still imports where it doesn't exist.
  • An existing key is read as before; that path is unchanged.

Verification.

8 processes released together, 40 fresh instance folders per run old code this
no key file split keys in 19–33 of 40 rounds, 10 of 10 runs one key, 25 of 25 runs
an empty key file split keys in 9–30 of 40 rounds, 10 of 10 runs one key, 25 of 25 runs
  • New tests/testSecretKeyRace.py, 7 tests: 0o600 on creation, a second call returning the same key, both races above (also checking the key equals the file and no temp file is left), and two deterministic losing paths, one where another worker links first and one where it replaces the empty file first.
  • Without fcntl the module imports, creates a 0o600 key and returns it again on the next call.
  • ruff check . is clean; python -m pytest gives 1275 passed, 7 skipped on top of the two PRs below this one.

Not covered. The directory isn't fsynced after the link, so a power cut in the first seconds after a first start could lose the file; the next start would make a new key and log everyone out once. It can't produce a split or empty key.

Relation to open PRs. No conflicts. Stacked on #262, with no code dependency; only the README changelog line sits on top of its line.

Closes #259.

load_or_create_secret_key read instance/secret_key and, finding it missing
or empty, wrote a fresh key with O_CREAT|O_TRUNC (secret_key.py:42). gunicorn
runs two workers without --preload, so create_app runs in both. On a first
start each could miss the file, generate its own key and write it: the file
kept one key while the other worker went on signing with its own, so sessions
and CSRF tokens issued by one worker were rejected by the other until a
restart. A worker could also open the file after another had created it but
before it wrote the key, read it as empty and truncate it with a key of its
own.

The new key is now written to a 0o600 temp file beside the real one (mkstemp,
so O_EXCL), fsynced and published with os.link, which fails instead of
overwriting when another process got there first; the loser reads and returns
the winner's key, and the temp file is removed either way. Nothing ever sees
a partly written key file any more.

This code no longer leaves an empty file, but older code or a crash could
have. It is still replaced rather than used, but only once: workers that find
it take a flock on it, and only the first through, after checking the path
still names that same empty file, swaps its key in with os.replace. The rest
read the key that replaced it. fcntl is imported only on that path,
so the module still imports where it does not exist.

testSecretKeyRace.py runs 8 spawned processes released by a barrier against
40 fresh instance folders, both with no file and with an empty one, and
checks every worker got the key that is on disk and no temp file is left.
Against the old code it failed on every one of 10 runs, with 9 to 33 of the
40 rounds ending in split keys; with this change it passed 25 runs out of 25.
Two monkeypatched tests pin the losing paths: another worker publishing just
before os.link, and replacing the empty file just before the flock.
@danielplohmann

Copy link
Copy Markdown
Collaborator

Merged into master via #318 (integration of batch E, merge commit fe6b313): this PR's head is an ancestor of master. It shows as open only because GitHub doesn't let a stacked PR's base be changed to master.

r0ny123 pushed a commit to r0ny123/mcritweb that referenced this pull request Oct 30, 2026
Bumps setup.py and adds the version history entry, folding in the five
"unreleased" lines (familiary#263, familiary#262, familiary#261, familiary#293, familiary#254).

48 PRs since 1.6.0: batches C1 (familiary#298), C2 (familiary#299), security E (familiary#318),
UI F (familiary#339) and fixes G (familiary#340). No dependency changes.

The entry leads with the three things that affect existing deployments:
instance/ backups under WAL mode, the bounded result cache, and the
optional retirement of pre-1.5.0 API tokens.

Test count measured on the batch G tree, which master 30cbc8a is
identical to: 2428 passed, 19 skipped against mcrit 1.16.0, 2447
collected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HERmVhHo4ibp7HSTXr3Ya8
@r0ny123
r0ny123 deleted the fix/secret-key-race branch October 30, 2026 06:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Two workers can each keep a different generated SECRET_KEY

2 participants