Repository navigation
Follow-ups from the batch E review: read-only database, backup command, key file without hard links, upper-case job ids - #322
Merged
Conversation
Switching to WAL is a write, and on a database this process may not write SQLite raises for it rather than keeping the old mode, so create_app died in migrate() where it used to start and serve reads. A SQLITE_READONLY error (any extended code) is now printed, as the docstring said, and the start goes on. Anything else still raises.
Run from the wrong directory, sqlite3.connect('instance/mcritweb.sqlite')
created an empty database and backed that up. mode=ro fails instead. The
backup now goes into instance/ too, so under docker exec it is kept on
the bind mount rather than left in the container.
os.link fails with EPERM on vfat and exFAT and ENOTSUP on some SMB and FUSE mounts, so a first start with instance/ on one of those no longer came up. On any OSError but EEXIST the key now goes into a file created with O_EXCL instead. That file is empty until the key is in it, so it is flocked before the write, which makes a worker that finds it empty wait in _replace_if_still_empty and read the key rather than replace it; and if one replaced it before the lock was taken, the creator sees that path names another file and returns that file's key.
JOB_ID_PATTERN accepts upper-case hex and the backend finds the job by it, but reports are cached under the id the backend answers with, which is lower case. So /data/result/<upper-case id> missed the cache on every visit and fetched and wrote the report again. The routes that check the pattern now lower-case the id before using it, for the cache and for the backend alike, and the diagram route does the same for the job id in a diagram name, which otherwise rendered a new file per casing.
mcrit's LocalQueue mints ObjectIds since 1.14, like MongoQueue. The comments and test docstrings that still said uuid4 now say that was before 1.14; the shapes they accept are unchanged, since setup.py still allows mcrit 1.12.
Without hard links the key was written into a file created with O_EXCL, empty until filled. A worker that opened that file in the gap could replace it and find the creator's key still written into the file it had displaced; a third worker that had opened the displaced file before the replacement then read the loser's key and used it, while everyone else used the winner's. One round in about 1600 ended that way. The staged key, complete and private from mkstemp, is now renamed into place under a lock on the instance folder, looking at the key file again once the lock is held. No key file is opened for writing any more, and none is ever visible unfinished. The test that pinned the old create-then-lock order is replaced by two that pin the new one.
This was referenced Oct 26, 2026
Label every form field, keep what was typed on a refusal, and check inputs before they are sent
#333
Open
danielplohmann
added a commit
that referenced
this pull request
Oct 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #319, one commit per point.
Read-only database (#262).
PRAGMA journal_mode=WALis a write, so on a read-onlymcritweb.sqliteit raised, andcreate_appdied inmigrate(). An instance with a read-only database used to start and serve reads.SQLITE_READONLY_DIRECTORY, since the-walfile would have to be created beside it) counts too.os.accesswouldn't do: as root, a chmod doesn't stop SQLite writing.Backup command (#262). The README's command now opens the source with
file:instance/mcritweb.sqlite?mode=roanduri=True, and writes the backup toinstance/.docker execthe backup lands on the bind mount, not in the container layer.-walfile.Key file without hard links (#263). On a filesystem without hard links (vfat and exFAT give EPERM, some SMB and FUSE mounts ENOTSUP),
os.linkfailed and the app wouldn't start on first run.os.linkerror but "exists", the staged key, already complete and0600frommkstemp, is renamed into place under anflockon the instance folder.pathis read again once the lock is held, so a worker that lost the race takes the key that won.O_EXCLand then locked and wrote it. A worker could replace that file in the gap, while a third worker read the displaced one, leaving workers with different keys: 1 round in 40 in CI. f24948c is the fix.Upper-case job ids (#279). The gate takes either case, but a cached report is named with the backend's lower-case id. So
/data/result/<UPPER CASE id>missed the cache on every visit and wrote another copy.Stale comments.
data.py(JOB_ID_RE,DIAGRAM_FILENAME_RE,load_cached_result),utility.pyand four test files no longer say that a local queue issues uuid4 ids; LocalQueue mints ObjectIds since mcrit 1.14. The id formats accepted are unchanged, sincesetup.pystill allows mcrit 1.12.Tests.
testSqliteConcurrency.py: a read-only database still starts and serves reads; a read-only directory is reported and doesn't stop the start; a writable database that can't be switched still stops the start. The first two fail withdb.pyreverted. The third passes both ways, and keeps that behaviour from being swallowed.testSecretKeyRace.py:The two deterministic tests fail on the earlier version of the commit.
testResultPages.pyandtestMatchDiagrams.py: an upper-case id reads the cached report, is served the cached diagram, reaches the backend lower-cased, and is deleted lower-cased. All 12 cases fail with the views reverted.No changelog line: #262, #263 and #279 are unreleased, so none of this shipped.
Full suite with Chromium: 2181 passed before f24948c; after it, the secret-key, race and atomic-write tests pass (25).
ruff check .is clean.