Skip to content

Follow-ups from the batch E review: read-only database, backup command, key file without hard links, upper-case job ids - #322

Merged
danielplohmann merged 6 commits into
masterfrom
fix/319-batch-e-followups
Oct 29, 2026
Merged

danielplohmann merged 6 commits into
masterfrom
fix/319-batch-e-followups

Conversation

@r0ny123

@r0ny123 r0ny123 commented Oct 25, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #319, one commit per point.

Read-only database (#262). PRAGMA journal_mode=WAL is a write, so on a read-only mcritweb.sqlite it raised, and create_app died in migrate(). An instance with a read-only database used to start and serve reads.

  • SQLite's read-only error is now printed, as the docstring said, and the start goes on in the journal mode the file has.
  • It is matched by its primary result code, so a writable file in a read-only directory (SQLITE_READONLY_DIRECTORY, since the -wal file would have to be created beside it) counts too.
  • Any other error still stops the start.
  • os.access wouldn't do: as root, a chmod doesn't stop SQLite writing.

Backup command (#262). The README's command now opens the source with file:instance/mcritweb.sqlite?mode=ro and uri=True, and writes the backup to instance/.

  • Run from the wrong directory, it now fails with "unable to open database file" instead of backing up a new, empty database.
  • Under docker exec the backup lands on the bind mount, not in the container layer.
  • Checked by hand: the backup includes a commit still sitting in the -wal file.

Key file without hard links (#263). On a filesystem without hard links (vfat and exFAT give EPERM, some SMB and FUSE mounts ENOTSUP), os.link failed and the app wouldn't start on first run.

  • On any os.link error but "exists", the staged key, already complete and 0600 from mkstemp, is renamed into place under an flock on the instance folder.
  • path is read again once the lock is held, so a worker that lost the race takes the key that won.
  • An empty leftover file from a crashed start is replaced once, under the same lock.
  • Nothing opens the key file for writing, so no worker can ever read a key file that is not finished.
  • An earlier version of this commit created the file empty with O_EXCL and then locked and wrote it. A worker could replace that file in the gap, while a third worker read the displaced one, leaving workers with different keys: 1 round in 40 in CI. f24948c is the fix.

Upper-case job ids (#279). The gate takes either case, but a cached report is named with the backend's lower-case id. So /data/result/<UPPER CASE id> missed the cache on every visit and wrote another copy.

  • Every route lower-cases the id once it has passed the check, before it reaches the cache or the backend: result, link hunt, download, job page, job status, delete, promote, and the job a sample edit redirects back to.
  • Diagram filenames too: otherwise every casing of one id renders and stores another image, and the diagram cache has no size bound.

Stale comments. data.py (JOB_ID_RE, DIAGRAM_FILENAME_RE, load_cached_result), utility.py and four test files no longer say that a local queue issues uuid4 ids; LocalQueue mints ObjectIds since mcrit 1.14. The id formats accepted are unchanged, since setup.py still allows mcrit 1.12.

Tests.

  • testSqliteConcurrency.py: a read-only database still starts and serves reads; a read-only directory is reported and doesn't stop the start; a writable database that can't be switched still stops the start. The first two fail with db.py reverted. The third passes both ways, and keeps that behaviour from being swallowed.

  • testSecretKeyRace.py:

    • a filesystem without hard links still gets a key;
    • 8 workers starting at once without hard links share one key, 40 rounds, which passed 30 runs of 30;
    • the key is whole and the folder lock is held at the moment it appears;
    • nothing ever opens the key file for writing;
    • a key file replaced before it was locked isn't claimed.

    The two deterministic tests fail on the earlier version of the commit.

  • testResultPages.py and testMatchDiagrams.py: an upper-case id reads the cached report, is served the cached diagram, reaches the backend lower-cased, and is deleted lower-cased. All 12 cases fail with the views reverted.

No changelog line: #262, #263 and #279 are unreleased, so none of this shipped.

Full suite with Chromium: 2181 passed before f24948c; after it, the secret-key, race and atomic-write tests pass (25). ruff check . is clean.

Switching to WAL is a write, and on a database this process may not
write SQLite raises for it rather than keeping the old mode, so
create_app died in migrate() where it used to start and serve reads.
A SQLITE_READONLY error (any extended code) is now printed, as the
docstring said, and the start goes on. Anything else still raises.
Run from the wrong directory, sqlite3.connect('instance/mcritweb.sqlite')
created an empty database and backed that up. mode=ro fails instead. The
backup now goes into instance/ too, so under docker exec it is kept on
the bind mount rather than left in the container.
os.link fails with EPERM on vfat and exFAT and ENOTSUP on some SMB and
FUSE mounts, so a first start with instance/ on one of those no longer
came up. On any OSError but EEXIST the key now goes into a file created
with O_EXCL instead. That file is empty until the key is in it, so it is
flocked before the write, which makes a worker that finds it empty wait
in _replace_if_still_empty and read the key rather than replace it; and
if one replaced it before the lock was taken, the creator sees that path
names another file and returns that file's key.
JOB_ID_PATTERN accepts upper-case hex and the backend finds the job by
it, but reports are cached under the id the backend answers with, which
is lower case. So /data/result/<upper-case id> missed the cache on every
visit and fetched and wrote the report again. The routes that check the
pattern now lower-case the id before using it, for the cache and for
the backend alike, and the diagram route does the same for the job id
in a diagram name, which otherwise rendered a new file per casing.
mcrit's LocalQueue mints ObjectIds since 1.14, like MongoQueue. The
comments and test docstrings that still said uuid4 now say that was
before 1.14; the shapes they accept are unchanged, since setup.py still
allows mcrit 1.12.
Without hard links the key was written into a file created with O_EXCL,
empty until filled. A worker that opened that file in the gap could
replace it and find the creator's key still written into the file it
had displaced; a third worker that had opened the displaced file before
the replacement then read the loser's key and used it, while everyone
else used the winner's. One round in about 1600 ended that way.

The staged key, complete and private from mkstemp, is now renamed into
place under a lock on the instance folder, looking at the key file again
once the lock is held. No key file is opened for writing any more, and
none is ever visible unfinished.

The test that pinned the old create-then-lock order is replaced by two
that pin the new one.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Follow-ups from the batch E review (#318)

2 participants