Skip to content

Run the shared SQLite database in WAL mode (#258) - #262

Closed
r0ny123 wants to merge 2 commits into
fix/jquery-ui-cve-2022-31160from
fix/sqlite-wal
Closed

r0ny123 wants to merge 2 commits into
fix/jquery-ui-cve-2022-31160from
fix/sqlite-wal

Conversation

@r0ny123

@r0ny123 r0ny123 commented Oct 3, 2026 •

Copy link
Copy Markdown
Collaborator

MCRITweb's SQLite file ran in the default rollback-journal mode while docker-mcrit's two gunicorn workers, with eight threads each, share it (#258). migrate() now switches it to WAL at startup, and both connects pass an explicit busy timeout.

What the journal mode changes. In delete mode a commit takes an exclusive lock that keeps every reader in both workers out until it's written, and a reader inside a transaction keeps a writer from committing. In WAL, readers carry on against the last commit while one writer appends. The busy timeout was already 5 s, by accident (Python's sqlite3.connect default), so what we had was waiting rather than "database is locked"; DATABASE_TIMEOUT = 5.0 now says so on purpose and is passed to both connects.

two connections, 0.05 s timeout delete (master) WAL (this)
a writer holds the commit lock, a reader reads "database is locked" reads the last commit
a reader is mid-transaction, a writer commits "database is locked" commits; the reader keeps its snapshot
a writer has an uncommitted UPDATE, a reader reads reads the old value reads the old value

The third row is there on purpose: an uncommitted write doesn't block readers in either mode, so it's kept as the baseline rather than claimed as an improvement.

Change.

  • enable_write_ahead_log(db) in db.py runs PRAGMA journal_mode=WAL and returns the mode SQLite actually kept. If WAL isn't possible (:memory: answers "memory") it prints that instead of failing; the app works as before.
  • migrate() calls it first thing, before its early return on a database without tables. flask init-db builds the app, and so migrates, before it creates the tables, so a fresh install is in WAL from the start; docker-mcrit's entry_web_prod.sh:15-16 runs exactly that on a first start.
  • The mode lives in the file, so the first start after upgrading converts an existing database, data intact.
  • README: a "SQLite database" section and an unreleased line, because this changes backups. mcritweb.sqlite-wal and -shm now sit next to the database, and copying mcritweb.sqlite alone while the app runs can miss recent writes. The README gives a backup command through SQLite's backup API in Python, since the docker-mcrit image has Python but no sqlite3 shell. Checked: with a write still in the -wal file, a plain copy of mcritweb.sqlite doesn't have it and the one-liner's backup does.
  • WAL needs the file on a local filesystem. docker-mcrit bind-mounts a host directory (docker-compose.yml:102), which is.

Verification.

  • New tests/testSqliteConcurrency.py, 10 tests: WAL after app start-up, after flask init-db on a path with no file, and on an upgraded delete-mode file; the non-WAL notice on :memory:; the busy timeout get_db() hands out; and the table above, each case under both modes. With db.py reverted, 7 fail; the 3 that pass describe the old behaviour on purpose.
  • 8 processes released together, each switching the same delete-mode file to WAL and then writing a row, 20 rounds: every round ended in WAL with all 8 rows and no "database is locked".
  • ruff check . is clean; python -m pytest gives 1268 passed, 7 skipped.
  • synchronous is left at its default.

Relation to open PRs. #254 adds an "API tokens issued before 1.5.0" README section at the same spot as the new "SQLite database" one; keep both. Nothing else conflicts.

Stacked on #261 (the jQuery UI update), so the diff here is this change only; it has no code dependency on it, only the README changelog line sits on top of #261's.

Closes #258.

Both gunicorn workers in docker-mcrit, with up to eight threads each, open
connections to the same instance/mcritweb.sqlite, but get_db() and migrate()
(mcritweb/db.py:620 and :664 before this change) connected with nothing but
the defaults. That left the file in rollback-journal mode: a reader holding a
transaction keeps a writer from committing, and a commit locks every reader
in both workers out until it is written. The 5 s busy timeout was there only
because it is Python's default.

migrate() now sets PRAGMA journal_mode=WAL as the first thing it does. The
mode is stored in the file, so once at startup covers every later connection.
It runs before migrate's early return on a database without tables, because
`flask init-db` builds the app (and so migrates) before it creates them; a
fresh install is in WAL from the start. If SQLite cannot use WAL it keeps the
old mode and says so, and that is printed rather than raised. Both connects
now pass timeout=DATABASE_TIMEOUT (5 s) explicitly. synchronous is left at
its default.

tests/testSqliteConcurrency.py checks the mode after start-up, after init-db
and on an upgraded rollback-journal file, the busy timeout get_db() hands
out, and the locking itself: with WAL a reader gets through while a writer
holds the exclusive lock, and a writer commits while a reader is mid
transaction; the same two cases in delete mode fail with "database is
locked". Seven of the new tests fail with db.py reverted. Full suite: 1268
passed, 7 skipped; ruff clean.

Operationally, WAL keeps two more files next to the database in instance/,
mcritweb.sqlite-wal and mcritweb.sqlite-shm. A backup has to copy all three
together, or better use the SQLite backup API (sqlite3 .backup), since
copying only the main file can miss commits still in the log. WAL needs the
file on a local filesystem shared by one host, which the docker-mcrit bind
mount is; it does not work over a network filesystem. The README gains a "SQLite database"
section that says this, with a backup command that works inside the
docker-mcrit image, which has Python but no sqlite3 shell.
@danielplohmann

Copy link
Copy Markdown
Collaborator

Merged into master via #318 (integration of batch E, merge commit fe6b313): this PR's head is an ancestor of master. It shows as open only because GitHub doesn't let a stacked PR's base be changed to master.

r0ny123 pushed a commit to r0ny123/mcritweb that referenced this pull request Oct 30, 2026
Bumps setup.py and adds the version history entry, folding in the five
"unreleased" lines (familiary#263, familiary#262, familiary#261, familiary#293, familiary#254).

48 PRs since 1.6.0: batches C1 (familiary#298), C2 (familiary#299), security E (familiary#318),
UI F (familiary#339) and fixes G (familiary#340). No dependency changes.

The entry leads with the three things that affect existing deployments:
instance/ backups under WAL mode, the bounded result cache, and the
optional retirement of pre-1.5.0 API tokens.

Test count measured on the batch G tree, which master 30cbc8a is
identical to: 2428 passed, 19 skipped against mcrit 1.16.0, 2447
collected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HERmVhHo4ibp7HSTXr3Ya8
@r0ny123
r0ny123 deleted the fix/sqlite-wal branch October 30, 2026 06:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The SQLite database runs in rollback-journal mode while two gunicorn workers share it

2 participants