Skip to content

feat(db): make SQLite durability configurable, defaulting to today's behaviour - #173

Draft
Pfannkuchensack wants to merge 2 commits into
mainfrom
feat/db-synchronous-opt-in
Draft

feat(db): make SQLite durability configurable, defaulting to today's behaviour#173
Pfannkuchensack wants to merge 2 commits into
mainfrom
feat/db-synchronous-opt-in

Conversation

@Pfannkuchensack

@Pfannkuchensack Pfannkuchensack commented Aug 28, 2026

Copy link
Copy Markdown
Member

Summary

Feature, opt-in, default unchanged. SqliteDatabase.__init__ sets journal_mode=WAL, foreign_keys and busy_timeout, but never sets synchronous — so it stays at SQLite's own default of FULL, which fsyncs on every commit. This adds a db_synchronous config setting so an operator can choose normal instead.

Measured on a copy of a real library (939 images), 300 single-row inserts each committed on its own:

Setting Median commit p95 commit
full 0.427 ms 0.811 ms
normal 0.035 ms 0.121 ms

Roughly 12× shorter commits at the median on this machine (NVMe SSD; the effect is storage-dependent). That matters more than the raw numbers suggest: SqliteDatabase holds one connection and one RLock through which all database work is serialised, so a shorter write is also a shorter window during which every read is blocked.

The default stays full. normal cannot corrupt the database — WAL guarantees a consistent database either way — but a power loss or OS crash can lose the most recent transactions: a just-written image record or a queue status, not the image file itself, which the orphan scan can recover. That is a durability decision for whoever runs the server, and making it for them on upgrade would be wrong.

off and extra are deliberately not offered. off can corrupt the database on an OS crash, and extra costs more than full for a guarantee this application does not need. The Literal is closed, which is also why the PRAGMA interpolates its value directly — PRAGMA takes no bind parameters, and no user-supplied string can reach that statement.

The setting is documented in configuration/invokeai-yaml.mdx alongside the other operational settings, with the trade-off stated rather than buried.

Related Issues / Discussions

From the local .ideas/sql-gallery-indizes-und-db-tuning.md §3. The numbers here were re-measured on this machine rather than taken from that document, which recorded ~6.4× on different hardware.

QA Instructions

What I ran:

  • tests/app/services/shared/sqlite/test_sqlite_synchronous.py — 15 tests: the config default, normal accepted, the other real SQLite values (off, extra) and malformed input rejected, the PRAGMA reaching the connection for each setting, in-memory databases behaving the same, WAL still on (without it the trade would be a different one), and the wiring.
  • tests/test_config.py::test_db_synchronous_defaults_to_full_and_loads_from_yaml — default plus a YAML round trip, matching the pattern of the neighbouring settings.
  • Mutation-checked, each caught: flipping the default to normal (2 failures), deleting the PRAGMA statement (2), widening the Literal to include off (1), dropping the argument in init_db (2).
  • tests/app/services/shared + tests/test_config.py — 180 passed. pytest --collect-only clean. ruff check / format --check clean.
  • openapi.json and schema.ts regenerated the way CI does. The diff is the new field plus its line in the InvokeAIAppConfig docstring — additive, with a default, so nothing existing changes shape.

To verify by hand: set db_synchronous: normal in invokeai.yaml, start the server, and check the connection agrees:

PRAGMA synchronous;  -- 2 = full (default), 1 = normal

One finding worth naming, because it is the kind that does not announce itself: at one point the config field and the PRAGMA were both correct while nothing connected them. The app booted, every test passed, and the setting silently did nothing. TestTheSettingReachesTheDatabase exists specifically for that — it stubs SqliteDatabase and asserts init_db passes the configured value through.

Merge Plan

Nothing special. No DB schema, no migration, no redux slice, no dependency change. The openapi.json change is additive.

Independent of #171 and #172, which are in flight in parallel — disjoint files, any merge order.

Checklist

  • The PR has a short but descriptive title, suitable for a changelog
  • Tests added / updated (if applicable) — 16 tests; four mutations verified as caught
  • ❗Changes to a redux slice have a corresponding migration — n/a, no redux changes
  • Documentation added / updated (if applicable) — new "Database Durability" section in invokeai-yaml.mdx
  • Updated What's New copy (if doing a release after this PR) — optional; the setting is opt-in and changes nothing by default

🤖 Generated with Claude Code

…behaviour

SqliteDatabase sets journal_mode=WAL, foreign_keys and busy_timeout, but
never sets `synchronous` -- so it stays at SQLite's default of FULL, which
fsyncs on every commit.

Measured on a copy of a real library, 300 single-row inserts each committed
on its own: median 0.427ms at FULL against 0.035ms at NORMAL, p95 0.811ms
against 0.121ms. Roughly 12x shorter commits on this machine. That matters
more than the raw numbers suggest, because all database work is serialised
through one lock: a shorter write is also a shorter time during which every
read is blocked.

The default stays FULL. NORMAL cannot corrupt the database -- WAL
guarantees consistency either way -- but a power loss or OS crash can lose
the most recent transactions: a just-written image record or queue status,
not the image file itself, which the orphan scan can recover. That is a
durability decision for whoever runs the server, not one to make for them
on upgrade.

`off` and `extra` are deliberately not offered: `off` can corrupt the
database on an OS crash, and `extra` costs more than `full` for a guarantee
this application does not need. The Literal is closed, which is also why
the PRAGMA can interpolate the value -- PRAGMA takes no bind parameters.

The wiring is covered by its own test. A config field and a working PRAGMA
can both be correct while nothing connects them, and that failure is
silent: the app boots, every other test passes, and the setting does
nothing. It happened once while writing this.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant