Skip to content

Give SQLite writes one pooled connection, and change the version only when the owner asks - #599

Merged
James-Chapman merged 3 commits into
claude/livekit-token-service-bde17efrom
fix/sqlite-write-starvation
Nov 18, 2026
Merged

James-Chapman merged 3 commits into
claude/livekit-token-service-bde17efrom
fix/sqlite-write-starvation

Conversation

@James-Chapman

@James-Chapman James-Chapman commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Stacked on #598 (merge order: #594 → #595 → #597 → #598 → this). Under sustained write load a SQLite server could refuse writes with SQLITE_BUSY: the write pool opened up to database.pool_size SQLite connections, and every connection after the first waited for the first one's lock in SQLite's polling busy handler, lost to its back-to-back commits and gave up after 5 s. SQLite runs one write at a time, so its write pool now holds exactly one connection and writers wait in the pool instead (ADR-0834).

At the owner's request this branch also changes the versioning rule: the version now changes only when the owner asks, so this branch stays at #598's 1.1.3 and adds its CHANGELOG bullets to that entry (ADR-0835).

What changed

  • SqliteWritePool holds a ConnectionPool of limit 1; set_write_connection_limit (database.pool_size) now changes only the PostgreSQL pool, and the SQLite overload of detail::set_write_pool_limit is gone.
  • ADR-0834 records the decision and the rejected alternatives (longer busy timeout or retrying the open, several connections behind a mutex, WAL); ADR-0154 links to it.
  • docs/database-persistence.md, the database.pool_size row of docs/user-manual.md, src/database/AGENTS.md rule 7 and the review skill's database invariants say SQLite has one write connection.
  • ADR-0835: the version changes only when the owner asks; a branch keeps its base's version, and every change adds its bullets to the newest CHANGELOG entry. It supersedes ADR-0464 and the per-branch increment of ADR-0647, both marked in their files and in docs/adr/index.md.
  • docs/versioning.md (scheme, history, bump policy, release process), the root AGENTS.md, docs/coding-rules.md, docs/AGENTS.md and the review skill (SKILL.md, references/obligations.md) state the new rule.
  • No version change: every location stays at 1.1.3. The 1.1.3 CHANGELOG entry gains a Features bullet (ADR-0835), a Bug fixes bullet (ADR-0834) and a release-notes sentence.

Why it changed

The DB-7 scenario lost 3 of 64 writes in a full run on a loaded VM (2026-11-17, during #598's work): three connection.rejected sqlite_code=5 / transaction.failed stage=open lines, all within 14 ms of each other and 5.3 s after the store opened. So they waited out the whole busy timeout together; nothing skipped the busy handler.

In rollback-journal mode a committing connection holds PENDING or EXCLUSIVE for the whole commit, including both journal syncs and the database sync (syncJournal takes EXCLUSIVE before its first sync), and PENDING refuses every new SHARED lock. One pooled connection committing back to back holds the lock almost all the time and takes it again within microseconds of releasing it. Another connection of the same process, polling every 100 ms by then, rarely lands in the gap. The opening connections needed SHARED to read the schema while pinning their pragmas, and were refused. A write behind one slow write for more than 5 s is refused the same way.

The busy timeout still governs this process's reads against its writer, and locks other processes take; reads are unchanged here.

The versioning rule changed because the owner asked that the version move only on request, not once per branch.

CI tests

Modified:

  • tests/integration/test_security_audit_write_pool_flow.cpp: "DB-7 concurrent writes to SQLite share one connection, whatever database.pool_size allows" (renamed from "... reuse at most database.pool_size connections"); 64 concurrent writers all commit and write_connections_opened == 1.

New:

  • tests/integration/test_security_audit_write_pool_flow.cpp: "A SQLite write waits for another write of this process for longer than SQLite's busy timeout" ([db7]). It holds SQLite's write lock on a leased slot of the store's own pool for 7 s while a second write runs. Run against the unfixed code on the VM, it failed as intended: transaction.failed stage=begin sqlite_code=5, second_committed false, 1 room stored instead of 2.

Verification: python build.py vm on the fix before it was stacked on #598 (based on main, with the SQLite change and both scenarios): Ok 85, Fail 0, no timeouts; integration-tests took 989 s. The stacked head (9b645b3a, with the versioning docs) was not run locally at the owner's request; CI covers it.

🤖 Generated with Claude Code

James-Chapman and others added 2 commits November 18, 2026 00:55
…Lite's busy handler for another (1.1.4)

The DB-7 scenario lost three of 64 writes under load: three pooled SQLite
connections being opened polled SQLite's busy handler for the first one's
lock, lost to its back-to-back commits (every journal and database sync
runs under EXCLUSIVE) and were refused after 5 s with SQLITE_BUSY.

SQLite runs one write at a time, so the SQLite write pool now holds one
connection whatever database.pool_size says, and writers wait in the pool
instead (ADR-0834). database.pool_size applies to PostgreSQL only.

A new scenario holds SQLite's write lock on a leased pool slot for 7 s
while a second write runs; on the old code that write was refused at
BEGIN IMMEDIATE after the busy timeout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nvariants

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Explain Change →

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: b211a604-0227-4ed2-862a-8c857a54d134

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@James-Chapman
James-Chapman added this pull request to stack #596 November 18, 2026 01:00
…1.1.3

The owner's rule from 2026-11-18: no branch increments the version on its
own; it changes only when the owner asks, and every change adds its
CHANGELOG bullets to the newest version's entry (ADR-0835, superseding
ADR-0464 and the per-branch increment of ADR-0647).

docs/versioning.md, the root AGENTS.md, docs/coding-rules.md,
docs/AGENTS.md and the review skill now say so. Under the new rule this
branch drops its 1.1.4 bump: every version location is back to its base's
1.1.3, and the SQLite fix's bullet and release-notes sentence move into
the 1.1.3 entry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@James-Chapman
James-Chapman removed this pull request from stack #596 November 18, 2026 01:03
@James-Chapman James-Chapman changed the title Give SQLite writes one pooled connection so a write is never refused by SQLite's busy timeout (1.1.4) Give SQLite writes one pooled connection, and change the version only when the owner asks Nov 18, 2026
@James-Chapman
James-Chapman merged commit a20f5da into claude/livekit-token-service-bde17e Nov 18, 2026
3 of 7 checks passed
@James-Chapman
James-Chapman deleted the fix/sqlite-write-starvation branch November 18, 2026 01:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant