fix(database): resolve SQLite concurrency lock errors during parallel scans (#32)

[Myparadox-creator]

Summary of Changes

Closes

#32

Problem

FastAPI handles requests asynchronously and spawns background tasks for repository ingestion and rescans. When multiple users submit repository scans concurrently, multiple tasks execute write operations against the SQLite database simultaneously. Since SQLite supports only a single concurrent writer and uncommitted transactions were held during long CPU metric extraction loops, this triggered \sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) database is locked\ errors, failing concurrent ingestion jobs.

Solution

1. Config & Concurrency Throttling:

   • Added \MAX_CONCURRENT_INGESTIONS\ configuration parameter (default: \2) in \�ackend/config.py.
   • Introduced \get_ingestion_semaphore()\ in \�ackend/features/repo_ingestion/router.py\ to queue and throttle concurrent background ingestion and rescan jobs without overwhelming SQLite writer locks.

2. Connection Parameters & PRAGMAs:

   • Configured SQLite connection pool with \connect_args={'check_same_thread': False, 'timeout': 60}.
   • Applied \PRAGMA journal_mode = WAL;, \PRAGMA busy_timeout = 30000;, and \PRAGMA synchronous = NORMAL;\ on all SQLite connections.

3. Exponential Backoff Retry & Transaction Hardening:

   • Enhanced \commit_with_retry\ in \�ackend/database.py\ to handle transient \sqlite3.OperationalError\ locks with exponential backoff up to 5 retries.
   • Refactored _update_job\ to reuse open session objects when provided rather than opening conflicting simultaneous writer connections.
   • Replaced all raw \�wait db.commit()\ calls with \�wait commit_with_retry(db)\ across repository ingestion, rescans, cancellation, and deletion endpoints.

4. Testing:

   • Added multi-worker concurrent write stress tests and semaphore verification in \�ackend/tests/test_sqlite_concurrency.py.
   • 100% test pass rate (290 backend pytest tests passing; 116 frontend vitest tests passing; frontend build clean).

[vercel]

@Myparadox-creator is attempting to deploy a commit to the Eshaan's projects Team on Vercel.

A member of the Team first needs to authorize it.