Skip to content

perf: bulk materialization publication writes - #411

Merged
suguanYang merged 2 commits into
mainfrom
fix/wangbinqi/remove-duplicate-migration-test
Sep 12, 2026
Merged

perf: bulk materialization publication writes#411
suguanYang merged 2 commits into
mainfrom
fix/wangbinqi/remove-duplicate-migration-test

Conversation

@suguanYang

Copy link
Copy Markdown
Contributor

Summary

  • replace ORM row-by-row map-unit/token persistence with SQLAlchemy Core bulk batches
  • cap demo materialization database publication with a cross-instance Redis leased semaphore (default concurrency 2)
  • add Logfire stage timing for uploads, base rows, publication, snapshot, commit, and semaphore wait

Evidence

  • local production dump: 50k token rows, ORM flush 12.54s vs Core bulk 5.74s (~2.2x faster) with the existing five indexes
  • Core bulk wall time: concurrency 1 = 4.63s, 2 = 4.99s, 4 = 6.85s
  • no production schema, index, or data changes

Validation

  • demo contract: 10 passed
  • retrieval contract: 16 passed
  • worker contract: 330 passed
  • Ruff and Pyright passed

Comment thread packages/shared-python/shared/services/redis/__init__.py Fixed
Comment thread packages/shared-python/shared/services/redis/__init__.py Fixed
@suguanYang
suguanYang merged commit c37276c into main Sep 12, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants