Skip to content

feat: optimize blockchain indexer with single-query NFT and last_activity skip#247

Merged
danoctua merged 3 commits into
mainfrom
feat/per-user-blockchain-index
Jun 3, 2026
Merged

feat: optimize blockchain indexer with single-query NFT and last_activity skip#247
danoctua merged 3 commits into
mainfrom
feat/per-user-blockchain-index

Conversation

@danoctua
Copy link
Copy Markdown
Collaborator

@danoctua danoctua commented Jun 3, 2026

feat: optimize blockchain indexer (single-query NFT & last_activity skip)

Description

This pull request optimizes the indexer-blockchain service to reduce TonAPI traffic (currently >1TB/day) and database load.

  1. Elimination of N+1 NFT Queries: Instead of looping through all 80+ whitelisted NFT collections and making an API request for each, we now query all user-owned NFTs in a single, unfiltered API request.
  2. last_activity-Based Skipping: We store the blockchain's last_activity timestamp on the user's wallet. If the timestamp hasn't changed since the last check, we skip the entire Jettons/NFTs fetch, saving
    significant network traffic and DB operations.
  3. Database Migration Bugfix: Fixed an issue where the existing migration 1750345690-ec82e8aa0d67-telegram_chat_rule_group.py crashed on empty/clean databases due to parameter binding on an empty list.

Checklist

Before submitting your pull request, please ensure the following:

  • I have read the contributing guidelines.
  • I ran all tests and they passed successfully.
  • I added/updated documentation (if applicable).
  • I reviewed my own code and followed the project's code style. - [x] I included relevant details in the PR description or commit messages.

Changes

  • Single-Query NFT Fetch: Updated get_all_nfts_per_user in tasks.py to fetch all user NFTs at once.
  • In-Memory NFT Pre-filtering: Added in-memory filtering against whitelisted collection addresses in tasks.py before passing NFT items to bulk_create_or_update and delete_missing, preventing unnecessary queries for non-whitelisted items.
  • Wallet Model Update: Added last_activity (BIGINT) to UserWallet and generated a database migration.
  • Sync Skipping: Refactored fetch_wallet_details to check last_activity returned from get_account_info at the start of the task against the database, skipping Jetton and NFT fetches entirely if they are equal.
  • Atomic updates: Modified WalletService.set_balance to update last_activity atomically within the database update transaction.

How Has This Been Tested?

  • Added unit tests in backend/tests/unit/indexer_blockchain/test_tasks.py covering:
    • Initial sync behavior when last_activity is None.
    • Skip behavior when last_activity is unchanged.
    • Sync trigger behavior when last_activity is updated on the blockchain.
  • Ran the full test suite on a clean, dockerized test database. All 99 unit tests passed successfully.

@danoctua danoctua self-assigned this Jun 3, 2026
@danoctua danoctua added the enhancement New feature or request label Jun 3, 2026
@danoctua danoctua merged commit 4c1a5bb into main Jun 3, 2026
1 check passed
@danoctua danoctua deleted the feat/per-user-blockchain-index branch June 3, 2026 19:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants