feat: optimize blockchain indexer with single-query NFT and last_activity skip#247
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
feat: optimize blockchain indexer (single-query NFT & last_activity skip)
Description
This pull request optimizes the
indexer-blockchainservice to reduce TonAPI traffic (currently >1TB/day) and database load.last_activity-Based Skipping: We store the blockchain'slast_activitytimestamp on the user's wallet. If the timestamp hasn't changed since the last check, we skip the entire Jettons/NFTs fetch, savingsignificant network traffic and DB operations.
1750345690-ec82e8aa0d67-telegram_chat_rule_group.pycrashed on empty/clean databases due to parameter binding on an empty list.Checklist
Before submitting your pull request, please ensure the following:
Changes
get_all_nfts_per_userintasks.pyto fetch all user NFTs at once.tasks.pybefore passing NFT items tobulk_create_or_updateanddelete_missing, preventing unnecessary queries for non-whitelisted items.last_activity(BIGINT) toUserWalletand generated a database migration.fetch_wallet_detailsto checklast_activityreturned fromget_account_infoat the start of the task against the database, skipping Jetton and NFT fetches entirely if they are equal.WalletService.set_balanceto updatelast_activityatomically within the database update transaction.How Has This Been Tested?
backend/tests/unit/indexer_blockchain/test_tasks.pycovering:last_activityisNone.last_activityis unchanged.last_activityis updated on the blockchain.