catch up#1
Open
Corey-Code wants to merge 998 commits into
Open
Conversation
Background ticker that calls limiter.sweep() every cleanupInterval, so TTL-expired idle entries are evicted even when no new connections arrive to drive cleanup. The goroutine is started once in NewWebsocketServer
Add <NETWORK>_WS_TRUSTED_PROXIES env var to extend X-Real-Ip trust beyond loopback/RFC1918 for non-Cloudflare deployments. Fails startup on /0 or otherwise overly broad prefixes (< /8 IPv4, < /16 IPv6) so misconfig can't silently turn the header into a spoofing primitive.
Wrap the four DB-touching go ... spawn sites with a sync.WaitGroup gate and add WebsocketServer.Shutdown(ctx) that flips a shuttingDown flag, closes all registered channels, and waits for in-flight goroutines to drain. PublicServer.Shutdown now drives it after http.Server.Shutdown, so a long getAccountInfo can no longer race rocksdb_close in cgo and SIGSEGV on graceful restart.
Agent-Logs-Url: https://github.com/trezor/blockbook/sessions/0d3d9125-200e-45e5-ae57-a54919829a60 Co-authored-by: pragmaxim <8983344+pragmaxim@users.noreply.github.com>
Agent-Logs-Url: https://github.com/trezor/blockbook/sessions/0d3d9125-200e-45e5-ae57-a54919829a60 Co-authored-by: pragmaxim <8983344+pragmaxim@users.noreply.github.com>
Less invasive and strict monitoring of consensus node that does not flood logs with errors
Map Avalanche "cannot query unfinalized data" RPC errors to ErrBlockNotFound instead of swallowing them. This prevents failed trace calls from being treated as successful empty results, which caused trace length mismatches during sync.
* feat: add AlternativeFeeProviderRequests metric to monitor (un)successfull requests * chore: check if method exists in handlers * chore: add cap for /api/block-filters * fix: references of the response --------- Co-authored-by: pragmaxim <pragmaxim@gmail.com>
When a batched balanceOf returned per-element errors or unparseable results, the worker fell back to a single eth_call per affected contract — a wallet with N tokens could turn into N+1 RPCs on a single bad batch, and parse failures triggered retries that were never going to succeed (malformed returns indicate non-conforming contracts, not transient faults).
Alternative/private relays (MEV protection) often expire pending transactions an they keep hanging in blockbook mempool for hours
… fees than native fees
* chore: remove legacy socket.io interface * chore: remove dead code using OnNewTxAddr used only in obsolete socket.io interface * remove empty socketio.go
… calls costs (#1502) * fix(erc20): stop ERC20 batch fallback from amplifying RPC calls When a batched balanceOf returned per-element errors or unparseable results, the worker fell back to a single eth_call per affected contract — a wallet with N tokens could turn into N+1 RPCs on a single bad batch, and parse failures triggered retries that were never going to succeed (malformed returns indicate non-conforming contracts, not transient faults). * chore(erc4626): accountInfo only returns erc4626 support in tokens accountInfo for an address holding any number of ERC4626 vaults now costs 0 RPC for vault flagging, regardless of vault count. The flag is populated for free during block indexing. * feat(eth): add Multicall3 aggregate3 primitive Add ABI encoding/decoding and an EthereumRPC aggregate3 method for batched eth_calls. This prepares ERC4626 contractInfo enrichment to use one RPC round-trip. * chore(erc4626): store asset contract on ContractInfo Add Erc4626AssetContract to ContractInfo serialization so ERC4626 enrichment can cache the vault asset address without a DB version bump. * feat(erc4626): use multicalls for contractInfo enrichment Replace sequential vault eth_calls with Multicall3 batches, cache the asset address, and keep cold/warm paths covered by focused tests. * feat(erc4626): cache per-block enrichment calls Add a contract/block-keyed LRU with singleflight so repeated or concurrent contractInfo requests share ERC4626 enrichment work. * fix(erc4626): require totalAssets before persisting vaults Persist ERC4626 metadata only after asset() returns a non-zero address and totalAssets() decodes successfully; cover failed totalAssets paths. * feat(erc4626): backfill vault flags from accountInfo Probe ERC4626 support lazily from accountInfo and persist negative results so already-synced vaults can be recognized without repeated RPC work. * fix(erc4626): persist only confirmed vault probes Stop treating negative ERC-4626 probes as authoritative. Re-probe non-confirmed fungible contracts on demand and persist only positive vault detections. * fix(erc4626): pin vault data to response height Use the best block selected by getContractInfo for both the reported blockHeight and the ERC-4626 multicall/cache key, so protocol data is built from the same chain state. * chore(4626): negative vault caching * chore(4626): multicall chunking * chore(4626): cleanup * chore(4626): simplification and code deduplication * chore(4626): remove problematic optimization that saves one multicall per new vault discovery globally * fixing Base test data * chore(erc4626): new cfContract protocol format contractInfo := baseContractInfo [protocolExtensions] baseContractInfo := name: string symbol: string standard: string decimals: vuint createdInBlock: vuint destructedInBlock: vuint protocolExtensions := extensionHeader: vuint extensionCount: vuint repeated extensionCount times: protocolId: vuint payloadLength: vuint payload: bytes * chore(erc4626): more unit tests for the protocol * chore(erc4626): rocksdb contracts column family documentation * chore(erc4626): bound a concurrent unit test of singlefliht optimization * chore(erc4626): tighten singleFlight cache * chore(erc4626): multicall3 probe * chore(erc4626): singleflight cache fix * chore(erc4626): bestBlock warning * chore(erc4626): cleanup * chore(erc4626): avg block time config * chore(erc4626): reorg safety * chore(erc4626): simplification and cleanup * fix(erc4626): populate-after-write race * chore(erc4626): polishing API * chore(erc4626): cleanup * chore(erc4626): use averageBlockTimeMs for the negative-cache TTL * chore(erc4626): rename towards new convention
…D temporal skew on stablecoins
…n rates when "platformVsCurrency" was changed
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Agent-Logs-Url: https://github.com/trezor/blockbook/sessions/114623cb-c06f-422f-a952-bc35ad93c8a2 Co-authored-by: pragmaxim <8983344+pragmaxim@users.noreply.github.com>
ParseInputData panicked with "index out of range [N] with length N" when a transaction's input data was not 32-byte aligned. Standard ABI-encoded calldata is always word-aligned, but a malformed/non-standard tx is not: while marking the words of a dynamic field (bytes/string/slice), tryParseParams could step one slot past `processed`, which is sized len(data)/64. The panic was caught by ParseInputData's deferred recover, so it did not crash the process, but it returned nil and logged a full stack trace (debug.PrintStack) for every offending transaction — heavy log spam when such an address is rendered or crawled. Bound the dynamic-field marking loop: an index past `processed` means the declared layout runs past the data, so reject the signature (return false) instead of writing out of range. Add a regression test built from a deliberately misaligned dynamic field. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A balanceOf eth_call that succeeds but returns empty/non-32-byte data (typical of dead/self-destructed or non-ERC20-conforming tokens that linger in holders' contract lists) was logged at Warning level on every holder request. Since these warnings were added in e8558f1 they have dominated the logs - tens of thousands of lines per minute for a single widely-airdropped dead token - because glog.Warningf prints regardless of the -v verbosity. Demote all five log sites of this benign event to glog.V(2).Infof and add a sentinel eth.ErrInvalidErc20Balance so the single-contract path in api/worker.go can tell it apart from genuine RPC errors, which stay at Warning. The observeEthCallError(..., "invalid") metric is unchanged, so the rate is still observable via Prometheus. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The public HTTP surface (explorer UI + REST API) is intended for individual human use -- Trezor Suite uses the WebSocket interface, not REST/UI -- so the previous defaults (180 req/min, burst 40) were far more generous than a person browsing the explorer needs. That headroom let a distributed crawler sustain heavy aggregate traffic while every per-IP key stayed comfortably under the limit, so the limiter never rejected anything. Lower the per-client defaults: REST_UI_RATE_LIMIT 180 -> 20 and REST_UI_BURST 40 -> 20. The explorer renders each page as a single request, so 20 requests/min comfortably covers real browsing while the tighter sustained rate forces a crawler onto far more source IPs for the same aggregate rate; a burst of 20 still absorbs a short flurry of page loads. Both remain overridable via the <network>_REST_UI_* env vars. Updates the matching defaults in docs/env.md and the config test that locks the values. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
DefaultBalanceHistoryMaxTxsREST bounds how many transactions a single
REST balance-history request (address or xpub) may aggregate. Each
aggregated transaction costs one DB read, so an unbounded range over a
heavy account is a cheap-to-send, expensive-to-serve request on the open,
unauthenticated REST surface.
The cap is enforced against a bounded index scan (txids are collected up
to cap+1 via early StopIteration) and the request is rejected before any
of the per-transaction reads run, so lowering it directly shrinks the
worst-case work an anonymous request can trigger. 100000 stays well above
any realistic single-account history a human would inspect via REST while
tightening the ceiling from 250000.
WS keeps its generous 1000000 default (Trezor Suite requests full account
history over WS for its balance graph). Both remain overridable via
<network>_{WS,REST}_BALANCE_HISTORY_MAX_TXS (or the shared
<network>_BALANCE_HISTORY_MAX_TXS).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
EthereumTypeGetEip1559Fees called setLatestBlockGas on every request,
which issued an extra synchronous eth_getBlockByNumber("latest") RPC to
populate Eip1559Fees.BlockGasUsed/BlockGasLimit. This had three problems:
- Extra overhead: a third blocking RPC was added to every websocket
estimateFee call (on top of eth_maxPriorityFeePerGas + eth_feeHistory),
and it also fired on the alternative-fee-provider fast path that was
deliberately RPC-free (served from an in-memory cache). On fast L2s with
many subscribers polling fees this scales linearly with load.
- Data race: the alternative provider returns the same cached *Eip1559Fees
pointer to every caller; setLatestBlockGas then mutated BlockGasUsed/
BlockGasLimit on that shared struct with no lock, racing concurrent
estimateFee goroutines (and the field reads in server/websocket.go).
- Incoherent data: baseFeePerGas came from eth_feeHistory (newest=pending)
or an off-chain oracle, while gasUsed/gasLimit came from a separate
"latest" block, so the three values described different blocks and could
not be combined into a correct EIP-1559 next-base-fee projection.
Remove getLatestBlockGas/setLatestBlockGas and the now-unpopulated
BlockGasUsed/BlockGasLimit fields from the pull-side Eip1559Fees (bchain,
api, blockbook-api.ts) and the websocket mapping. The push path
(subscribeNewBlock evm_data) is unaffected: it still carries the block's
baseFeePerGas/gasUsed/gasLimit from a single header, with no extra RPC.
A follow-up commit exposes the next-block base fee from the eth_feeHistory
response already fetched, so the pull path keeps a (consistent, free)
projection input.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The new-block notification payload used a snake_case json tag (evm_data), the only such tag among the websocket/JSON wire types, which otherwise use lowerCamelCase (height, hash, baseFeePerGas, feePerUnit, ...). Rename it to evmData for consistency before the field ships, since changing a wire-level key after release would be a breaking change. Updates the Go json tag, the JSON shape test, blockbook-api.ts and a code comment. The field is documented in openapi.yaml in the next commit. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The subscribeNewBlock notification grew an evmData field (WsNewBlock / EthereumGasData) that was undocumented in openapi.yaml. Because these types are not in the parity allow-list they did not break CI, but the new websocket payload was unspecified. Add EthereumGasData and WsNewBlock schemas (evmData modeled as nullable via the 3.1 oneOf/"null" pattern, matching the Go *EthereumGasData json tag without omitempty), and add both to the parity check so the generated blockbook-api.ts and openapi.yaml stay in sync. Parity typecheck passes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The push path (subscribeNewBlock -> evmData) is the consistent, RPC-free delivery mechanism for the EIP-1559 projection inputs, but only its JSON wire shape was tested, not the block -> payload mapping. Extract that mapping out of onNewBlockAsync into a pure newBlockNotification helper (behavior unchanged) and unit-test it: non-EVM blocks and EVM pre-London blocks leave evmData nil, while EVM post-London blocks carry baseFeePerGas/gasUsed/gasLimit taken from the connected block header. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
WsNewBlock's height, hash and evmData have no omitempty in Go, so they are always present on the wire (evmData serializes as null for non-EVM/pre-London blocks). Mark them required in the schema so the spec reflects the actual always-present (evmData nullable) shape. Parity typecheck still passes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…e fee Records the conclusion of the fee-history investigation inline: eth_feeHistory's extra projected element has backend-dependent semantics (dropped by no-pending nodes like Erigon so baseFeePerGas is already the next-block fee; kept but index-shifted to N+1 or a noisy N+2 by some L2s), so a single "next block" field cannot be defined consistently. Clients needing an exact next-block base fee use the subscribeNewBlock evmData push instead. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The on-chain estimate (used when no alternative fee provider is configured — e.g. ethereum non-archive and the ethereum testnets) set each tier's maxFeePerGas to the priority fee alone, omitting the base fee entirely. That value is below the block's base fee and therefore not mineable. Trezor Suite consumes maxFeePerGas directly (connect EthereumFeeLevels: it only clamps to a floor, it does not add the base fee), so the on-chain path produced unusable EIP-1559 fees. Compute each tier correctly from the eth_feeHistory response: - maxPriorityFeePerGas = the tier's reward percentile (20/70/90/99), i.e. the actual tip distribution, instead of the node's single suggestion doubled. - maxFeePerGas = eip1559BaseFeeMultiplier*baseFee + tip, with the multiplier (2) documented as the EIP-1559-standard buffer (survives ~6 full blocks of +12.5%/block base-fee growth). Tunable via the named constant. This also drops the now-redundant eth_maxPriorityFeePerGas call (Suite floors the tip with coinInfo.minPriorityFee), so the on-chain path makes one RPC (eth_feeHistory) instead of two. Adds a unit test asserting the per-tier values and the maxFeePerGas > baseFee invariant. Note: this does not touch the alternative-fee (Infura) path, where the "fees too high" reports originate — that is a fee-policy decision pending coordination with the Suite team. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Document, with mermaid diagrams, how EIP-1559 fees are produced across Blockbook and Trezor Suite and which component owns which decision: blockbook provides ground-truth inputs (chain base fee, priority-fee tiers, congestion/trend, block gas), while the wallet owns the fee policy (base-fee source, 2x head-room buffer, maxFeePerGas composition, per-coin clamps). Covers the pull path (EthereumTypeGetEip1559Fees), the push path (subscribeNewBlock evmData), Suite's EthereumFeeLevels, the end-to-end flow, and the maxFeePerGas formula. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Move the two existing fee panels (estimated fee rate, alternative fee provider requests) out of General into a new top-level "Fees" section, so fee observability has one home as EVM EIP-1559 fee metrics are added. Pure relocation: panels, queries and the metrics they read are unchanged; only their x-panel-keys move general.* -> fees.* and a row.fees section is introduced. grafana.json is the git-ignored artifact rendered from these two sources by contrib/scripts/render_grafana.py. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
EVM EIP-1559 fees were invisible in metrics (blockbook_estimated_fee is
UTXO-only). Add two gauges populated on the successful estimateFee /
EthereumTypeGetEip1559Fees paths (provider cache hit and on-chain estimate):
- blockbook_eth_eip1559_fee{tier,kind}: per-tier maxFeePerGas / priority fee
- blockbook_eth_eip1559_base_fee: the next-block base fee underlying them
Values are raw wei (base units, like estimated_fee); Grafana divides by 1e9
to show Gwei. Emitted only on the two successful returns so error/disabled
paths never write zeros, and nil tiers are skipped so no empty series appear.
Dashboard (Fees section): maxFeePerGas by tier, priority tip by tier (own
axis - tips are ~1000x smaller), and the maxFee/baseFee buffer ratio with a
dashed 2x reference, which separates a congested chain from an estimator
regression - the "fees too high" diagnostic.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add blockbook_eth_eip1559_fee_source_total{source} incremented on each served
estimate at the serve boundary: provider (alternative provider cache hit),
onchain_fallback (provider configured but cache stale/unready, so eth_feeHistory
was used) or onchain (no provider configured).
This is the one signal neither alternative_fee_provider_requests (background
fetch outcome) nor the provider cache-age gauge captures: how often the wallet-
facing estimate actually bypassed the provider. Emitted only on the two
successful returns, alongside the fee gauges.
Dashboard: a non-stacked "EIP-1559 fee source" panel so a provider->fallback
transition shows as a visible crossover.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add blockbook_eth..._provider_last_sync_timestamp_seconds{provider}, the unix
time of the last successful refresh of the cached EVM provider (infura/1inch)
fees, set via a shared observeSync() called from both providers' processData
(replacing the bare p.lastSync = time.Now()) so lastSync and the metric always
share one instant.
Cache age is plotted as time() - metric. A timestamp gauge, not a computed
*_age_seconds one (the repo's usual form): it is written only on a successful
refresh, so the plotted age keeps climbing when a provider wedges and survives
restarts. This is the leading indicator that explains the onchain_fallback
counted by eth_eip1559_fee_source_total - once age crosses the stale window
(30x the poll period) the pull path falls back to on-chain.
Dashboard: a cache-age panel with a dashed default-cutoff (30m) reference.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add blockbook_eth_block_gas_used_ratio, gasUsed/gasLimit of the most recently connected EVM block (0..1) - the congestion signal that drives the next base fee (>0.5 it rises up to +12.5%/block, <0.5 it falls). This is the push path's own view of why fees move, independent of any provider. Set synchronously in OnNewBlock before the async broadcast: OnNewBlock is invoked in monotonic height order by the single writeBlockWorker, whereas the per-block broadcast goroutines can reorder and let an older block clobber a newer gauge value. Last-value semantics, nil-guarded for non-EVM/pre-London. Dashboard: a "Latest block gas-used ratio" panel (percentunit) with a dashed 50% balance-point reference. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…verlay Add blockbook_eth_block_base_fee, the realized base fee of the most recently connected block (push path), in raw wei. Set in the same OnNewBlock helper as the gas-used ratio, nil-guarded. Paired with eth_eip1559_base_fee (the pull path's next-block projection) on one overlay panel - solid mined vs dashed projected, both in Gwei. The two should track within a block's +/-12.5% step; a persistent gap means the feeHistory projection (or a provider's estimatedBaseFee) is drifting from the chain - the base-fee half of a "fees too high" investigation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Documentation-only follow-up from an adversarial review of the new Fees section (no functional change, no Go change): - order the section pull-estimate -> provider-health -> push -> legacy, and label the relocated UTXO "Estimated fee rate" so an EVM $coin selection reads as expected-empty rather than an outage - buffer-ratio: note Infura's high tier sits ~2.5x by design, so the dashed 2x line is the on-chain estimator's target only - base-fee overlay & cache-age: scope the "tracks within +/-12.5%" and "30x poll period" expectations to the path/provider they actually hold for (provider coins serve a cached projection up to the stale window old) - eth_eip1559_base_fee help: drop the unconditional "next-block projection" claim (true only on Erigon-like backends) and Infura's estimatedBaseFee field name (1inch feeds the same gauge from baseFee) - priority-tip: "separate panel", not "own axis" Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Bounds-check the per-tier column index so a non-compliant backend returning a reward row shorter than the requested percentile count is skipped instead of panicking EthereumTypeGetEip1559Fees. Divide the tip average by the rows that actually contributed, so a skipped short row does not deflate the tier. Document the accepted zero-tip case on idle chains, and add a regression test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The 1inch alternative fee provider never set staleSyncDuration, so it defaulted to the zero value (0s), making the cache immediately stale after every write. GetEip1559Fees always fell back to on-chain estimation via eth_feeHistory, defeating the purpose of having a provider cache. Mirror the same pattern used by the Infura provider: introduce oneInchFeeStalePeriods=30 and oneInchFeeStaleDuration, and set p.staleSyncDuration in the constructor so cached fees are reused within the 30×pollPeriod window (e.g. 30min for a 60s poll).
Replace the Infura gas API with 1inch Gas Price API for Ethereum mainnet archive. The 1inch API returns raw wei integers and uses Bearer token auth (ONE_INCH_API_KEY). The staleSyncDuration bug that made 1inch cache always immediately stale was fixed in the preceding commit.
Grafana: - alt_fee_provider_requests tooltip mode: single -> multi (inconsistent with other fee panels) - eip1559_max_fee description: generalize Infura-only language to mention both providers and clarify instant tier behavior differs per provider - eip1559_buffer_ratio description: generalize Infura-only padding reference - alt_fee_provider_cache_age description: generalize "Infura (the EVM default)" to mention both providers use the same 30x stale window docs/fees.md: - Replace Infura-only references with provider-agnostic language throughout - Keep accurate historical context (Infura as original source of fees-too-high reports) - Update mermaid flowcharts from "Infura fees" to "provider fees" - Document both Infura and 1inch behavior in the provider section
1inch returns 4 tiers (low/medium/high/instant) with tighter per-tier values than Infura (3 tiers). The previous 1:1 mapping (suite low/medium/high ← 1inch low/medium/high) produced fees that were too conservative compared to Infura, causing ~2x fee differences between providers. Remap so the suite's low/medium/high tiers are fed from comparable aggressiveness: 1inch medium → suite low 1inch high → suite medium 1inch instant → suite high 1inch's low tier is discarded (too conservative to be useful). The fee formula (maxFeePerGas = 2×baseFee + tip) is already correct on the suite side (fix/evm-fee branch) — this change ensures the input tier values are comparable regardless of which provider serves the response.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.