Skip to content

Add Persistent Soroban Event Indexer with Resumable Backfill and Historical Query Command (starforge events) #57

Description

@Nanle-code

Problem Statement

starforge monitor can stream live Soroban contract events via getEvents, but it holds no memory: every invocation starts from the current ledger, nothing is persisted, and there is no way to ask "what happened on this contract between ledger X and ledger Y" after the fact. Developers debugging a production incident, reconciling contract state, or building support tooling on top of StarForge currently have to stand up their own indexing infrastructure or re-implement pagination against Soroban RPC from scratch.

Proposed Solution

Introduce a new starforge events command family backed by a local, embedded SQLite database (~/.starforge/data/events.db) that durably indexes Soroban contract events:

  • starforge events index --contract <C...> [--from-ledger N] [--network testnet|mainnet] — backfills historical events via getEvents pagination, persisting a resumable cursor (last processed ledger + pagination token) so an interrupted backfill can continue where it left off rather than restarting from genesis.
  • starforge events tail --contract <C...> [--follow] — indexes new events as they arrive, sharing the same storage layer as index so live and historical data are queried identically.
  • starforge events query --contract <C...> [--topic <matcher>] [--from-ledger N] [--to-ledger N] [--limit N] [--format table|json|csv] — queries the local index with topic wildcard matching (reusing the existing filter syntax from monitor), ledger-range filtering, and pagination.
  • starforge events export --contract <C...> --output <file> — dumps the full indexed history for a contract to JSON or CSV for offline analysis.

Technical Scope

  • New src/utils/event_index.rs: SQLite schema (via rusqlite), CRUD/query layer, cursor persistence, and a migration path consistent with the existing schema-versioning pattern in utils/config/migrations.rs.
  • New src/commands/events.rs: clap subcommand definitions for index, tail, query, export.
  • Reuse the topic-matching and event-type filtering logic already implemented for monitor (utils/stream.rs) rather than duplicating it.
  • Backfill must page through getEvents respecting the RPC's retention window and surface a clear error when requested ledgers have already been pruned by the RPC provider.
  • Concurrent-safe writes (a tail --follow process and a manual query should not corrupt the database).
  • Unit tests for cursor resume logic, topic filtering against the local store, and schema migration; an integration test against a mocked RPC provider exercising backfill interruption/resume.

Acceptance Criteria

  • Interrupting starforge events index (Ctrl+C) and re-running it resumes from the last successfully persisted ledger without re-fetching or duplicating events.
  • starforge events query returns correct results for ledger-range and topic-filter combinations, verified by tests.
  • Database size and query latency are documented for a benchmark of 100k indexed events.
  • Existing starforge monitor command and its tests are unaffected.
  • README updated with the new command family and its use cases (incident debugging, support tooling, offline analytics).

Estimated Scope

Approximately 700 lines of new Rust across the two new modules, CLI wiring, and tests.

Metadata

Metadata

Assignees

No one assigned

    Labels

    GrantFox OSSIssue tracked in GrantFox OSSMaybe RewardedIssue may be eligible for a GrantFox rewardOfficial Campaign | FWC26Campaign: Official Campaign | FWC26enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions