Start DIN-Indexer

[umeradl]

Branch: feat/din-indexer
Assignee: @robertocarlous
Roadmap ref: P4-IDX1 (design) · P4-IDX2 (implementation) · P4-IDX3 (integration) · Developer/issues/indexer.md

Long-running issue. This was Option B of Robbert's original two-option task (June 2026); he chose Option A — upgradeable platform contracts, delivered as PR #13 and tracked in Developer/tasks/task_240626_0b.md. Option B now becomes its own effort here, aligned with the P4-IDX work-package series (scheduled Oct–Nov 2026, but design work can start earlier). Detailed task files under Developer/tasks/ will follow.

Goal

Build an indexer over the events of the 4 platform-level contracts (DinCoordinator, DinToken, DinValidatorStake, DINModelRegistry) and wire at least one real dincli RPC-loop call site to query it instead of enumerating on-chain counters over RPC.

Guiding principle (from Developer/issues/indexer.md): contracts stay the canonical coordination/settlement layer with strong events; the indexer owns filtering, pagination, dashboards, and historical queries. No storage-heavy on-chain enumeration for read convenience.

Scope

Design (P4-IDX1)

• Choose the indexing approach and document the tradeoff: The Graph Protocol subgraph (preferred — production experience) vs. a lighter alternative (Ponder, or a custom event-poller backed by a local DB) if local subgraph test infra is a blocker
• Entity schema covering all event streams of the 4 platform contracts (registrations, requests/approvals, stake/slash/blacklist, fee changes, transfers)
• subgraph.yaml + schema.graphql (or the equivalent for the chosen tool)
• Local Graph node (or equivalent) running against the local Hardhat chain for development testing
• Event-coverage audit: which meaningful state transitions currently emit no event, or events with insufficient/unindexed fields, for clean state reconstruction — propose minimal contract-side event additions rather than storage changes
• Daemon event schema: define which events dind needs to subscribe to and current coverage gaps — this feeds P4-7.1 (on-chain event listening) in the "start DIN daemon" issue (start DIN daemon #21)

Implementation (P4-IDX2)

• Mapping handlers for all platform-contract events; verify entities index correctly against test transactions on the local node
• Replace the pending-request enumeration loop in dincli/cli/dindao.py (lines ~441–465: for idx in range(totalModelRequests) and the matching manifest-request loop) with an indexer-backed query, proving the integration end-to-end
• Local run instructions (setup steps, docker-compose if needed) + ≥3 example queries covering validator registry, model registry, and reward/slash history
• Write-up: approach chosen and why, integration point, and the follow-up candidate loop sites — known ones today: dincli/cli/modelownerd/lms.py (~line 62) and aggregation.py (~lines 76, 95, 136)

Integration & wrap-up (P4-IDX3)

• Wire the indexer into the dincli test suite; the replaced call site passes tests against the indexed local node
• Handoff notes: what follows in P5+ (task-level contract indexing via dynamic data sources, event-schema extensions for dind)
• Doc-correction pass on Developer/issues/indexer.md: note stale references or contradictions with current contract code (same convention as the PR Feature/platform upgradeable #13 doc-inaccuracy list)

Explicitly out of scope

• Dynamic data sources for task-level contracts (DINTaskCoordinator/DINTaskAuditor) — materially harder Graph pattern with its own deploy/hosting story; deferred to P5+, document as follow-up only
• Rewriting dincli's general data-access layer (DinContext stays as-is beyond the one wired call site)
• Hosting / production deployment of the indexer
• Governance dashboards — the DIN-DAO issue (Start DIN-DAO #23) designs its contracts events-first so this indexer can cover them later

Dependencies & sequencing

Per Developer/ROADMAP.md, P4-IDX1 depends on stable contract ABIs (P3-6.3b) and proxy contracts deployed (PR #13) — indexing against ABIs that are still churning means redoing mappings. Design work (approach tradeoff, entity schema, event-coverage audit) is safe to start earlier; hold mapping implementation until the ABI surface settles.

References

• Developer/issues/indexer.md — the indexer-first architecture blueprint (events-first contract design, what stays on-chain vs. moves to the indexer)
• Developer/ROADMAP.md — P4-IDX1/IDX2/IDX3 work packages
• Original task spec: Option B in Robbert's June task (kept outside the repo); condensed Option A record at Developer/tasks/task_240626_0b.md
• Related issues: start DIN daemon #21 "start DIN daemon" (consumes the daemon event schema), Start DIN-DAO #23 "Start DIN-DAO" (future governance event streams)

Branching note

feat/din-indexer on the main repo (contributor branch, not a fork-based trial). Indexer code should live in its own top-level directory (e.g. indexer/ or subgraph/ — pick one and document it) so it doesn't tangle with dincli/, foundry/, or hardhat/.

[robertocarlous]

Hi Umer, Please i need some clarification and answer to the following Questions

1. Missing event fields — propose additions or use call handlers?

ModelRegistrationRequested only emits requestId and requester. The isOpenSource flag and fee amount are not in the payload to filter pending requests by type in the indexer I'd either need to add those fields to the event, or use a contract call inside the mapping handler. Same for ManifestUpdateRequested which emits requestId and modelId but not the requester address. My preference is to propose minimal event additions (one or two extra fields) rather than call handlers, since call handlers make the subgraph less portable. Is that acceptable, or do you want zero contract changes and call handlers instead?

2. RewardClaimed, GIStarted, GIEnded in the ROADMAP entity schema

The ROADMAP (P4-IDX1 description) lists these as target events for the entity schema, but none of them exist in the 4 platform contracts. They look like task-level contract events (DINTaskCoordinator). Are these planned additions to the platform contracts, or were they listed by mistake and the schema should only cover what the platform contracts actually emit today?

3. Directory name

The issue says pick one: indexer/ or subgraph/. I lean toward subgraph/ if we go with The Graph (names the tool clearly), indexer/ if we keep it tool-agnostic. Which do you prefer?

4. Local Graph node vs Hardhat chainId

The Hardhat local network is configured with chainId: 1337. The Graph node's networks.toml needs to match that chain. Should the local dev setup target hardhat network (chainId 1337) or the localhost fork of Optimism Sepolia (chainId 11155420 as configured in hardhat.config.ts)?

5. Confirming the only in-scope P4-IDX2 replacement target

The lms.py (~line 62) and aggregation.py (~lines 76, 95, 136) loops are against DINTaskAuditor / DINTaskCoordinator — task-level contracts, which are explicitly out of scope. Confirming: the only loop I replace in P4-IDX2 is the dindao.py pending-requests enumeration (lines 443–467), correct?

[umeradl]

Hi Robert, answers to each below.

1. Missing event fields — event additions confirmed, no call handlers.

Yes — propose minimal event additions. That's the direction Developer/issues/indexer.md already commits to ("adding missing events for important transitions / improving indexed event fields" are listed as the good minimal contract changes), and the issue's event-coverage audit checkbox says exactly this: "propose minimal contract-side event additions rather than storage changes." Call handlers are also a practical problem beyond portability — they don't work on all Graph node backends and would slow down indexing.

Concretely, for the two events you named:

• ModelRegistrationRequested(uint256 indexed requestId, address indexed requester) → add bool isOpenSource and uint256 fee
• ManifestUpdateRequested(uint256 indexed requestId, uint256 indexed modelId) → add address indexed requester

Since these land as part of the event-coverage audit (P4-IDX1 checkbox), write them up as proposals in the audit doc first rather than pushing contract changes directly — then land all event additions together as one small reviewed PR once the audit is complete. Note the platform contracts are behind proxies now (PR #13), but since they're only deployed locally (anvil/hardhat) at this point, redeploying/upgrading with the new event signatures is low-friction — no live-chain migration concern.

2. RewardClaimed/GIStarted/GIEnded — scope the schema to what the platform contracts actually emit today.

You're right — none of those three events exist in the platform contracts (they're not even task-level events under those names: DINTaskAuditor has RewardDeposited, not RewardClaimed, and neither task contract emits GI start/end events). Treat the ROADMAP wording as placeholder text from before any event audit was done.

The priority is: work on the platform contracts first and get a running indexer/subgraph end-to-end — schema, mappings, local Graph node, the dindao.py call-site integration. Only after that's running do we start on task-level contracts. Task-level indexing is also a different kind of effort: the events follow the federated-learning lifecycle (Global Iterations, LMS, auditor scoring, two-tier aggregation, slashing), so understanding the FL workflow matters a lot there — Documentation/public/workflows/model-workflow.md is the place to build that understanding when the time comes. For now, anything you notice about missing GI/reward events goes in the P4-IDX3 handoff notes, not this schema.

One heads-up for the schema: SlasherContractAdded/SlasherContractRemoved are emitted by both DinCoordinator and DinValidatorStake with identical signatures — keep the source contract distinguishable in your entities so the two streams don't get conflated.

3. Directory name: subgraph/.

Go with subgraph/ — we're committing to The Graph per P4-IDX1, so name the directory after the tool. Alongside the code, document how to run it locally (setup steps, docker-compose if needed — this is already a P4-IDX2 checkbox) and how dincli integrates with it. Integration order is: dincli now (the dindao.py call site), then the SDK and the daemon (dind) later, in that order, as each becomes available — write the run/integration docs so those follow-on consumers can plug in without rework (your daemon event schema from P4-IDX1 already feeds dind's P4-7.1).

4. Local Graph node — target chainId 1337.

Point the Graph node at the plain local chain, chainId 1337. Both local options use it: foundry/anvil.sh runs anvil with --chain-id 1337, and the hardhat network in hardhat.config.ts is also chainId: 1337. The 11155420 you saw is only the sepolia_op_devnet network (the live chain). No forked chain needed for indexer dev — deploy the platform contracts fresh to the local node and index that. Use anvil (./foundry/anvil.sh) as the canonical local chain, since that's what the documented devnet workflow uses (dincli system configure-network --network local).

5. Confirmed — dindao.py is the only in-scope replacement.

Correct. The only loop you replace in P4-IDX2 is the pending-requests enumeration in dincli/cli/dindao.py (list-pending-requests, both the model-request and manifest-request loops). You read it right: the lms.py and aggregation.py loops run against DINTaskCoordinator/DINTaskAuditor — task-level contracts, out of scope here. List them in your write-up as follow-up candidate call sites (per the P4-IDX2 checkbox) and leave the code untouched — we'll sort those loops out once we start working on the task-level contracts.

Timeline. No hard deadline on this issue — the ROADMAP dates were drawn up for a one-developer setup and don't apply now that we're working as a team. The indexer/graph is needed for DevNet 3.0, so there's runway. And nothing is blocking anything: the contracts and ABIs are there, so go straight through design into implementation and tests. If contract changes land later (e.g. the event additions from your audit), just update the mappings then — that's cheap compared to waiting.

[robertocarlous]

Thank You Umer, i will look into this!

[umeradl]

@robertocarlous One correction and a note on what to work on next.

Correction — there is no Phase 5. The "deferred to P5+" wording in this issue body and my earlier comment was a mistake: Developer/ROADMAP.md has no P5 — the roadmap currently runs through P3 (cryptoeconomic layer) and P4 (daemon + indexer). Please disregard the P5+ references.

Next step: until your next weekly task is assigned, continue working on the subgraph — extend it to the task-level contracts (DINTaskCoordinator / DINTaskAuditor) via dynamic data sources, the follow-up previously labelled out of scope here. Since task-level events follow the federated-learning lifecycle (Global Iterations, LMS, auditor scoring, two-tier aggregation, slashing), read Documentation/public/workflows/model-workflow.md first to build that context before designing the task-level entities. The lms.py and aggregation.py RPC-loop call sites from your PR #29 handoff notes are the natural integration targets.

[robertocarlous]

Thanks for the correction noted. I'll treat the task-level dynamic data sources as the immediate continuation of this work, not a future milestone.

Read through Documentation/Model-workflow.md. The GI lifecycle is clear: aggregator/auditor registration → LMS submission by clients → auditor evaluation/scoring → T1 and T2 aggregation → slashing → GI end. Each phase maps to a set of events on DINTaskCoordinator and DINTaskAuditor. Will audit those contract events next, then design the task-level entities and dynamic data source config before writing any handlers.

The lms.py (~line 62) and aggregation.py (~lines 76, 95, 136) RPC loops from the handoff notes are the planned integration targets once the schema is in place.

[robertocarlous]

Hi @umeradl Pls can you take look at this

Task-level contract event audit — findings and a design question before proceeding

Read through both contracts. Here is what exists and what is missing before I design the entities.

Existing events

DINTaskCoordinator emits 4 events: DINValidatorRegistered (aggregator registration), Tier1BatchAuto, Tier2BatchAuto (batch creation), and AggregatorSlashed.

DINTaskAuditor emits 8 usable events: DINAuditorRegistered, AuditorsBatchAuto, AuditorsBatchesCreated, AuditScoreSubmitted, EligibilityVoted, EligibilityFinalized, PassScoreUpdated, AuditorSlashed. One declared event — RewardDeposited — is never emitted (no depositRewards function exists); flagging as dead code.

Critical gaps

Three functions that are direct targets for replacing RPC loops emit no event:

• submitLocalModel() on DINTaskAuditor — no event. lms.py ~line 62 iterates getClientModels() over RPC and cannot be replaced without a LocalModelSubmitted event.
• submitT1Aggregation() and submitT2Aggregation() on DINTaskCoordinator — no events. aggregation.py ~lines 76, 95, 136 iterates T1/T2 batches over RPC and cannot be replaced without submission events.

Additionally, the entire 23-state GIstates machine is silent — none of the 14 phase-transition functions (startGI, startLMsubmissions, closeLMsubmissions, startT1Aggregation, endGI, etc.) emit an event.

Proposed minimal additions (same approach as the platform-contract audit)

Contract | Addition | Signature -- | -- | -- DINTaskAuditor | LocalModelSubmitted | (uint indexed gi, uint indexed modelIndex, address indexed client, bytes32 modelCID) DINTaskCoordinator | T1AggregationSubmitted | (uint indexed gi, uint indexed batchId, address indexed aggregator, bytes32 cid) DINTaskCoordinator | T2AggregationSubmitted | (uint indexed gi, uint indexed batchId, address indexed aggregator, bytes32 cid)

Design question before proceeding

For the GI state machine: should I propose a single catch-all GIStateChanged(uint indexed gi, uint8 indexed newState) emitted by all 14 silent transition functions — or would you prefer individual named events per phase (GIStarted, LMSOpened, LMSClosed, T1Started, GIEnded, etc.)? The catch-all is fewer contract changes; individual events are more readable in the subgraph and easier to filter in dind. Flagging this before designing the entities so I pick the right schema shape.

[umeradl]

@robertocarlous Good audit — I checked everything against develop and it's all accurate: the event inventory, RewardDeposited declared but never emitted (no depositRewards function exists), no events on submitLocalModel / submitT1Aggregation / submitT2Aggregation, and the fully silent GI state machine. Answers below.

1. GI state machine — go with the single catch-all GIStateChanged.

event GIStateChanged(uint indexed GI, uint8 indexed newState);

Reasons:

• The state machine is bigger than the 14 owner-callable transition functions: GIstate is written at 23 sites in DINTaskCoordinator (constructor, the pre-GI setup steps AwaitingDINTaskAuditorToBeSet → GenesisModelCreated, and writes inside the batch-creation and aggregation-close flows). Individual named events would mean ~20 declarations and 23 emit sites to keep in sync — a lot of ABI surface for one enum.
• The clean implementation is an internal setter: replace every GIstate = GIstates.X; with _setGIstate(GIstates.X) that assigns and emits. One declaration, one emit site, and it's impossible to miss a transition when states are added later.
• Filterability in dind isn't actually lost: with newState indexed, any consumer can topic-filter for a single state — e.g. subscribe to GIStateChanged with newState == 16 (T1AggregationStarted) to queue aggregation jobs. That's exactly the P4-7.1 wiring. The named T1AggregationStarted event in the ROADMAP is placeholder wording from before this audit, same as the GIStarted/GIEnded names we already corrected.
• Readability belongs in the subgraph, not the Solidity ABI: define a GraphQL enum in schema.graphql mirroring GIstates (DINShared.sol is the source of truth — dincli already mirrors it in its GI state helpers) and map uint8 → enum in one handler. Queries read just as well as named events would, with one handler instead of fourteen near-identical ones.
• It's also the Developer/issues/indexer.md default: minimal contract changes for event coverage.

One simplification for you: the entire state machine lives in DINTaskCoordinator — DINTaskAuditor only reads state cross-contract — so this is a single-contract change.

2. Three proposed submission events — confirmed, signatures are right.

Verified against storage: modelIndex / client / modelCID match the LMSubmission struct, and bytes32 is correct for the T1/T2 CIDs. Two small notes:

• Match the existing param style (uint indexed GI, capitalized) so the new events stay consistent with the current ABI.
• T2AggregationSubmitted.batchId is always 0 today (TC_OnlyOneTier2Batch enforces it) — keep it anyway for symmetry, just noting so it doesn't surprise you in the data.

One more candidate to run through the same audit filter: closeT1Aggregation / closeT2Aggregation select the majority CID per batch, and that winner lives only in storage. With submission events alone, the indexer would have to re-implement the vote-majority logic to know the finalized CID — and the T2 winner is the new global model, the single most important value in a GI. Consider proposing T1BatchFinalized(uint indexed GI, uint indexed batchId, bytes32 winningCID) and T2Finalized(uint indexed GI, bytes32 globalModelCID) alongside. Your call after reading the close functions.

3. RewardDeposited dead code — agreed, flag it.

Document it in the audit write-up as dead code with a removal proposal, but don't remove it yet — the P3 cryptoeconomics work (fee routing / reward distribution, P3-5.1–5.3) may introduce a real deposit flow, and that work should decide whether the event gets an emitter or gets deleted.

Process — same as the platform-contract audit: write these up as proposals in the audit doc first, then land all task-contract event additions as one small reviewed PR. Put the audit doc under Documentation/technical/audits/ (the foundry security review already lives there — follow that convention). One difference from the platform contracts: task contracts aren't proxied — they're deployed fresh per model by the model owner — so the new events only apply to newly deployed task contracts; already-deployed tasks on sepolia_op_devnet won't emit them. That's fine for dynamic data sources going forward, just state it explicitly in the write-up.

[umermjd11]

Status check (2026-09-08)

Comprehensive implementation in PR #29 — open, not draft, appears to close this issue on merge:

• Covers P4-IDX1 (event-coverage audit, entity schema, subgraph manifest, daemon event schema), P4-IDX2 (AssemblyScript mapping handlers, docker-compose local Graph node stack, 14 example queries), P4-IDX3 (both RPC enumeration loops in dincli/cli/dindao.py replaced with GraphQL + silent RPC fallback, 6 unit tests, handoff notes)
• Goes beyond this issue's stated scope: also indexes the task-level contracts (DINTaskCoordinator/DINTaskAuditor) via dynamic data sources, which this issue explicitly listed as out of scope ("deferred to P5+") — not a problem, just wider than asked
• 5 handlers are commented out pending contract event additions proposed in the audit — flagged clearly in the PR as the one real gap, not hidden

Not yet merged: dincli/cli/dindao.py:443 on current develop still has the raw for idx in range(totalModelRequests) RPC loop this PR is meant to replace.

Leaving this open until #29 merges — but of the four P4-kickoff issues (#20/#21/#23/#24), this looks closest to being done outright.