fix: upgrade to allora-sdk v1.1.0 (emissions v9 → v10)

[jefferythewind]

Summary

The Allora testnet upgraded from emissions v9 to v10, breaking all deployed workers with "unknown service emissions.v9.QueryService". This PR updates the builder kit to target the new SDK.

• worker_runtime.py — AlloraWorker() constructor → AlloraWorker.inferer() factory method; run_fn(nonce: int) → run_fn(ctx: RunContext) where ctx.nonce is the block height
• deploy_worker_raw.py — same factory-method + RunContext wrapper; result.prediction → result.submission
• worker_monitor.py — full v10 migration: proto import paths (allora_sdk.protos.* → allora_sdk.rpc_client.protos.*), emissions.v9 → emissions.v10, renamed GetWorkerLatestInputInferenceByTopicIdRequest, client.tx.query.get_txs_event, all gRPC queries now async; sync call sites wrapped with asyncio.run()
• pyproject.toml — SDK minimum bumped to >=1.1.0
• requirements.txt — documents the manual SDK install (clone → codegen → pip install) required until v1.1.0 is on PyPI; adds cosmpy for wallet-link extra
• README.md — prominent info box at the top: testnet-only scope, v10 install steps, mainnet one-liner for users still on emissions v9
• tests/test_worker_monitor.py — two new tests covering the async fetcher path and AlloraSDKEventFetcher coroutine contract

⚠️ SDK install note

allora-sdk v1.1.0 is not yet on PyPI. See the README info box for the required install steps (clone → make codegen → pip install -e .).

Test plan

• 115 tests pass locally (pytest tests/ excluding live-network tests)
• Verified full install sequence in a clean venv: clone SDK → codegen → install builder kit → all tests pass
• 102 workers restarted and confirmed submitting on testnet post-upgrade

🤖 Generated with Claude Code

---

Summary by cubic

Upgrade the builder kit to allora-sdk v1.1.0 and emissions v10 to restore testnet compatibility and move workers/monitor to async v10 RPCs. Also fixes a monitor crash by creating an AlloraRPCClient per call and streamlines v10 install steps.

• Migration

  • Use AlloraWorker.inferer() with run_fn(ctx: RunContext); replace result.prediction with result.submission.
  • Switch to allora_sdk.rpc_client.protos.*, use emissions v10 event types, rename request to GetWorkerLatestInputInferenceByTopicIdRequest, and call client.tx.query.get_txs_event.
  • Make SDK calls async: AlloraSDKEventFetcher.__call__ and _expected_worker_nonces are async; WorkerMonitor.sync_once handles async fetchers via asyncio.run().

• Bug Fixes

  • Eliminate loop-affinity crash (“Future attached to a different loop”) by instantiating a fresh AlloraRPCClient inside each fetcher call and _expected_worker_nonces; regression test asserts a new client per asyncio.run().
  • Tighten types: EventFetcherProtocol.__call__ is async; WorkerMonitor.__init__ accepts Awaitable[list[dict]] | list[dict]. Additional tests cover async fetchers and the coroutine contract.

^{Written for commit 16c5015. Summary will update on new commits.}

[cubic-dev-ai]

No issues found across 7 files

Architecture diagram

sequenceDiagram
    participant Worker as Worker Script
    participant SDK as then_sdk v1.1.0
    participant Fetcher as AlloraSDKEventFetcher
    participant Monitor as WorkerMonitor
    participant DB as SQLite DB

    Note over Worker,DB: Worker Runtime (v10 migration)

    Worker->>SDK: AlloraWorker.inferer(run_fn, ...)
    SDK->>SDK: NEW: factory method instead of constructor
    SDK->>Worker: ctx: RunContext
    Worker->>Worker: run_fn(ctx.nonce)
    Worker-->>SDK: submission value
    SDK->>SDK: Submit inference to chain
    SDK-->>Worker: result.submission (renamed from .prediction)

    Note over Fetcher,DB: Event Fetcher (async v10 migration)

    Monitor->>Fetcher: __call__(topic_id, address, since)
    alt Fetcher is async coroutine
        Fetcher->>Fetcher: CHANGED: async call
        Fetcher->>SDK: await client.emissions.query.get_topic()
        Fetcher->>SDK: await client.emissions.query.get_worker_latest_input_inference_by_topic_id()
        Fetcher->>SDK: await client.emissions.query.get_inferer_score_ema()
        Fetcher->>SDK: await client.emissions.query.get_previous_inference_reward_fraction()
        Fetcher->>SDK: await client.emissions.query.is_whitelisted_topic_worker()
        Fetcher->>SDK: await client.emissions.query.can_submit_worker_payload()
        Fetcher->>SDK: await client.tx.query.get_txs_event()
        SDK-->>Fetcher: results
        Fetcher-->>Monitor: event list
    else Sync fallback
        Monitor->>Fetcher: sync call (unchanged path)
        Fetcher-->>Monitor: event list
    end

    Note over Monitor,DB: Sync wrapper (async bridging)

    Monitor->>Monitor: CHANGED: _expected_worker_nonces() now async
    Monitor->>Monitor: CHANGED: _sync_target() wraps fetcher with asyncio.run()
    Monitor->>DB: insert events
    DB-->>Monitor: inserted count
    Monitor-->>Worker: sync results

    Note over Worker,Monitor: Proto imports migrated
    Note over Fetcher: allora_sdk.rpc_client.protos.emissions.v10 (was v9)
    Note over Fetcher: allora_sdk.rpc_client.protos.cosmos.tx.v1beta1 (was cosmo.tx.v1beta1)

Loading

_{Re-trigger cubic}

[cubic-dev-ai]

All reported issues were addressed across 2 files (changes from recent commits).

<sub>Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic</sub>

[clementupshot]

Review summary

This PR migrates the builder kit from allora-sdk emissions v9 to v10: AlloraWorker() → AlloraWorker.inferer(), run_fn(nonce) → run_fn(ctx: RunContext), proto imports to allora_sdk.rpc_client.protos.emissions.v10, all gRPC queries made async, and sync call sites bridged with asyncio.run() + inspect.isawaitable. The worker-runtime migration (worker_runtime.py, deploy_worker_raw.py) is correct — result.submission and ctx.nonce match SDK v1.1.0 (WorkerResult.submission, RunContext.nonce), and the factory-method call is right. The v9→v10 proto/request renames are complete (no stragglers found repo-wide).

The headline concern is in the monitor/dashboard path, not the worker path: AlloraSDKEventFetcher.__call__ is now async, but callers bridge back to sync via asyncio.run() on a long-lived AlloraRPCClient whose grpclib.Channel is loop-affine. The web dashboard's 5-second sync loop and per-request timeline calls will fail after the first successful gRPC connection, silently (the bare except Exception blocks swallow the RuntimeError). The author's reported success ("102 workers restarted and confirmed submitting") covers the worker submission path, not the monitoring surface this affects.

Verdict — needs attention

• 🔴 Critical: 1
• 🔴 High: 0
• 🟡 Medium: 4
  ▸ Minor (non-blocking): 7

Audit & simplification findings (non-blocking)

Two high-leverage reframings for human review (advisory, not blocking):

1. Construct AlloraRPCClient per __call__ instead of in __init__ — deletes the entire loop-affinity + cross-thread race bug class (FIND-001) in one move. Each call gets its own client+channel+loop, all torn down together. Slight TCP reconnect cost per 5s sync cycle is acceptable. The long-lived client was the mistake, not the asyncio.run bridging.

2. Make EventFetcherProtocol.__call__ async-only — deletes the inspect.isawaitable conditional branch, the inline import inspect, and fixes FIND-003 (type contract) simultaneously. Cost: breaks backward compat for any user-supplied sync fetcher (judgment call — if the only real fetcher is the now-async AlloraSDKEventFetcher, the bridge is dead code).

Next actions

• FIND-001 (critical): Fix the asyncio.run() + loop-affine grpclib.Channel interaction in worker_monitor.py. Either construct AlloraRPCClient per call, run a persistent background event loop, or make WorkerMonitor async-native.
• this finding: Narrow the 5 bare except Exception blocks in AlloraSDKEventFetcher.__call__ and _expected_worker_nonces so the loop-affinity failure surfaces instead of silently blacking out monitoring.
• this finding: Update EventFetcherProtocol.__call__ and the WorkerMonitor.__init__ type annotation to -> Awaitable[list[dict]] / async def.
• this finding: Pin requirements.txt to a commit SHA (not the mutable sdk-v10-upgrade branch) and resolve the pip install . + pip install -r requirements.txt redundancy in README Step 1.
• this finding: Fix the README mainnet fallback — pip install allora-forge-builder-kit fails (kit not on PyPI). Restore a pinned v9 GitHub commit or publish to PyPI.
• this finding: Pin the SDK clone in README to the v1.1.0 tag/commit.

---

Out-of-diff findings

Posted in the body because the referenced code isn't in this PR's diff (re-review scope, cross-module impact, or out-of-diff file).

🟡 MEDIUM — Bare except Exception swallows the loop-affinity failure, causing silent permanent monitoring blackout

allora_forge_builder_kit/worker_monitor.py

AlloraSDKEventFetcher.call has 5 bare except Exception: pass / break blocks (worker_monitor.py ~624,695,716,738,755,772) and _expected_worker_nonces has except Exception: return [] (worker_monitor.py:369). When the loop-affinity bug (this finding) raises RuntimeError: Event loop is closed, these catch it silently: call returns an empty/partial event list, _expected_worker_nonces returns []. sync_once() then reports {inserted: 0, errors: []} and the web dashboard _sync_loop (web_dashboard.py:274) sets _last_sync ok:True. The dashboard shows zero data with no error signal to the operator — monitoring is permanently blacked out with a green status.

The first tx-query try/except (worker_monitor.py:624) logs a WARNING on page 1 failure, but the dashboard does not surface logs to the user — only the sync status object, which reads ok:True. The other 4 blocks pass silently with no log at all.

Pre-existing pattern, but the async migration (this PR) materially worsens it: the new failure mode (loop-affinity) is now the most likely error these blocks will swallow, and it's a permanent failure (not transient network blip). this finding correction: CancelledError concern is moot (CancelledError inherits BaseException, not Exception, in Python 3.10+) — the real issue is swallowing RuntimeError/AttributeError from the async bridge.

Cross-dim: correctness (this finding, this finding), runtime-semantics (this finding, this finding), adversarial (this finding).

🟡 MEDIUM — EventFetcherProtocol and WorkerMonitor type annotations still declare sync, but AlloraSDKEventFetcher.call is now async

allora_forge_builder_kit/worker_monitor.py

EventFetcherProtocol.call (worker_monitor.py:21) declares def __call__(self, topic_id, address, since) -> list[dict] (sync). WorkerMonitor.init event_fetcher annotation (worker_monitor.py:58) is Optional[EventFetcherProtocol | Callable[[int, str, Optional[str]], list[dict]]] (sync return). But AlloraSDKEventFetcher.call (worker_monitor.py:595) is now async def __call__ → returns Coroutine[..., list[dict]].

Pyright confirms a type error at tests/test_worker_monitor.py:123 (async_fetcher passed to WorkerMonitor). The runtime compensates via inspect.isawaitable in _sync_target (worker_monitor.py:480), so this is a static-analysis / type-contract issue, not a runtime crash (severity downgraded from CRITICAL per this finding: the type system lies but the code runs).

The declared contract misleads any caller or static-analysis tool that relies on the type: they'd expect a sync list[dict] return and get a coroutine. The Protocol should declare -> Awaitable[list[dict]] or async def __call__.

Cross-dim: correctness (this finding), architecture (this finding), style (this finding).

Minor findings — 7 · non-blocking (style, tests, nits)

3 architecture · 1 style · 1 runtime-semantics · 1 test-coverage · 1 performance

🔵 Confirms cubic's finding: README testnet SDK clone not pinned — installs default branch, may not be v1.1.0
README.md · architecture

🔵 Hardcoded emissions version string will silently break on the next chain upgrade
allora_forge_builder_kit/worker_monitor.py · architecture

🔵 Inline import asyncio as _asyncio / import inspect as _inspect inside methods is non-idiomatic
allora_forge_builder_kit/worker_monitor.py · style

🔵 No asyncio.wait_for / timeout on outbound gRPC queries — a slow chain stalls the sync loop indefinitely
allora_forge_builder_kit/worker_monitor.py · runtime-semantics

🔵 getattr(fetcher, 'client', None) bypasses EventFetcherProtocol contract and masks missing-client AttributeError
allora_forge_builder_kit/worker_monitor.py · architecture

🔵 New tests only cover the happy path — no error-path, sync-compat, or get_submission_timeline async bridge test
tests/test_worker_monitor.py · test-coverage

⚪ 6 independent point-in-time gRPC queries run sequentially — could be asyncio.gather'd for latency
allora_forge_builder_kit/worker_monitor.py · performance

[jefferythewind]

Review fixes (commit 16c5015)

Addressed all review findings from @clementupshot and @cubic-dev-ai:

Critical — FIND-001: loop-affinity crash in AlloraSDKEventFetcher (resolved)

AlloraSDKEventFetcher.__init__ was creating a single AlloraRPCClient (backed by a grpclib.Channel) outside any asyncio.run() context. grpclib captures self._loop = asyncio.get_event_loop() at construction; each asyncio.run() call drives a different event loop, so _create_connection registers I/O on the wrong loop → RuntimeError: Future attached to a different loop. This was swallowed silently by the five bare except Exception blocks, returning [] and reporting ok: True in the dashboard — permanent monitoring blackout.

Fix: store self._cfg in __init__, construct a fresh AlloraRPCClient at the top of each __call__ invocation. Same fix applied to _expected_worker_nonces (also bridged via asyncio.run()). New regression test confirms: pre-fix produces 1 client total and fails; post-fix produces 1 per call and passes.

Medium — type annotations (resolved)

• EventFetcherProtocol.__call__ updated to async def to match the real implementation
• WorkerMonitor.__init__ event_fetcher annotation updated to Awaitable[list[dict]] | list[dict]

Style — inline imports (resolved)

import asyncio as _asyncio / import inspect as _inspect inside methods moved to module-level imports.

README mainnet fallback (resolved)

Restored pip install "allora-forge-builder-kit @ git+https://github.com/allora-network/allora-forge-builder-kit.git@8ef3200" — bare pip install allora-forge-builder-kit fails because the kit is not on PyPI.

requirements.txt mutable branch ref (resolved)

Changed @sdk-v10-upgrade → @main.

Not addressed (by design)

• SDK clone not pinned in README — left unpinned intentionally; the clone always pulls the latest SDK.

[cubic-dev-ai]

2 issues found across 4 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="allora_forge_builder_kit/worker_monitor.py">

<violation number="1" location="allora_forge_builder_kit/worker_monitor.py:601">
P1: Each monitoring invocation now creates a new gRPC client but never closes it. The dashboard calls `sync_once()` every five seconds and can process many targets, so the new per-call clients can accumulate open grpclib channels/sockets and eventually exhaust resources. The SDK's `AlloraRPCClient` exposes an async `close()` that closes its `grpc_client`; closing each fresh client in `finally` would preserve the loop-affinity fix without leaking connections.</violation>
</file>

<file name="requirements.txt">

<violation number="1" location="requirements.txt:1">
P1: The documented v10 install can silently replace the checkout with the old v9 package. `pip install -r requirements.txt` resolves this direct URL from `main`, while the verified target branch `origin/main` is still the v9 code (`pyproject.toml` requires `allora-sdk>=1.0.6` and `worker_monitor.py` imports `allora_sdk.protos.emissions.v9`). Because the README first installs the current checkout and then runs this requirements file, pip can reinstall the same `3.0.0` distribution from `main`, leaving the user with v9 monitor code and defeating the manual v1.1.0 SDK setup. Referencing the local checkout (or otherwise pinning the v10 commit) keeps the install aligned with the code being documented.</violation>
</file>

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Fix all with cubic | Re-trigger cubic</sub>

[cubic-dev-ai]

P1: Each monitoring invocation now creates a new gRPC client but never closes it. The dashboard calls sync_once() every five seconds and can process many targets, so the new per-call clients can accumulate open grpclib channels/sockets and eventually exhaust resources. The SDK's AlloraRPCClient exposes an async close() that closes its grpc_client; closing each fresh client in finally would preserve the loop-affinity fix without leaking connections.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At allora_forge_builder_kit/worker_monitor.py, line 601:

<comment>Each monitoring invocation now creates a new gRPC client but never closes it. The dashboard calls `sync_once()` every five seconds and can process many targets, so the new per-call clients can accumulate open grpclib channels/sockets and eventually exhaust resources. The SDK's `AlloraRPCClient` exposes an async `close()` that closes its `grpc_client`; closing each fresh client in `finally` would preserve the loop-affinity fix without leaking connections.</comment>

<file context>
@@ -602,6 +598,8 @@ async def __call__(self, topic_id: int, address: str, since: Optional[str]) -> l
             IsWhitelistedTopicWorkerRequest,
         )
 
+        client = AlloraRPCClient(network=self._cfg)
+
         since_dt = _parse_dt(since)
</file context>

[cubic-dev-ai]

P1: The documented v10 install can silently replace the checkout with the old v9 package. pip install -r requirements.txt resolves this direct URL from main, while the verified target branch origin/main is still the v9 code (pyproject.toml requires allora-sdk>=1.0.6 and worker_monitor.py imports allora_sdk.protos.emissions.v9). Because the README first installs the current checkout and then runs this requirements file, pip can reinstall the same 3.0.0 distribution from main, leaving the user with v9 monitor code and defeating the manual v1.1.0 SDK setup. Referencing the local checkout (or otherwise pinning the v10 commit) keeps the install aligned with the code being documented.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At requirements.txt, line 1:

<comment>The documented v10 install can silently replace the checkout with the old v9 package. `pip install -r requirements.txt` resolves this direct URL from `main`, while the verified target branch `origin/main` is still the v9 code (`pyproject.toml` requires `allora-sdk>=1.0.6` and `worker_monitor.py` imports `allora_sdk.protos.emissions.v9`). Because the README first installs the current checkout and then runs this requirements file, pip can reinstall the same `3.0.0` distribution from `main`, leaving the user with v9 monitor code and defeating the manual v1.1.0 SDK setup. Referencing the local checkout (or otherwise pinning the v10 commit) keeps the install aligned with the code being documented.</comment>

<file context>
@@ -1 +1 @@
-allora-forge-builder-kit[dev,wallet-link] @ git+https://github.com/allora-network/allora-forge-builder-kit.git@sdk-v10-upgrade
+allora-forge-builder-kit[dev,wallet-link] @ git+https://github.com/allora-network/allora-forge-builder-kit.git@main
</file context>

Suggested change

	allora-forge-builder-kit[dev,wallet-link] @ git+https://github.com/allora-network/allora-forge-builder-kit.git@main
	-e ".[dev,wallet-link]"