Add the ITN client (mina_sdk.itn) and execute_query

.github/workflows/ci.yml

@@ -108,7 +108,7 @@ jobs:
       - name: Run unit tests with coverage
         run: |
           pytest tests/test_types.py tests/test_daemon_client.py \
-            tests/test_spec.py tests/test_common_api.py \
+            tests/test_spec.py tests/test_common_api.py tests/test_itn.py \
             -v --cov=mina_sdk --cov-report=term-missing --cov-report=xml
 
       - name: Upload coverage

CHANGELOG.md

@@ -26,6 +26,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   epoch data, dates, ledger hashes, coinbase, fee transfers and user
   commands; kind, source, receiver, amount, fee and memo of a sent command.
 - `AccountNotFoundError`, a `ValueError`, from `get_account`.
+- `execute_query`, for custom GraphQL documents, as in the other SDKs.
+- ITN client, `mina_sdk.itn` (extra `itn`, which adds `cryptography`):
+  `ItnClient` for the daemon's ITN GraphQL server (`--itn-graphql-port`),
+  with ed25519 request signing (`ItnKey`), the `auth` handshake, sequence
+  numbers (one request at a time, a new auth after HTTP 412) and no repeat
+  after a transport error. It covers every operation of
+  `spec/itn-operations.graphql`, as the Rust, Go and JS SDKs do.
+- `spec/` is mina-sdk-spec v0.1.1.
 - Integration tests of the new methods.
 - `DaemonConnectionError` exception (replaces shadowed `ConnectionError`)
 - `Currency.__rmul__` for `3 * Currency(10)` support

README.md

@@ -67,7 +67,7 @@ The Mina SDKs have the same API, defined in
 [mina-sdk-spec](https://github.com/o1-labs/mina-sdk-spec). `spec/` is a copy
 of it at the tag in `spec/VERSION`. `tests/test_spec.py` checks that this
 SDK's queries are the specification's documents, and CI checks that `spec/`
-is the tag's copy. The ITN client of the specification is not in this SDK yet.
+is the tag's copy.
 
 ### Queries
 
@@ -89,6 +89,7 @@ is the tag's copy. The ITN client of the specification is not in this SDK yet.
 | `get_tracked_accounts()` | `list[TrackedAccount]` | Accounts in the daemon's keystore |
 | `get_snark_pool()` | `list[CompletedWork]` | Completed snark work |
 | `get_fork_config()` | `Any` | The daemon's fork configuration (JSON) |
+| `execute_query(query, variables=None, query_name="custom")` | `dict` | Run a custom GraphQL query |
 
 ### Mutations
 
@@ -101,6 +102,53 @@ is the tag's copy. The ITN client of the specification is not in this SDK yet.
 | `set_snark_worker(public_key)` | `str \| None` | Set/unset SNARK worker |
 | `set_snark_work_fee(fee)` | `str` | Set SNARK work fee |
 
+### ITN server (`mina_sdk.itn`)
+
+A daemon started with `ITN_FEATURES=1`, `--itn-graphql-port` and `--itn-keys`
+serves a second GraphQL API, which load testing tools use. `ItnClient` signs
+each request with an ed25519 `ItnKey` whose public half must be in
+`--itn-keys`, and it handles the daemon's sequence numbers (a new `auth`
+after a daemon restart, HTTP 412). It needs the `itn` extra:
+
+```bash
+pip install 'mina-sdk[itn]'
+```
+
+```python
+from mina_sdk.itn import ItnClient, ItnKey
+
+key = ItnKey.from_base64(open("itn_sk").read())
+print("start the daemon with --itn-keys", key.public_key_base64())
+
+with ItnClient("http://127.0.0.1:3086/graphql", key) as itn:
+    logs = itn.internal_logs(0)
+    if logs:
+        itn.flush_internal_logs(logs[-1].id)
+```
+
+| Method | GraphQL |
+|--------|---------|
+| `auth()` | `auth` (server UUID, sequence number, peer ID, block producer) |
+| `slots_won()` | `slotsWon` |
+| `internal_logs(start)` / `flush_internal_logs(end)` | `internalLogs` / `flushInternalLogs` |
+| `schedule_payments(PaymentsDetails)` | `schedulePayments` |
+| `schedule_zkapp_commands(ZkappCommandsDetails)` | `scheduleZkappCommands` |
+| `stop_scheduled_transactions(handle)` | `stopScheduledTransactions` |
+| `update_gating(GatingUpdate)` | `updateGating` |
+| `stop_daemon(delay_seconds=None, clean_config=None)` | `stopDaemon` |
+| `set_zkapp_command_limit(limit)` | `zkAppCommandLimit` |
+| `execute_query(query, variables, query_name)` | any document, sequenced and signed |
+
+Requests of one client are sent one at a time, because the daemon accepts
+only its exact next sequence number; the client is safe to share between
+threads. A sequenced request is never repeated after a transport error,
+because the daemon may already have run it. Errors: `ItnUnauthorizedError`
+(401), `ItnSequencingError` (412 after a new auth), `InvalidItnKeyError`, and
+`DaemonConnectionError` whose `__cause__` is an `ItnHttpError` with the
+status. The documents are those of `spec/itn-operations.graphql`. Integration
+tests: set `MINA_ITN_URI` and `MINA_ITN_KEY` and run
+`pytest tests/test_itn_integration.py`.
+
 ### Currency
 
 ```python

pyproject.toml

@@ -32,7 +32,10 @@ dependencies = [
 
 [project.optional-dependencies]
 archive = ["asyncpg>=0.29"]
+# The ITN client (mina_sdk.itn) signs requests with ed25519.
+itn = ["cryptography>=42"]
 dev = [
+    "cryptography>=42",
     "pytest>=8.0",
     "pytest-asyncio>=0.24",
     "pytest-cov>=6.0",
@@ -86,6 +89,7 @@ ignore = [
 "examples/*" = ["T201"]
 # The documents are copied from spec/operations.graphql and are not wrapped.
 "src/mina_sdk/daemon/queries.py" = ["E501"]
+"src/mina_sdk/itn/queries.py" = ["E501"]
 
 [tool.ruff.format]
 quote-style = "double"

spec/ITN.md

@@ -3,11 +3,12 @@
 A daemon started with `ITN_FEATURES=1`, `--itn-graphql-port <port>` and
 `--itn-keys <base64 ed25519 public keys>` serves a second GraphQL API, the
 ITN server (schema `Mina_graphql.schema_itn`, see
-[`schema/itn.graphql`](schema/itn.graphql)). Load testing tools use it to
+[`schema/itn.graphql`](https://github.com/o1-labs/mina-sdk-spec/blob/main/schema/itn.graphql)). Load testing tools use it to
 schedule payments and zkApp commands, read internal logs, change connection
 gating and stop the daemon.
 
-The Mina SDKs have the same ITN client. This document and
+The Rust, Go and JS SDKs have the same ITN client; `mina-sdk-python` has no
+ITN client yet (– in the Python column). This document and
 [`itn-operations.graphql`](itn-operations.graphql) define it. The rules of
 [`SPEC.md`](SPEC.md) apply: each method sends one named operation, and
 nullable variables are always sent, as null when the caller omits them.
@@ -43,21 +44,21 @@ A client must:
 
 ## Methods
 
-| Operation | Rust | Go | JS | Arguments | Returns |
-|:--|:--|:--|:--|:--|:--|
-| `Auth` | `auth` | `Auth` | `auth` | – | server UUID, sequence number, libp2p port, peer ID, block producer flag |
-| `SlotsWon` | `slots_won` | `SlotsWon` | `slotsWon` | – | list of global slots |
-| `InternalLogs` | `internal_logs` | `InternalLogs` | `internalLogs` | start log ID | list of logs (`id`, `timestamp`, `message`, `metadata` (`item`, `value`), `process?`) |
-| `FlushInternalLogs` | `flush_internal_logs` | `FlushInternalLogs` | `flushInternalLogs` | end log ID | string |
-| `SchedulePayments` | `schedule_payments` | `SchedulePayments` | `schedulePayments` | `PaymentsDetails` | handle |
-| `ScheduleZkappCommands` | `schedule_zkapp_commands` | `ScheduleZkappCommands` | `scheduleZkappCommands` | `ZkappCommandsDetails` | handle |
-| `StopScheduledTransactions` | `stop_scheduled_transactions` | `StopScheduledTransactions` | `stopScheduledTransactions` | handle | string |
-| `UpdateGating` | `update_gating` | `UpdateGating` | `updateGating` | `GatingUpdate` | string |
-| `StopDaemon` | `stop_daemon` | `StopDaemon` | `stopDaemon` | [delay seconds], [clean config] | string |
-| `ZkappCommandLimit` | `set_zkapp_command_limit` | `SetZkappCommandLimit` | `setZkappCommandLimit` | limit or null | limit now in force |
+| Operation | Rust | Go | JS | Python | Arguments | Returns |
+|:--|:--|:--|:--|:--|:--|:--|
+| `Auth` | `auth` | `Auth` | `auth` | – | – | server UUID, sequence number, libp2p port, peer ID, block producer flag |
+| `SlotsWon` | `slots_won` | `SlotsWon` | `slotsWon` | – | – | list of global slots |
+| `InternalLogs` | `internal_logs` | `InternalLogs` | `internalLogs` | – | start log ID | list of logs (`id`, `timestamp`, `message`, `metadata` (`item`, `value`), `process?`) |
+| `FlushInternalLogs` | `flush_internal_logs` | `FlushInternalLogs` | `flushInternalLogs` | – | end log ID | string |
+| `SchedulePayments` | `schedule_payments` | `SchedulePayments` | `schedulePayments` | – | `PaymentsDetails` | handle |
+| `ScheduleZkappCommands` | `schedule_zkapp_commands` | `ScheduleZkappCommands` | `scheduleZkappCommands` | – | `ZkappCommandsDetails` | handle |
+| `StopScheduledTransactions` | `stop_scheduled_transactions` | `StopScheduledTransactions` | `stopScheduledTransactions` | – | handle | string |
+| `UpdateGating` | `update_gating` | `UpdateGating` | `updateGating` | – | `GatingUpdate` | string |
+| `StopDaemon` | `stop_daemon` | `StopDaemon` | `stopDaemon` | – | [delay seconds], [clean config] | string |
+| `ZkappCommandLimit` | `set_zkapp_command_limit` | `SetZkappCommandLimit` | `setZkappCommandLimit` | – | limit or null | limit now in force |
 
 The input types `PaymentsDetails`, `ZkappCommandsDetails` and `GatingUpdate`
-have the fields of [`schema/itn.graphql`](schema/itn.graphql). Every SDK also
+have the fields of [`schema/itn.graphql`](https://github.com/o1-labs/mina-sdk-spec/blob/main/schema/itn.graphql). Every SDK also
 has a custom-query method that signs and sequences any document.
 
 ## Notes

spec/SPEC.md

@@ -1,6 +1,7 @@
 # Mina SDK common API
 
-The Mina SDKs (`mina-sdk-rust`, `mina-sdk-go`, `mina-sdk-js`) have the same
+The Mina SDKs (`mina-sdk-rust`, `mina-sdk-go`, `mina-sdk-js`,
+`mina-sdk-python`) have the same
 client API for the daemon's public GraphQL server (default port 3085). This
 document and [`operations.graphql`](operations.graphql) define it.
 [`ITN.md`](ITN.md) defines the client of the daemon's ITN server.
@@ -15,33 +16,34 @@ repository first, make a release, then update every SDK with
 
 Names follow each language's style. Arguments in brackets are optional.
 
-| Operation | Rust | Go | JS | Arguments | Returns |
-|:--|:--|:--|:--|:--|:--|
-| `SyncStatus` | `get_sync_status` | `GetSyncStatus` | `getSyncStatus` | – | `SyncStatus` |
-| `DaemonStatus` | `get_daemon_status` | `GetDaemonStatus` | `getDaemonStatus` | – | `DaemonStatus` |
-| `DaemonMetrics` | `get_daemon_metrics` | `GetDaemonMetrics` | `getDaemonMetrics` | – | `DaemonMetrics` |
-| `NetworkId` | `get_network_id` | `GetNetworkID` | `getNetworkId` | – | string |
-| `Account` | `get_account` | `GetAccount` | `getAccount` | public key, [token ID] | `AccountData` |
-| `BestChain` | `get_best_chain` | `GetBestChain` | `getBestChain` | [max length] | list of `BlockInfo` |
-| `GenesisBlock` | `get_genesis_block` | `GetGenesisBlock` | `getGenesisBlock` | – | `BlockInfo` |
-| `Block` | `get_block` | `GetBlock` | `getBlock` | state hash **or** height | `BlockInfo` |
-| `Peers` | `get_peers` | `GetPeers` | `getPeers` | – | list of `PeerInfo` |
-| `PooledUserCommands` | `get_pooled_user_commands` | `GetPooledUserCommands` | `getPooledUserCommands` | [public key] | list of `PooledUserCommand` |
-| `PooledZkappCommands` | `get_pooled_zkapp_commands` | `GetPooledZkappCommands` | `getPooledZkappCommands` | [public key] | list of `ZkappCommandResult` |
-| `TransactionStatus` | `get_transaction_status` | `GetTransactionStatus` | `getTransactionStatus` | payment ID **or** zkApp transaction ID | `TransactionStatus` |
-| `GenesisConstants` | `get_genesis_constants` | `GetGenesisConstants` | `getGenesisConstants` | – | `GenesisConstants` |
-| `TrackedAccounts` | `get_tracked_accounts` | `GetTrackedAccounts` | `getTrackedAccounts` | – | list of `TrackedAccount` |
-| `SnarkPool` | `get_snark_pool` | `GetSnarkPool` | `getSnarkPool` | – | list of `CompletedWork` |
-| `ForkConfig` | `get_fork_config` | `GetForkConfig` | `getForkConfig` | – | JSON value |
-| `SendPayment` | `send_payment` | `SendPayment` | `sendPayment` | payment, [signature] | `SubmittedCommand` |
-| `SendDelegation` | `send_delegation` | `SendDelegation` | `sendDelegation` | delegation, [signature] | `SubmittedCommand` |
-| `SendZkapp` | `send_zkapp` | `SendZkapp` | `sendZkapp` | zkApp command (JSON) | `ZkappCommandResult` |
-| `UnlockAccount` | `unlock_account` | `UnlockAccount` | `unlockAccount` | public key, password | public key |
-| `SetSnarkWorker` | `set_snark_worker` | `SetSnarkWorker` | `setSnarkWorker` | [public key] | previous worker or null |
-| `SetSnarkWorkFee` | `set_snark_work_fee` | `SetSnarkWorkFee` | `setSnarkWorkFee` | fee | previous fee |
-
-Every SDK also has a custom-query method (`execute_query`, `Request`,
-`executeQuery`) that uses the same transport, retries and errors.
+| Operation | Rust | Go | JS | Python | Arguments | Returns |
+|:--|:--|:--|:--|:--|:--|:--|
+| `SyncStatus` | `get_sync_status` | `GetSyncStatus` | `getSyncStatus` | `get_sync_status` | – | `SyncStatus` |
+| `DaemonStatus` | `get_daemon_status` | `GetDaemonStatus` | `getDaemonStatus` | `get_daemon_status` | – | `DaemonStatus` |
+| `DaemonMetrics` | `get_daemon_metrics` | `GetDaemonMetrics` | `getDaemonMetrics` | `get_daemon_metrics` | – | `DaemonMetrics` |
+| `NetworkId` | `get_network_id` | `GetNetworkID` | `getNetworkId` | `get_network_id` | – | string |
+| `Account` | `get_account` | `GetAccount` | `getAccount` | `get_account` | public key, [token ID] | `AccountData` |
+| `BestChain` | `get_best_chain` | `GetBestChain` | `getBestChain` | `get_best_chain` | [max length] | list of `BlockInfo` |
+| `GenesisBlock` | `get_genesis_block` | `GetGenesisBlock` | `getGenesisBlock` | `get_genesis_block` | – | `BlockInfo` |
+| `Block` | `get_block` | `GetBlock` | `getBlock` | `get_block` | state hash **or** height | `BlockInfo` |
+| `Peers` | `get_peers` | `GetPeers` | `getPeers` | `get_peers` | – | list of `PeerInfo` |
+| `PooledUserCommands` | `get_pooled_user_commands` | `GetPooledUserCommands` | `getPooledUserCommands` | `get_pooled_user_commands` | [public key] | list of `PooledUserCommand` |
+| `PooledZkappCommands` | `get_pooled_zkapp_commands` | `GetPooledZkappCommands` | `getPooledZkappCommands` | `get_pooled_zkapp_commands` | [public key] | list of `ZkappCommandResult` |
+| `TransactionStatus` | `get_transaction_status` | `GetTransactionStatus` | `getTransactionStatus` | `get_transaction_status` | payment ID **or** zkApp transaction ID | `TransactionStatus` |
+| `GenesisConstants` | `get_genesis_constants` | `GetGenesisConstants` | `getGenesisConstants` | `get_genesis_constants` | – | `GenesisConstants` |
+| `TrackedAccounts` | `get_tracked_accounts` | `GetTrackedAccounts` | `getTrackedAccounts` | `get_tracked_accounts` | – | list of `TrackedAccount` |
+| `SnarkPool` | `get_snark_pool` | `GetSnarkPool` | `getSnarkPool` | `get_snark_pool` | – | list of `CompletedWork` |
+| `ForkConfig` | `get_fork_config` | `GetForkConfig` | `getForkConfig` | `get_fork_config` | – | JSON value |
+| `SendPayment` | `send_payment` | `SendPayment` | `sendPayment` | `send_payment` | payment, [signature] | `SubmittedCommand` |
+| `SendDelegation` | `send_delegation` | `SendDelegation` | `sendDelegation` | `send_delegation` | delegation, [signature] | `SubmittedCommand` |
+| `SendZkapp` | `send_zkapp` | `SendZkapp` | `sendZkapp` | `send_zkapp` | zkApp command (JSON) | `ZkappCommandResult` |
+| `UnlockAccount` | `unlock_account` | `UnlockAccount` | `unlockAccount` | `unlock_account` | public key, password | public key |
+| `SetSnarkWorker` | `set_snark_worker` | `SetSnarkWorker` | `setSnarkWorker` | `set_snark_worker` | [public key] | previous worker or null |
+| `SetSnarkWorkFee` | `set_snark_work_fee` | `SetSnarkWorkFee` | `setSnarkWorkFee` | `set_snark_work_fee` | fee | previous fee |
+
+Every SDK also has a custom-query method (Rust and Python `execute_query`,
+Go `Request`, JS `executeQuery`) that uses the same transport, retries and
+errors.
 
 ## Rules
 
@@ -65,14 +67,18 @@ Every SDK also has a custom-query method (`execute_query`, `Request`,
 
 ## Result types
 
-Field names are shown in camelCase; Rust uses snake_case and Go PascalCase.
+Field names are shown in camelCase; Rust and Python use snake_case and Go
+PascalCase. Python names the daemon's `from` field `from_`, because `from`
+is a keyword.
 `?` marks a field that can be null or absent.
 
 The types are the union of what the SDKs returned before this API, so no
 caller loses a field. Where an SDK already had a field or a type under
 another name, it keeps that name (for example Go `CreatorPK` and Rust
 `creator_pk` for `creatorPublicKey`, and the JS type `Block` for the result
-of `getBlock`), and adds the fields of this document to it.
+of `getBlock`), and adds the fields of this document to it. Python's
+`PooledUserCommand` also answers the dictionary keys of its earlier
+dictionary result.
 
 **`DaemonStatus`**: `syncStatus`, `blockchainLength?`,
 `highestBlockLengthReceived?`, `highestUnvalidatedBlockLengthReceived?`,

spec/VERSION

@@ -1 +1 @@
-v0.1.0
+v0.1.1

src/mina_sdk/daemon/client.py

@@ -190,6 +190,18 @@ def _request(
             f"Failed to execute {query_name} after {self._retries} attempts: {last_error}"
         )
 
+    def execute_query(
+        self, query: str, variables: dict[str, Any] | None = None, query_name: str = "custom"
+    ) -> dict[str, Any]:
+        """Run a custom GraphQL document, with the client's transport, retries
+        and errors. Returns the ``data`` field of the response.
+
+        Raises:
+            GraphQLError: If the response contains GraphQL-level errors (not retried).
+            DaemonConnectionError: If all retry attempts fail due to network errors.
+        """
+        return self._request(query, variables=variables, query_name=query_name)
+
     # -- Queries --
 
     def get_sync_status(self) -> str:

src/mina_sdk/itn/__init__.py

@@ -0,0 +1,47 @@
+"""Client for the daemon's ITN GraphQL server (``ITN_FEATURES=1``,
+``--itn-graphql-port``, ``--itn-keys``), whose requests are signed with an
+ed25519 key. It needs the ``itn`` extra: ``pip install mina-sdk[itn]``.
+"""
+
+try:
+    import cryptography  # noqa: F401
+except ImportError as e:  # pragma: no cover
+    raise ImportError(
+        "mina_sdk.itn needs the 'cryptography' package: pip install 'mina-sdk[itn]'"
+    ) from e
+
+from mina_sdk.itn import queries
+from mina_sdk.itn.client import ItnClient
+from mina_sdk.itn.errors import (
+    InvalidItnKeyError,
+    ItnHttpError,
+    ItnSequencingError,
+    ItnUnauthorizedError,
+)
+from mina_sdk.itn.key import ItnKey
+from mina_sdk.itn.types import (
+    GatingUpdate,
+    ItnAuth,
+    ItnLog,
+    LogMetadatum,
+    NetworkPeer,
+    PaymentsDetails,
+    ZkappCommandsDetails,
+)
+
+__all__ = [
+    "GatingUpdate",
+    "InvalidItnKeyError",
+    "ItnAuth",
+    "ItnClient",
+    "ItnHttpError",
+    "ItnKey",
+    "ItnLog",
+    "ItnSequencingError",
+    "ItnUnauthorizedError",
+    "LogMetadatum",
+    "NetworkPeer",
+    "PaymentsDetails",
+    "ZkappCommandsDetails",
+    "queries",
+]