Skip to content

feat: add --no-blockchain mode for routing commits/identity updates to nodes#2004

Draft
api-Hypernova wants to merge 11 commits into
mainfrom
hypernova/no-blockchain-experiment
Draft

feat: add --no-blockchain mode for routing commits/identity updates to nodes#2004
api-Hypernova wants to merge 11 commits into
mainfrom
hypernova/no-blockchain-experiment

Conversation

@api-Hypernova
Copy link
Copy Markdown
Contributor

@api-Hypernova api-Hypernova commented May 14, 2026

Summary

  • Adds --no-blockchain config flags to payer and API services
  • Routes commits and identity updates to dedicated nodes instead of Arbitrum L2 blockchain
  • Reduces commit latency from 2-15s (blockchain) to ~13ms (node) — 150-1000x faster
  • All existing tests pass, no behavioral changes when flags are disabled

Code Changes

File Change
pkg/config/options.go NoBlockchain, CommitNodeID, IdentityNodeID flags
pkg/api/payer/service.go Route blockchain-bound messages to dedicated nodes
pkg/api/payer/config.go WithNoBlockchain option
pkg/api/message/service.go Conditional rejection points for commits/identity updates
pkg/gateway/builder.go Wire no-blockchain config to payer service
pkg/testutils/api/api.go WithTestNoBlockchain() test option

Test Coverage

  • Unit tests: identity update routing, commit routing, regular message unaffected
  • Failover tests: commit/identity node down → error (no silent drop), mixed batch atomic failure
  • E2E tests: full payer→node commit flow, identity update flow, mixed traffic, multi-group ordering
  • Benchmark: 50-iteration latency comparison (~13ms both paths)
  • Load test: 547 write RPS, 7224 read RPS, P99 <28ms

Performance Results

Path Avg Latency P99
No-blockchain commit (via node) ~14ms ~28ms
Regular message (baseline) ~14ms ~28ms
Blockchain commit (production) 2,000-15,000ms

Test plan

  • All existing payer tests pass
  • All new no-blockchain tests pass
  • Full go test ./... passes (51 packages)
  • go vet clean on all changed packages
  • Live 3-node cluster tested with failover
  • CI passes

Generated with Claude Code

Note

Add --no-blockchain mode to route commits and identity updates directly to nodes

  • Adds a NoBlockchain experiment flag to both the payer and API services, with CommitNodeID and IdentityNodeID config fields specifying which nodes receive those message types instead of the blockchain.
  • In payer/service.go, groupEnvelopes routes identity updates to IdentityNodeID and commit/proposal messages to CommitNodeID when no-blockchain mode is active; determineRetentionPolicy now handles these types without panicking.
  • In api/message/service.go, the API accepts commits, proposals, and identity updates when NoBlockchain is enabled, and allows DependsOn node IDs ≥ 100.
  • Adds a dev/run-no-chain script and a dev/run-3 script, plus a triple Docker Compose profile with a third Postgres on port 8767, for local multi-node testing.
  • Covers the new routing paths with unit, failover, E2E, and latency tests in pkg/api/payer/.

Macroscope summarized 01347dd. (Automatic summaries will resume when PR exits draft mode or review begins).

Ubuntu and others added 7 commits May 14, 2026 18:25
…ates to nodes

Adds feature flags (XMTPD_PAYER_NO_BLOCKCHAIN, XMTPD_API_NO_BLOCKCHAIN) that
bypass blockchain for commit and identity update messages, routing them to
dedicated nodes instead. This enables testing XMTPD without Arbitrum L2
dependency as proposed in the "xmtpd without chain" experiment.

Changes:
- Add NoBlockchain, CommitNodeID, IdentityNodeID to PayerOptions and APIOptions
- Modify payer groupEnvelopes() to route blockchain-bound messages to nodes
- Make 3 node-side rejection points conditional on NoBlockchain flag
- Wire config through gateway builder with WithNoBlockchain option
- Add triple profile to docker-compose with 3rd postgres (port 8767)
- Add dev/run-3 and dev/run-no-chain convenience scripts
- Add 3rd Anvil account to local.env and node registration in dev/docker/up

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The determineRetentionPolicy function panicked for identity updates and
commit messages because they were never expected to go through the node
publish path. In no-blockchain mode they do, so return default retention
instead of panicking.

Adds unit test verifying identity updates route to node (not blockchain)
when WithNoBlockchain is enabled.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three tests verify the no-blockchain experiment:
1. Identity updates route to dedicated node (not blockchain publisher)
2. Commit messages route to dedicated node and are stored successfully
3. Regular messages still use normal node selector path

Also adds WithTestNoBlockchain() option to test API server configuration
to enable NoBlockchain on the message service for test scenarios.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Measures 50 iterations of commit publish via no-blockchain node path
vs regular message publish. Results show identical latency (~13ms),
confirming the blockchain elimination works. Production blockchain
path adds 2-15s per commit (Arbitrum L2 finality).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Verifies that unreachable commit/identity nodes produce errors (not silent
drops), and that mixed batches fail atomically when one target is down.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- GatewayE2E_CommitThroughPayer: 5 sequential commits through payer → node
- GatewayE2E_IdentityUpdateThroughPayer: identity update through dedicated node
- MultiGroupOrdering: 10 groups verify monotonic sequence IDs at commit node
- MixedWorkload: regular message + commit + identity update + welcome in sequence
- SequentialCommitLatency: 50-iteration latency benchmark (avg=13.8ms)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Commit flow: payer → dedicated commit node → stored successfully
- Identity update flow: payer → dedicated identity node
- Mixed traffic: regular messages + commits routed to correct destinations
- Multi-group ordering: 10 groups, all commits → single commit node

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@octane-security-app
Copy link
Copy Markdown

Summary by Octane

New Contracts

No new contracts were added.

Updated Contracts

  • service.go: Introduced a NoBlockchain option to bypass blockchain checks, affecting identity updates, message validation, and client info processing.

  • config.go: Added no-blockchain option for routing commits and identity updates to specific nodes.

  • service.go: In no-blockchain mode, route identity updates and group messages to specific nodes instead of the blockchain.

  • options.go: The smart contract now allows commits and identity updates directly from nodes, bypassing blockchain, with dedicated node routing.

  • builder.go: Added payer options for publish timeout, retries, and a no-blockchain mode with logging capabilities.

  • api.go: Added a feature to disable blockchain integration via a new configuration option in the API server test setup.


🔗 Commit Hash: 01347dd

Ubuntu and others added 4 commits May 14, 2026 19:39
- Break long AssertNotCalled/EqualValues lines for golines
- Remove extra blank line after closing paren for gofumpt
- Use integer range loops (Go 1.22+) for intrange

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Replace EqualValues with Equal where types match
- Keep EqualValues where type coercion needed (int vs uint32)
- Use range-over-int for iteration loops
- Break long lines for golines

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The publishToNodeWithRetry path calls GetNodes for banlist retry,
which wasn't mocked in TestNoBlockchain_IdentityUpdateRoutedToNode.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@octane-security-app
Copy link
Copy Markdown

Overview

Vulnerabilities found: 20                                                                                
Severity breakdown: 1 Critical, 1 High, 4 Medium, 9 Low, 5 Informational
Warnings found: 6                                                                                

Detailed findings

pkg/api/message/service.go

  • No-blockchain reroute of identity/commit messages in payer/message services causes unintended fee accrual to the gateway payer. See more
  • Unsigned message_retention_days in PayerEnvelope under NoBlockchain routing causes misbilling and incorrect expiry. See more
  • FIFO publish worker without poison-message handling in NoBlockchain staging path causes node-level stall of commit/identity streams. See more
  • Bypassed identifier-size checks in Payer/Node no-blockchain mode causes acceptance and storage of non-canonical group/inbox topics. See more
  • NoBlockchain feature-flag acceptance split without capability negotiation in Replication PublishPayerEnvelopes causes mixed-cluster publish rejections and inconsistent behavior. See more
  • TTL-cached originator enumeration in topic catch-up (no-blockchain mode) causes incomplete topic history. See more
  • Replica-stale reads in getAvailableBalance under NoBlockchain admission cause incorrect payer balance enforcement. See more
  • Relaxed commit/proposal and identity-update validation in NoBlockchain mode in Replication API causes acceptance and serving of unsequenced envelopes. See more

pkg/api/payer/service.go

  • No-blockchain dedicated-node routing without isolation/prioritization in Payer/API causes network-wide control-plane DoS. See more
  • Non-atomic multi-node fan-out with non-idempotent replication in NoBlockchain mode causes duplicated envelopes and payer overcharge. See more
  • Missing size guard for rerouted blockchain-bound envelopes in Payer API no-blockchain mode causes large off-chain payload injection and DB/replication load. See more
  • Retry retargeting of dedicated nodes in payer no-blockchain mode causes cross-originator duplicates and double unsettled usage. See more
  • Missing response length/content validation in Payer PublishClientEnvelopes (NoBlockchain mode) causes node crash/DoS. See more

pkg/config/validation.go

  • Lack of coordination between NoBlockchain routing and chain indexer in mixed deployments causes view-level divergence. See more
  • Unconditional App Chain validation in startup configuration for API no-blockchain mode causes API node startup failure. See more

pkg/gateway/builder.go

  • Unconditional app-chain publisher/oracle initialization in gateway builder with no-blockchain mode causes gateway startup failure. See more

pkg/registrant/registrant.go

  • Finite retention in no-blockchain mode (payer/API) causes pruning of commit/identity histories and downstream validation failures. See more

pkg/server/server.go

  • No-blockchain flag relaxation in API message service and gateway routing causes unauthenticated off-chain ingestion of commits/proposals and identity updates and bypasses blockchain payload-size guard. See more
  • NoBlockchain off-chain admission without default balance gating in message publishing causes cheap spam/DoS risk. See more

pkg/sync/originator_stream.go

  • Missing NoBlockchain/topic filtering in replication ingest causes blockchain-only nodes to serve no-blockchain messages. See more

Warnings

pkg/api/message/service.go

  • Gap-tolerant replication with oversized-envelope skip in NoBlockchain mode causes DependsOn acceptance without prior causal envelopes. See more
  • Originator-grouped ordering in topic catch-up queries under no-blockchain mixed-originator identity topics causes incorrect client identity state reconstruction. See more

pkg/api/payer/service.go

  • Missing verification of remote OriginatorEnvelope receipts in Payer API (no-blockchain mode) causes misleading client receipts. See more

pkg/gateway/builder.go

  • Missing validation of CommitNodeID/IdentityNodeID in gateway no-blockchain configuration causes availability loss or unintended routing of commit/identity traffic. See more
  • Unused api.no-blockchain flag in gateway config parsing causes silent continued on-chain publishing. See more

pkg/server/server.go

  • Relaxed guards under NoBlockchain in ReplicationApi.PublishPayerEnvelopes allow unauthenticated off-chain identity/commit/proposal injection causing storage/processing overhead. See more

🔗 Commit Hash: 01347dd
🛡️ Octane Dashboard: All vulnerabilities

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant