fix(tests): stop the NetworkState height check racing block production

[dkijania]

Problem

The Lightnet test "Fetched max block height from archive node should match with the one from mina node" (tests/resolvers.test.ts, describe('NetworkState')) fails at random.

The test read two values, one after the other:

1. The archive's pendingMaxBlockHeight (the highest block in the archive database).
2. The daemon's blockchainLength, through fetchNetworkState.

If Lightnet produces a block between step 1 and step 2, or the archive has not ingested the newest block yet, the daemon value is one more than the archive value. The test uses strict equality, so it fails. The archive did nothing wrong.

Evidence

Run	PR	Job	Result
37059557939	#238	Run-Tests	expected 7, actual 6
37587269827	#243	Run-Tests (24)	failure
37591733132	#243	Run-Tests (24)	failure
37602137672	#243	Run-Tests (24)	expected 10, actual 9

In each case the daemon is exactly one block ahead. Since #235, Lightnet runs two times per PR (Node 22 and Node 24), so the race has two chances to occur in each CI run.

Fix

The test now reads the daemon tip with fetchLastBlock from o1js. This is a plain bestChain read: it sends no transaction, so it does not wait for a block.

sampleStableHeights reads three values, in this order:

1. daemonBefore: the daemon blockchainLength.
2. archive: the archive pendingMaxBlockHeight.
3. daemonAfter: the daemon blockchainLength again.

A sample is accepted only when no block arrived during the sample (daemonBefore === daemonAfter) and the archive has caught up (archive >= daemonBefore). Otherwise the helper waits 1 s and tries again, a maximum of 30 times.

assertArchiveMatchesDaemon then asserts exact equality: daemonBefore === daemonAfter and archive === daemonBefore. The failure message gives all three values.

The retry condition is "caught up", not "equal". Thus a wrong height gives a clear assertion diff. It does not cause retries until the test passes by chance.

The "Advance a block" test had the same race. It gets the same fix. fetchNetworkState is no longer imported by this test.

Why the test still finds a wrong height

• Too high (+1): the first stable sample has archive = daemonBefore + 1, so the equality fails.
• Too low (-1): a sample never becomes both stable and caught up. When the retries stop, the equality fails.
• Block during the sample, or archive ingestion lag: the sample is retried. The test does not fail.

Verification

• npm run build, tsc --noEmit, npm run lint and npm run test:unit are clean.
• Local Lightnet (o1labs/mina-local-network:compatible-latest-lightnet, single-node, PROOF_LEVEL=none), full resolvers.test.js:
  • As is: 33 pass, 0 fail.
  • pendingMaxBlockHeight - 1: both NetworkState height tests fail (daemon before 40, archive 39, daemon after 40).
  • pendingMaxBlockHeight + 1: both NetworkState height tests fail (daemon before 54, archive 55, daemon after 54).
• The diff changes only tests/resolvers.test.ts. Prettier wants to reformat other lines in that file, which were not clean before. I did not include those changes.

🤖 Generated with Claude Code

[SanabriaRusso]

Review: changes requested

Scope and compatibility. This is test-only: the diff touches only tests/resolvers.test.ts and nothing under src/, so o1js, mina-explorer and the SDKs are unaffected. CI is green on Node 22 and 24. Locally, npm run build, npm run test:unit and tsc --noEmit are clean.

1. The new assertion passes when pendingMaxBlockHeight is one block stale

Description. fetchNetworkState (zkapp/utils.ts:243) reads Mina.getNetworkState().blockchainLength inside Mina.transaction(). That value comes from o1js fetchLastBlock during the transaction's fetch phase, so it is the daemon tip a few ms after archiveBefore.

It then calls sendTransaction, which awaits pendingTx.wait() until the transaction is in a block. That needs at least one new block. This PR's CI log shows the timing:

• Fetching network state. at 12:59:22.54
• transaction sent at 12:59:22.58
• the suite resumes at 12:59:42.6

So by the time the polling loop runs, the archive is already at daemon + 1 or higher. daemon <= archiveAfter holds on the first iteration, even if the resolver returns trueHeight - 1:

resolver returns	archiveBefore	daemon	archiveAfter	this PR	old strict equality
correct	10	10	11	PASS	PASS
true − 1 (stale)	9	10	10	PASS	FAIL
true + 1	11	10	12	FAIL	FAIL

Consequences.

• The test exists to catch a one-block-stale or off-by-one-low networkState, for example a regression in the chain_status filter or in the canonical/pending fallback in NetworkService. With this PR it no longer can.
• The polling loop is effectively dead code.
• The original 9 !== 10 flake is mostly archive ingestion lag: the daemon read lands milliseconds after the archive read, not seconds later.

Reproduction. On Lightnet:

1. Temporarily change src/services/network-service/network-service.ts to

   const pendingMaxBlockHeight = (pendingRow ? Number(pendingRow.height) : canonicalMaxBlockHeight) - 1;

2. Run npm run test.

With this PR, both NetworkState height tests pass. On main, they fail.

2. Solution

Sample the daemon tip with a plain fetchLastBlock(), with no transaction and so no wait for inclusion:

1. Read the daemon, then the archive, then the daemon again.
2. Retry, bounded, until no block landed inside that window and the archive has caught up.
3. Then assert exact equality.

The retry condition is "caught up" (>=), not "equal". A wrong value therefore ends in a clean assertion diff instead of retrying until it happens to pass.

@@ imports
   UInt64,
+  fetchLastBlock,
 } from 'o1js';
@@
   emitMultipleFieldsEvents,
-  fetchNetworkState,
   randomStruct,
@@ replace sampleHeightsAroundDaemon / assertArchiveTracksDaemon with:
+  type HeightSample = {
+    results: NetworkQueryResult;
+    daemonBefore: number;
+    archive: number;
+    daemonAfter: number;
+  };
+
+  async function daemonHeight(): Promise<number> {
+    // A plain bestChain read: no transaction, so no wait for inclusion.
+    return Number((await fetchLastBlock()).blockchainLength.toString());
+  }
+
+  /**
+   * Read the daemon tip, the archive's pendingMaxBlockHeight, then the daemon
+   * tip again. A sample counts only when no block landed inside it
+   * (daemonBefore === daemonAfter) and the archive has ingested that tip
+   * (archive >= daemonBefore). Retries absorb archive ingestion lag and a block
+   * produced mid-sample; the retry predicate is "caught up", not "equal", so a
+   * wrong height still ends in a clean assertion diff, not a silent pass.
+   */
+  async function sampleStableHeights({
+    attempts = 30,
+    delayMs = 1000,
+  } = {}): Promise<HeightSample> {
+    let sample!: HeightSample;
+    for (let attempt = 1; attempt <= attempts; attempt++) {
+      const daemonBefore = await daemonHeight();
+      const results = await executeNetworkStateQuery();
+      const archive =
+        results.data.networkState.maxBlockHeight!.pendingMaxBlockHeight;
+      const daemonAfter = await daemonHeight();
+      sample = { results, daemonBefore, archive, daemonAfter };
+      if (daemonBefore === daemonAfter && archive >= daemonBefore) break;
+      if (attempt < attempts)
+        await new Promise((resolve) => setTimeout(resolve, delayMs));
+    }
+    return sample;
+  }
+
+  function assertArchiveMatchesDaemon({
+    daemonBefore,
+    archive,
+    daemonAfter,
+  }: HeightSample) {
+    const values = `daemon before ${daemonBefore}, archive ${archive}, daemon after ${daemonAfter}`;
+    assert.strictEqual(
+      daemonBefore,
+      daemonAfter,
+      `no block-free sampling window within the retry budget: ${values}`
+    );
+    assert.strictEqual(
+      archive,
+      daemonBefore,
+      `archive pendingMaxBlockHeight must equal the daemon blockchainLength: ${values}`
+    );
+  }
@@ describe('NetworkState')
-    let heights: { archiveBefore: number; daemon: number; archiveAfter: number };
+    let heights: HeightSample;
     before(async () => {
-      const sample = await sampleHeightsAroundDaemon(zkApp, senderKeypair);
-      results = sample.results;
+      heights = await sampleStableHeights();
+      results = heights.results;
       blockResponse = results.data.networkState;
-      heights = sample;
     });
     ...
-      assertArchiveTracksDaemon(heights);
+      assertArchiveMatchesDaemon(heights);
     // same two substitutions in the 'Advance a block' before() and test

Why this closes both races:

• A block lands mid-sample: daemonBefore !== daemonAfter, so the sample is retried.
• The archive lags: archive < daemonBefore, so the sample is retried.
• The resolver reports one too high: the first stable sample has archive = daemonBefore + 1, and strictEqual fails.
• The resolver reports one too low: a sample never becomes both stable and caught up. The archive only reports the next block after the daemon already has it, which makes daemonAfter > daemonBefore. The test fails after about 30 s, and the message shows all three values.
• Bounded: at most 30 × 1 s plus three reads per attempt. A Lightnet slot is about 20 s, so a stable window normally appears on the first or second try.
• Faster: it also removes a 20–40 s wait for transaction inclusion from each NetworkState before().

3. Testing methodology and acceptance criteria

1. npm run build, npx tsc --noEmit and npm run test:unit are clean. I checked this locally with the diff applied.
2. CI Run-Tests (22) and Run-Tests (24) pass. Re-run them at least 3 times, with no NetworkState failure.
3. Negative check (required). On Lightnet, apply the - 1 mutation from the reproduction. Both "Fetched max block height … should match" tests must fail. Repeat with + 1; both must fail again. Then revert the mutation. Post the failing output in the PR.
4. The failure message always prints daemon before, archive and daemon after.

[dkijania]

Thank you. Your finding is correct: fetchNetworkState waits until its transaction is in a block, so the archive was already at daemon + 1, and a resolver that was one block low passed.

Your solution is in 19998a9. Before it, I merged main into the branch, because the branch was behind. I did not rebase or force-push.

What changed

• daemonHeight() reads the daemon tip with fetchLastBlock() from o1js 2.0.0. It sends no transaction. I checked that it uses the endpoint that Mina.Network sets (networkConfig.minaEndpoint), and that it returns blockchainLength as a UInt32.
• sampleStableHeights() reads the daemon, then the archive, then the daemon again. It tries again (30 × 1 s) until daemonBefore === daemonAfter and archive >= daemonBefore.
• assertArchiveMatchesDaemon() asserts exact equality with strictEqual. The message gives daemon before, archive and daemon after.
• Both NetworkState before() hooks and both height tests use the new helpers. I removed sampleHeightsAroundDaemon, assertArchiveTracksDaemon and the fetchNetworkState import. fetchNetworkState stays in zkapp/utils.ts.
• The PR description now describes this method. It becomes the squash commit message.

Acceptance criteria

1. ✅ npm run build, npx tsc --noEmit, npm run lint and npm run test:unit are clean.
2. ✅ CI run 37784383105, three attempts. Run-Tests (22) and Run-Tests (24) passed in all three, so 6 of 6 Lightnet legs passed, with no NetworkState failure.
3. ✅ Negative check on a local Lightnet (o1labs/mina-local-network:compatible-latest-lightnet, single-node, PROOF_LEVEL=none), full resolvers.test.js:
   • As is: 33 pass, 0 fail.
   • pendingMaxBlockHeight - 1: both height tests fail. The other 29 tests pass.

     ✖ Fetched max block height from archive node should match with the one from mina node
       AssertionError [ERR_ASSERTION]: archive pendingMaxBlockHeight must equal the daemon blockchainLength: daemon before 40, archive 39, daemon after 40
     ✖ Fetched max block height from archive node should match the one from mina node after one block
       AssertionError [ERR_ASSERTION]: archive pendingMaxBlockHeight must equal the daemon blockchainLength: daemon before 41, archive 40, daemon after 41
     

   • pendingMaxBlockHeight + 1: both height tests fail. The other 29 tests pass.

     ✖ Fetched max block height from archive node should match with the one from mina node
       AssertionError [ERR_ASSERTION]: archive pendingMaxBlockHeight must equal the daemon blockchainLength: daemon before 54, archive 55, daemon after 54
     ✖ Fetched max block height from archive node should match the one from mina node after one block
       AssertionError [ERR_ASSERTION]: archive pendingMaxBlockHeight must equal the daemon blockchainLength: daemon before 55, archive 56, daemon after 55
     

   • I reverted the mutation after each run. No mutation is committed.
4. ✅ Each failure message above gives daemon before, archive and daemon after.

One note about the method: with --test-name-pattern, Node did not run the nested Advance a block suite. Thus each negative check above is a full, unfiltered run of resolvers.test.js.

🤖 Generated with Claude Code