network: a TransactionListQuery that resolves to zero transactions is never answered, blocking sync with that peer for 30 seconds

[stevenvegt]

Summary

When a peer answers a TransactionListQuery and none of the requested references resolve to a transaction it has, it sends no message at all. The requester's conversation is blocking, so it cannot send another TransactionListQuery or TransactionRangeQuery to that peer until the conversation expires 30 seconds later. Sync with that peer stalls for the full 30 seconds.

Observed on the e2e test nuts-network/private-transactions in https://github.com/nuts-foundation/nuts-node/actions/runs/35200908093/job/105135215724, which failed with:

Waiting for service 'NodeA' to contain 6 transactions..........FAILED: Service 'NodeA' did not get 6 transaction within 10 seconds

What happens

handleTransactionListQuery looks up every requested reference. A reference it does not have is logged and skipped (network/transport/v2/handlers.go:385-394):

if errors.Is(err, dag.ErrTransactionNotFound) {
    // TODO: Should the entire ListQuery be aborted?
    log.Logger().
        WithFields(connection.Peer().ToFields()).
        WithField(core.LogFieldTransactionRef, ref.String()).
        Warn("Peer requested transaction we don't have")
    continue
}

If every reference is skipped, the handler calls sendTransactionList with an empty slice. chunkTransactionList returns zero chunks for an empty slice, because its trailing-chunk guard is if startIndex != len(transactions), so the send loop in sendTransactionList never runs and no TransactionList message is put on the wire (network/transport/v2/senders.go:93-111).

A partial result is fine: if at least one reference resolves, a list is sent, handleTransactionList runs on the requester and calls p.cMan.done(cid). Only the all-missing case is silent.

Why the requester is stuck

p.cMan.done(cid) is reachable only from transactionlist_handler.go:108 and :122, both on receipt of a TransactionList. With no message, nothing clears the conversation.

Envelope_TransactionListQuery and Envelope_TransactionRangeQuery both implement blockingConversation() (conversation.go:227,255) and share one slot per peer through lastPeerConversationID, so while the stranded conversation is live the requester logs "Did not request a TransactionList while another conversation is in progress" (senders.go:81) and cannot request a transaction range either. State queries are not blockable and keep going, which is why the node looks healthy in the logs while making no progress.

The block lasts exactly maxValidity, 30 seconds (conversation.go:35). The eviction ticker runs at the same interval but only cleans up the map; hasActiveConversation gates on conversation.expiry.After(time.Now()), so the gate opens at 30 seconds regardless of when eviction runs.

Observed sequence

From the failed run, nodeA and nodeB each created a private credential transaction within 60ms of each other. nodeA decoded nodeB's IBLT, asked for one transaction, and nodeB did not have it yet:

nodeA-1 | msg="Peer IBLT decode succesful, requesting 1 transactions"
nodeA-1 | msg="Requesting transactionList from peer (1 transactions)" conversationID=0de0f84d-58d6-41f5-a8f0-fac9401617a0
nodeB-1 | level=warning msg="Peer requested transaction we don't have" txRef=e624601e767fecabf3db5768469057ec229c9f97717fb0ae5bb4c5294a6f56e5
nodeB-1 | level=info msg="Transaction created" txRef=e624601e767fecabf3db5768469057ec229c9f97717fb0ae5bb4c5294a6f56e5
nodeA-1 | msg="Did not request a TransactionList while another conversation is in progress"   (repeated for the rest of the run)

Note the order of the last two nodeB lines: nodeB advertised the transaction before it could read it back. That is a separate defect, tracked in #4560. This issue is about the silent drop, which strands the requester whatever the cause of the empty result: a transaction not yet committed, a pruned transaction, or a reference produced by an IBLT decode against a state the peer no longer has.

Impact

Sync with one peer stalls for 30 seconds, then recovers by itself on the next state exchange. No data loss and no corruption. On a node with several peers the missing transactions arrive from another peer in the meantime, so the practical impact is small. On a two-node setup it is a 30 second delay on every occurrence.

For CI it means a hard failure: waitForTXCount allows 10 seconds, which can never outlast a 30 second block. Issue #4321 reports the same symptom shape on the gossip test but attributes it to slow propagation under load; this is a different mechanism.

Suggested fix

Answer the query in every case. Sending an empty TransactionList for the conversation would let handleTransactionList close it normally, which needs either a guard in sendTransactionList for the zero-chunk case or a fix in chunkTransactionList so an empty input yields one empty chunk. Worth checking what a peer on an older version does with an empty list first.

Answering the // TODO: Should the entire ListQuery be aborted? in the same pass would be sensible: aborting and telling the requester is also an answer, and is arguably the more honest one when nothing could be resolved.

Independently, hasActiveConversation blocking on the last conversation per peer means any unanswered blocking query costs 30 seconds of sync with that peer. A shorter validity for query conversations, or clearing the block when the connection produces other traffic, would limit the damage from future cases of this shape.