Skip to content

Optional deferred acknowledgment and a bounded response wait in the RPC call request node - #36

Open
ShvaykaD wants to merge 14 commits into
rpc-tests-inflight-recoveryfrom
rpc-tests-inflight-recovery-and-rpc-node-fix
Open

Optional deferred acknowledgment and a bounded response wait in the RPC call request node#36
ShvaykaD wants to merge 14 commits into
rpc-tests-inflight-recoveryfrom
rpc-tests-inflight-recovery-and-rpc-node-fix

Conversation

@ShvaykaD

@ShvaykaD ShvaykaD commented Jul 27, 2026

Copy link
Copy Markdown
Owner

Problem

TbSendRPCRequestNode acknowledges the incoming message right after dispatching the RPC, so the queue offset commits before the outcome is known while the pending state lives only in per-pod memory. A rule engine restart mid-flight therefore loses the RPC silently — it was counted as a success — and a timeout is reported as a new message on the node's Failure output, which queue retries reprocess as an error wrapper without ever re-executing onMsg. There was also no way to bound the rule engine's wait: it is always derived from the RPC expiration time, which for REST-initiated RPC comes from the request body with a 5 s floor and no maximum.

Goal

Let an operator make server-side RPC through the rule engine recoverable and bound how long it waits, with existing rule chains behaving exactly as before.

Approach

Two opt-in options, both defaulting to today's behavior:

  • forceAck (default true)false holds the incoming message and completes it on the outcome (tellSuccess / tellFailure on the original message) instead of acknowledging at dispatch. The original request becomes the completion unit, so it stays uncommitted: a restart mid-flight redelivers and re-executes it, and a timeout becomes a retryable failure of the real request. Downstream payloads are identical in both modes.
  • overrideResponseTimeout (default false) — makes the node's timeoutInSeconds bound the wait. The node sends an absolute ruleEngineResponseDeadline (expirationTime when off, now + timeoutInSeconds when on); scheduleTimeout keeps its original arithmetic and only reads that deadline, so the default path is unchanged and no sentinel logic lives in the service.
  • Validationinit() rejects a negative timeoutInSeconds unrecoverably, which removes the need for any runtime guard. 0 stays valid, since the UI has always allowed it.
  • Migrationversion = 1 plus upgrade(), and a single atomic UPDATE rule_node appended to the LTS 4.3.1.4 patch, committed together with the schema version so the new code never sees an un-migrated configuration.
  • UI — two toggles and a hint on the existing timeout field.

No proto / wire / device / transport change, and no change to core-side RPC persistence.

With forceAck=false, reprocessing means at-least-once delivery — the same command can reach the device twice — a real re-attempt requires a queue whose pack timeout and processing strategy allow it, and the node emits Success/Failure debug entries instead of ACK. A configured cap gets the same +1s grace as the expiration-derived wait and no longer clamps to the remaining expiration.

Screenshot from 2026-07-27 19-42-52 Screenshot from 2026-07-27 19-40-34

Tests

TbSendRPCRequestNodeTest covers both acknowledgment modes (the two pre-existing response tests are the untouched legacy guard), the deadline with the override on and off including timeoutInSeconds = 0, the init() validation, and upgrade() from version 0. DefaultTbRuleEngineRpcServiceTest is unchanged — with the deadline computed in the node, the service has no branch left to test.

ShvaykaD added 13 commits July 27, 2026 16:15
… migration, fall back to expiration time for the response deadline
…ESCE, parameterized deadline test, clearer nodeDetails order
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant