fix(runtime): discard stale inference after watchdog reset and warmup timeout

[ShubhJain09]

Summary

• _force_reset in AsyncExecution now bumps _incarnation so a stuck inference that finally returns after the watchdog fires gets thrown away instead of landing in the action queue
• warmup in RTCExecution does the same when it times out and also clears the pending observation slot
• Moved the hardcoded 120s warmup timeout into a _WARMUP_TIMEOUT_S constant so the test can shorten it
• Added two regression tests in test_execution that cover both paths

Why

• When inference hangs longer than watchdog_timeout_s the watchdog clears the busy state but never invalidates the call that is still running
• Once that call returns it passes the incarnation check and gets pushed with an offset close to zero because _pops_at_request was already overwritten by the next request
• On real hardware the robot jumps back and replays actions predicted from an observation that is 30 seconds or more old and the only trace is the reset warning in the logs
• The RTC warmup timeout had the same gap where warmup reports failure but the late chunk still ends up in the queue

Validation

• Both new tests fail on main with 4 and 20 stale actions reaching the queue and pass with this change
• pytest tests/unit packages/*/tests gives 2275 passed and 0 failed
• prek run --all-files is clean including ruff and pyrefly
• Ran the fuzz harnesses locally and the action queue one is clean while the stats normalizer crash also happens on main and is not related to this change

Breaking changes

• None

Related issues

• None

[Copilot]

Copilot review overview

🟡 Changes recommended

RTC timeout handling retains a result-acceptance race, and its regression test uses nondeterministic synchronization.

Review effort: Balanced
Findings: 2

Open (2)

• Recheck completion before invalidating after timeout · New
• Synchronize test with worker result processing · New

What changed in this PR

This PR prevents stale inference results from reaching robot action queues after watchdog or warmup timeouts.

Changes:

• Invalidates stuck asynchronous inference results during watchdog resets.
• Invalidates and clears timed-out RTC warmup requests.
• Adds regression tests for both stale-result paths.

File	Description
async_execution.py	Invalidates watchdog-reset inference results.
rtc.py	Adds configurable warmup timeout and stale-result invalidation.
test_execution.py	Tests watchdog and warmup timeout behavior.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[ShubhJain09]

Good catch fixed in bcccaee by rechecking the signal under the lock and added a test that forces this exact interleaving

[ShubhJain09]

Switched to waiting on the worker finishing _accept_result instead of a sleep so the test is deterministic now

[AlexanderBarabanov]

Note fore reviewer: please ignore fuzzing failure, this is not related with this PR and will be fixed separately. @openvinotoolkit/physicalai-maintain

[ShubhJain09]

@AlexanderBarabanov thanks for approving the CI run and for the note on the fuzz failure
I hit the same stats normalizer crash locally and traced it to a float32 overflow in the quantiles and min max modes
I have a fix with a regression test ready and can open a separate PR if that is useful

[AlexanderBarabanov]

@ShubhJain09 Thanks for investigating this -- the fix and regression test will be very valuable. Please go ahead and open a separate PR. Thanks!

[ShubhJain09]

@AlexanderBarabanov thank you for the suggestion
I have opened #327 with the fix and regression tests covering all three normalization modes
Please let me know if you would like any changes