fix(eval): hard rollout wall-clock timeout for HUDRuntime training (HUD-2099)

[ChinmayK0607]

Summary

• Bound local rollouts with a hard wall-clock deadline for provision + agent, and a short teardown grace for grade/cancel, so a wedged tunnel cannot stall Taskset.run.
• Wire HUDRuntime(run_timeout=…) into rollout_timeout; HostedRuntime timeouts return a terminal failed Run (no raise) so batch gather cannot collapse.
• Document TrainingClient / GRPO policy: keep timed-out members in the batch so group cardinality stays intact.

Fixes HUD-2099.

Test plan

• test_agent_loop_timeout_grades_the_partial_trajectory
• test_wedged_grade_cannot_stall_the_rollout
• test_run_timeout_requests_platform_cancel — Hosted returns failed Run + cancel
• test_hud_runtime_drives_local_rollout / test_taskset_uses_hud_runtime_run_timeout
• test_timed_out_runs_keep_grpo_group_full
• Full rollout + hosted + train group suites (36 passed)

Made with Cursor

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit e398638. Configure here.}

[cursor]

Cancel after grade timeout is inert

Low Severity

After grade hits the shared deadline-plus-grace via _await_teardown, the follow-up cancel recomputes remaining time against that same budget and gets roughly zero. Because HudClient.grade only times out through that wrapper (not an internal TimeoutError), the cancel path never runs with a usable timeout, so the intended post-grade abort does not take effect.

Additional Locations (1)

• hud/eval/run.py#L245-L247

 

^{Reviewed by Cursor Bugbot for commit e398638. Configure here.}