Conversation
The mbpp reward guards exec() with signal.alarm, which only fires between Python bytecodes. Generated code that sits inside one long C-level call (a huge integer power, sum(range(10**13)), a catastrophic regex) is never interrupted, and in non-main threads the guard is skipped altogether. In practice a single non-terminating candidate pinned one core for hours and stalled a whole evaluation run. Run the evaluation body in a forked child and SIGKILL it when the budget (timeout_sec per exec, as before) is exhausted. The per-exec alarms are kept inside the child so pure-Python loops still fail fast; the fork path is used only where os.fork exists, other platforms keep the previous behaviour. Verified locally: normal pass/fail unchanged; a pure-Python infinite loop still times out per test; sum(range(10**13)) and a loop that swallows BaseException, both of which hang the old implementation indefinitely, now return "Code execution timed out" at the budget. Signed-off-by: khazic <khazzz1c@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
baselines/verl/utils/reward_score/mbpp.pyboundsexec()with asignal.alarmcontext manager. That guard can only fire between Python bytecodes, so two classes of generated code escape it:sum(range(10**13)), a huge integer power, a catastrophic regex): the alarm is delivered but the handler never gets to run until the call returns;timeout()detects the non-main thread and silently yields with no limit at all.In a real run a single candidate whose program never terminated pinned one core at 100% for 7.5 hours and stalled the entire evaluation, with the 5 s alarm armed the whole time.
Fix
execute_code_with_testsnow runs its body in a forked child and the parent SIGKILLs the child when the budget is exhausted. The budget istimeout_sec * (1 + len(test_list)), the same upper bound the per-exec alarms already implied. The per-exec alarms are kept inside the child so pure-Python loops still fail fast on their own test. Whereos.forkdoes not exist the previous in-process path is used unchanged, so non-POSIX behaviour is identical.The result contract (
passed,passed_count,total_tests,error) andcompute_scoreare untouched; a killed child reports"Code execution timed out"exactly like an alarm did.Verification
Local run against the patched module with
test_list = ["assert f(2) == 4", "assert f(3) == 9"]:x * xxwhile True: passsum(range(10**13))BaseExceptionThe unpatched module on the
sum(range(10**13))case does not return within 30 s with its 5 s alarm armed (killed by an external cap).