Repository navigation
Let output a batch has finished with be freed during the run - #25
Merged
Merged
Conversation
A batch kept every finished program's output in memory until it returned, even when -OnComplete had already set $result.Stdout and .Stderr to $null. The dispatch loop held the group's job list, and each job still referenced its output-reading task. A batch is a single group, so that list lived for the whole run. A dispatcher run with one big group behaved the same. Building and queueing a group's jobs is now its own function, so the list dies with that call. Nulling the variables in the loop wasn't enough: once PowerShell had compiled the loop, its own temporaries kept the list alive. The loop still references a few of the most recent jobs (7 at -MaxConcurrent 4, however long the run). The README's Limits section now says a batch keeps every result until it returns, and how to drop output in -OnComplete. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What was wrong
A batch kept every finished program's output in memory until it returned, about 2 × (stdout + stderr) bytes per result. That held even when
-OnCompletehad already set$result.Stdoutand.Stderrto$null, so the obvious way to keep memory down did nothing.The cause: the dispatch loop kept the group's job list, and every job still referenced the task that had read its output. A batch is a single group, so the list lived for the whole run. A dispatcher run with one big group behaved the same.
What changed
Add-PkDispatchGroupJobs, so the job list dies with that call.-MaxConcurrent 4, whether the batch has 20 jobs or 40.-OnComplete.Testing
New harness section Y, 2 checks. 30 programs each print about 150 KB:
-OnCompletekeeps a weak reference to each output and then drops it;Each runs a full GC as the last job finishes and asserts at most 15 of the first 29 outputs are still alive.
Against the kit on
main: both fail with 29 of 29 held (2 FAILED, 269 passed).With the change: 5 of 29 held.
ALL PASSED (271 assertions)on 5.1 and 7.Outside the harness, after warming the kit up: this branch held 7 outputs with both 20 and 40 jobs;
mainheld all of them.🤖 Generated with Claude Code