Skip to content

Let output a batch has finished with be freed during the run - #25

Merged
HalRiveria merged 2 commits into
mainfrom
fix/batch-output-memory
Oct 29, 2026
Merged

HalRiveria merged 2 commits into
mainfrom
fix/batch-output-memory

Conversation

@HalRiveria-Bot

Copy link
Copy Markdown
Collaborator

What was wrong

A batch kept every finished program's output in memory until it returned, about 2 × (stdout + stderr) bytes per result. That held even when -OnComplete had already set $result.Stdout and .Stderr to $null, so the obvious way to keep memory down did nothing.

The cause: the dispatch loop kept the group's job list, and every job still referenced the task that had read its output. A batch is a single group, so the list lived for the whole run. A dispatcher run with one big group behaved the same.

What changed

  • Group building moved into its own function. Building, tagging and queueing a group's jobs is now Add-PkDispatchGroupJobs, so the job list dies with that call.
  • Why nulling wasn't enough. I tried nulling the variables in the loop first. That worked in a fresh session, but not once the kit had already run many times in the same session, as it has by the end of the test suite. My guess is that PowerShell's compiled code keeps its own references to the list; that part is inferred, not proven. A separate call fixed it in both cases.
  • What still stays alive. The loop still references a few of the most recent jobs: 5–7 at -MaxConcurrent 4, whether the batch has 20 jobs or 40.
  • Docs: the README's Limits section now says a batch keeps every result until it returns, and shows how to drop output in -OnComplete.

Testing

  • New harness section Y, 2 checks. 30 programs each print about 150 KB:

    • a batch whose -OnComplete keeps a weak reference to each output and then drops it;
    • a dispatcher with one group of 30 jobs.

    Each runs a full GC as the last job finishes and asserts at most 15 of the first 29 outputs are still alive.

  • Against the kit on main: both fail with 29 of 29 held (2 FAILED, 269 passed).

  • With the change: 5 of 29 held. ALL PASSED (271 assertions) on 5.1 and 7.

  • Outside the harness, after warming the kit up: this branch held 7 outputs with both 20 and 40 jobs; main held all of them.

🤖 Generated with Claude Code

HalRiveria and others added 2 commits October 29, 2026 17:53
A batch kept every finished program's output in memory until it
returned, even when -OnComplete had already set $result.Stdout and
.Stderr to $null. The dispatch loop held the group's job list, and each
job still referenced its output-reading task. A batch is a single group,
so that list lived for the whole run. A dispatcher run with one big
group behaved the same.

Building and queueing a group's jobs is now its own function, so the
list dies with that call. Nulling the variables in the loop wasn't
enough: once PowerShell had compiled the loop, its own temporaries kept
the list alive. The loop still references a few of the most recent jobs
(7 at -MaxConcurrent 4, however long the run).

The README's Limits section now says a batch keeps every result until
it returns, and how to drop output in -OnComplete.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@HalRiveria
HalRiveria merged commit 355e908 into main Oct 29, 2026
2 checks passed
@HalRiveria
HalRiveria deleted the fix/batch-output-memory branch October 29, 2026 22:26
@HalRiveria-Bot HalRiveria-Bot mentioned this pull request Oct 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants