Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
73 changes: 53 additions & 20 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# ParallelKit

[![Tests](https://github.com/HalRiveria/PowerShell-5.1-Parallelization-and-Multithreading-Script/actions/workflows/tests.yml/badge.svg)](https://github.com/HalRiveria/PowerShell-5.1-Parallelization-and-Multithreading-Script/actions/workflows/tests.yml)
[![Tests](https://github.com/HalRiveria/PowerShell-ParallelKit/actions/workflows/tests.yml/badge.svg)](https://github.com/HalRiveria/PowerShell-ParallelKit/actions/workflows/tests.yml)

Run thousands of command-line programs, or blocks of PowerShell code, in parallel from Windows PowerShell 5.1, and stop cleanly when you press Ctrl+C. One file, nothing to install.

Expand All @@ -13,7 +13,26 @@ It was pulled out of [Manga Conversion Script](https://github.com/HalRiveria/Man

Built with Claude Code.

**Contents:** [Who it's for](#who-its-for) · [Setup](#setup) · [Run a list of programs](#run-a-list-of-programs) · [Run PowerShell code in parallel](#run-powershell-code-in-parallel) · [The three rules for worker code](#the-three-rules-for-worker-code) · [Process dispatch with groups](#process-dispatch-with-groups) · [Single processes](#single-processes) · [Messages](#messages) · [Abort](#abort) · [Choosing limits](#choosing-limits) · [Limits](#limits) · [Testing](#testing)
**Contents:** [Quick start](#quick-start) · [Who it's for](#who-its-for) · [Setup](#setup) · [Run a list of programs](#run-a-list-of-programs) · [Run PowerShell code in parallel](#run-powershell-code-in-parallel) · [The three rules for worker code](#the-three-rules-for-worker-code) · [Process dispatch with groups](#process-dispatch-with-groups) · [Single processes](#single-processes) · [Messages](#messages) · [Abort](#abort) · [Choosing limits](#choosing-limits) · [Limits](#limits) · [Testing](#testing)

## Quick start

Most uses are this: a list of command-line programs, run a few at a time, where Ctrl+C lets the running ones finish instead of cutting them off.

```powershell
. "$PSScriptRoot\ParallelKit.ps1"

$commands = foreach ($f in Get-ChildItem *.png) {
@{ FilePath = 'cjxl.exe'; Arguments = ('"{0}" "{0}.jxl"' -f $f.FullName) }
}
$batch = Invoke-PkRun { Invoke-PkProcessBatch -Commands $commands -MaxConcurrent 6 }

foreach ($r in $batch.Results) {
if ($r.ExitCode -ne 0) { Write-Warning ('Failed: ' + $r.Command.Arguments) }
}
```

That's all you need for this case. The worker rules, closures and callback details further down don't apply to it: they only matter for running PowerShell code in parallel (`Invoke-PkRunspaceJobs`) or for the group dispatcher. [Run a list of programs](#run-a-list-of-programs) covers the result fields and options.

## Who it's for

Expand Down Expand Up @@ -63,16 +82,21 @@ Each public function has a comment block at the top of its definition in `Parall
```powershell
# Any command-line tool works. cjxl is a JPEG XL image encoder.
$commands = foreach ($f in Get-ChildItem *.png) {
@{ FilePath = "cjxl.exe"; Arguments = "`"$($f.FullName)`" `"$($f.FullName).jxl`""; Source = $f }
@{ FilePath = 'cjxl.exe'; Arguments = ('"{0}" "{0}.jxl"' -f $f.FullName); Source = $f }
}
$batch = Invoke-PkRun {
Invoke-PkProcessBatch -Commands $commands -MaxConcurrent 6 -OnComplete {
$batchParams = @{
Commands = $commands
MaxConcurrent = 6
OnComplete = {
param($command, $result)
if ($result.ExitCode -ne 0) { Write-PkMessage "FAILED: $($command.Source.Name)" -Level Error }
}
}
$batch = Invoke-PkRun { Invoke-PkProcessBatch @batchParams }
```

Long parameter lists in these examples use splatting (`@batchParams`) instead of trailing backticks: it's easier to read, and a stray space after a backtick can't break it.

Each command is a hashtable with `FilePath`, `Arguments` and optionally `WorkingDirectory`, plus any keys of your own. `Arguments` is passed to the program as one string, exactly as written, so put quotes around paths that may contain spaces (as above). Programs run hidden, with their output captured.

`Invoke-PkProcessBatch` returns `@{ Results; Aborted; ForceKilledCount }`. Each result is `@{ Command; ExitCode; Stdout; Stderr; StreamTimedOut; StreamError; StartError; Aborted }`, with every field always present:
Expand Down Expand Up @@ -105,16 +129,20 @@ $jobs = [System.Collections.Generic.List[hashtable]]::new()
foreach ($f in Get-ChildItem -File) { $jobs.Add(@{ Path = $f.FullName }) }
$shaByPath = @{}

$out = Invoke-PkRun {
Invoke-PkRunspaceJobs -MaxParallel 4 -JobParams $jobs -SharedFunctionNames @('Get-FileSha') -CheckWorkerSafety `
-WorkerScript {
param([string]$Path)
if (Test-PkCancelRequested) { return }
Write-PkMessage "hashing $Path"
return @{ Path = $Path; Sha = (Get-FileSha -Path $Path) }
} `
-ResultHandler { param($r) $shaByPath[$r.Path] = $r.Sha }.GetNewClosure()
$jobsParams = @{
MaxParallel = 4
JobParams = $jobs
SharedFunctionNames = @('Get-FileSha')
CheckWorkerSafety = $true
WorkerScript = {
param([string]$Path)
if (Test-PkCancelRequested) { return }
Write-PkMessage "hashing $Path"
return @{ Path = $Path; Sha = (Get-FileSha -Path $Path) }
}
ResultHandler = { param($r) $shaByPath[$r.Path] = $r.Sha }.GetNewClosure()
}
$out = Invoke-PkRun { Invoke-PkRunspaceJobs @jobsParams }
```

Each worker runs in its own runspace: a separate PowerShell session inside the same process, with its own variables and functions. That separation is what makes running them at the same time safe, and also why a worker can't see your script's variables or functions unless you hand them over.
Expand All @@ -132,6 +160,8 @@ Each worker runs in its own runspace: a separate PowerShell session inside the s

## The three rules for worker code

These only apply if you run PowerShell code in parallel with `Invoke-PkRunspaceJobs` (or call a dispatcher from inside one of its workers). Running external programs with `Invoke-PkProcessBatch` or `Invoke-PkProcessDispatch` on the main thread never touches them.

Worker code is the worker script, every function in `-SharedFunctionNames`, and anything they call. Breaking these rules doesn't fail loudly. It hangs, or quietly reads nothing.

1. **No `$script:` variables.** In a worker they can silently read as empty. Pass state in as job parameters. (`$script:` at the very top level of the worker script is fine; inside a function or a nested `{ }` block it isn't.)
Expand All @@ -155,12 +185,15 @@ Only worker code is affected. Callbacks that run on the main thread (`-ResultHan
`Invoke-PkProcessBatch` covers a flat list. `Invoke-PkProcessDispatch` is the engine under it, for work that comes in groups (a folder of files, say) where something should happen when each group finishes:

```powershell
Invoke-PkProcessDispatch -MaxConcurrent 6 `
-GroupProvider { <# next group object, or $null when done #> } `
-JobBuilder { param($group) <# hashtable job(s): @{ FilePath; Arguments; WorkingDirectory; ... } #> } `
-JobFinalize { param($job, $pending) $r = Complete-PkProcess -Job $job; ... } `
-JobAbortCleanup { param($job) <# delete a killed job's partial output #> } `
-GroupCompleteCallback { param($group) <# e.g. zip the folder #> }
$dispatchParams = @{
MaxConcurrent = 6
GroupProvider = { <# next group object, or $null when done #> }
JobBuilder = { param($group) <# hashtable job(s): @{ FilePath; Arguments; WorkingDirectory; ... } #> }
JobFinalize = { param($job, $pending) $r = Complete-PkProcess -Job $job; <# ... #> }
JobAbortCleanup = { param($job) <# delete a killed job's partial output #> }
GroupCompleteCallback = { param($group) <# e.g. zip the folder #> }
}
$dispatch = Invoke-PkProcessDispatch @dispatchParams
```

All of these callbacks run on whichever thread called the dispatcher: the main thread normally, or the worker if you call it from inside a runspace job (in which case the worker rules apply to them).
Expand Down
Loading