Pipeline

Pipeline

ZArchiveSharp.Pipeline is the shared engine behind ZArchiveTool, the XISO .zar bridges and the CLI: directory pack, archive extract, and parallel batches with progress, pause, cancellation and collision handling. One engine, one set of semantics.

Basic Usage

Pack

using ZArchiveSharp.Pipeline;

// Minimal: defaults are zstd level 6, Fail policy, 4 workers
string? result = ZarPipeline.Pack(@"C:\mydata");

// Returns the archive path written (after collision resolution),
// or null when Skip policy skipped an existing output.
string? path = ZarPipeline.Pack(
    sourceDirectory: @"C:\mydata",
    zarPath: @"C:\mydata.zar",           // null => <stem>.zar next to input
    options: new ZarPipelineOptions { Level = 9 },
    progress: new Progress<ZarProgress>(p => Console.WriteLine($"{p.Ratio:P1}")));

The source is validated and collected before the output is resolved, so an Overwrite policy cannot delete a previous archive when the pack cannot even start (missing/unreadable source). With DeleteSourceOnSuccess, the source is removed only after the archive was written successfully.

Extract

using ZArchiveSharp.Pipeline;

// Returns extracted file paths relative to the archive root ('/' separated)
IReadOnlyList<string> files = ZarPipeline.Extract(
    zarPath: @"C:\mydata.zar",
    destDir: @"C:\mydata_out",
    progress: new Progress<ZarProgress>(p => Console.WriteLine($"{p.Ratio:P1}")));

Custom Sources

Anything that can enumerate (path, size, content stream) pairs can be packed via IZarPackSource — the directory tree and the XISO walk are both built on it:

using ZArchiveSharp.Pipeline;

ZarPipeline.PackSource(mySource, @"C:\out.zar", options);

Options

ZarPipelineOptions controls all pack/extract work. Defaults keep zarchive.exe codec parity (zstd level 6, no checksums) and use 4 workers for batch/block parallelism.

Property	Type	Default	Meaning
Level	int	6	zstd level 1–22 for packing (ignored for extract)
Checksum	bool	false	Per-block content checksums
Compressor	IZarBlockCompressor?	null	Explicit compressor; overrides Level/Checksum
DeterministicOrder	bool	true	Sort entries ordinally for reproducible archives
CollisionPolicy	ZarCollisionPolicy	Fail	Output-exists behavior
MaxDegreeOfParallelism	int	4	Batch workers (clamped ≥ 1; effective = min(workers, items)) plus 64 KiB block fan-out inside a single pack/extract (capped by processor count; byte-identical)
DeleteSourceOnSuccess	bool	false	Delete pack source after success (off by default — a library must not destroy inputs unless asked)
Pause	PauseToken	default	Pause gate checked alongside the cancellation token
NameOrder	IReadOnlyList<string>?	null	Pre-seeded name-table order; null = pack order. Pass a source-walk (discovery) order for byte-parity with packers that write names in discovery order

var options = new ZarPipelineOptions
{
    Level = 19,
    CollisionPolicy = ZarCollisionPolicy.AutoRename,
    MaxDegreeOfParallelism = 8,
    DeterministicOrder = true,
};

Progress

Progress is reported via IProgress<ZarProgress>:

var progress = new Progress<ZarProgress>(p =>
{
    Console.WriteLine($"{p.Operation} {p.CurrentFile}: {p.FilesCompleted}/{p.FilesTotal} {p.Ratio:P1}");
});

ZarProgress fields:

Field	Meaning
Operation	Pack or Extract
SourcePath	Item being processed
DestinationPath	Output path
CurrentFile	File within the item (empty between files)
FilesCompleted / FilesTotal	Pre-scanned totals
BytesCompleted / BytesTotal	Byte progress when known
Ratio	0–1 fraction (bytes when known, else files)

Totals are pre-scanned, so Ratio moves monotonically 0→1 within one SourcePath. Batch runs re-base each item's ratio into its 1/n share (completed items plus the in-flight fraction).

Cancellation and Pause

using ZArchiveSharp.Pipeline;

using var cts = new CancellationTokenSource();
using var pauseSource = new PauseTokenSource();

var options = new ZarPipelineOptions { Pause = pauseSource.Token };

var task = Task.Run(() =>
    ZarPipeline.Pack(@"C:\bigdata", @"C:\big.zar", options, null, cts.Token));

// Pause mid-run (checked at the same points as cancellation)
pauseSource.Pause();
await Task.Delay(5000);
pauseSource.Resume();

// Or cancel entirely — in-flight work stops; incomplete pack outputs are deleted
cts.Cancel();

Cancellation deletes incomplete pack outputs (the zarchive.exe delete-incomplete-output contract). PauseTokenSource is IDisposable (since v1.2.0): dispose it after the workers have stopped, while no PauseToken obtained from it can still be waited on.

Collision Policies

What happens when an output path already exists:

Policy	Behavior
Fail	Throw IOException (default; preserves the zarchive.exe refuse-overwrite contract)
Skip	Skip the item — reported as ZarItemStatus.Skipped
Overwrite	Delete the existing output, then write
AutoRename	Write to {stem}_{n}{suffix}, first free n from 1

Skip, Overwrite and AutoRename serve batch runs; Fail preserves the native CLI refuse-overwrite contract.

Resolution is race-safe: parallel batch items (or other processes) can claim the chosen name between resolve and write/move, so the pack and move paths re-resolve from the requested path and retry, keeping AutoRename suffixes canonical (game.zar, game_1.zar, …) instead of compounding. Fail-policy collisions throw IOException prefixed with ZarPackEngine.OutputExistsMessage (batch callers map a batch whose only failures are these to exit -11).

Batch Operations

Pack Batch

using ZArchiveSharp.Pipeline;

var results = ZarPipeline.PackBatch(
    sourceDirectories: [@"C:\game1", @"C:\game2", @"C:\game3"],
    destDir: @"C:\archives",             // null => each .zar next to its source
    options: new ZarPipelineOptions { MaxDegreeOfParallelism = 4 },
    progress: progress);

foreach (var r in results)
{
    Console.WriteLine($"{r.SourcePath}: {r.Status} {r.ErrorMessage}");
}

Semantics:

• Worker count is min(MaxDegreeOfParallelism, items) — never spins up more tasks than items
• One item's failure does not stop the others; per-item outcomes come back as ZarItemResult
• Batch progress re-bases per-item ratios into 1/n shares

Block-Level Parallelism (v1.1.0)

Each 64 KiB .zar block is an independent zstd frame, so a single pack/extract also fans out: BlockWorkers() is min(max(1, MaxDegreeOfParallelism), ProcessorCount), and blocks are always emitted in input order — parallel bytes equal sequential bytes at every level, with or without dictionaries/checksums.

• ZArchiveWriter takes an optional per-worker compressorFactory (one ZstdCompressor per worker); explicit IZarBlockCompressor instances stay sequential.
• Extract decodes bounded waves (min(workers × 4, 64) blocks) with the sequential corruption contract.
• A default options pack now uses up to 4 block workers instead of 1 thread — pass MaxDegreeOfParallelism = 1 for the old profile. Note batch × block workers multiply (PackBatch at DOP 4 can reach ~16 codec threads on many-core hosts).

Extract Batch

var results = ZarPipeline.ExtractBatch(
    zarPaths: [@"C:\a.zar", @"C:\b.zar"],
    destRoot: @"C:\extracted",
    options: options);

Same-stem archives (a\game.zar, b\game.zar) are given unique destinations within the batch (game_extracted, game_extracted_1, …) so parallel items never extract over each other.

Extraction Safety

ZarPackEngine treats archive entry names as untrusted:

• names must be single plain components — .., separators, rooted or drive-qualified paths, and Windows reserved device names are rejected (InvalidOperationException);
• the resolved destination path is re-validated against the extraction root (zip-slip defense in depth);
• nesting deeper than ZarPackEngine.MaxExtractDepth (1024) fails catchably instead of risking a stack overflow;
• each file is written through a unique .part scratch file and moved into place only after the byte-count check, so disk-full, corruption, or cancellation never leaves a truncated file that looks complete.

Packing never descends directory symlinks/junctions (reparse points): the link itself is archived as an empty directory entry, so a tree cannot loop forever or pull in content from outside the source root.

Batch Requests

For UIs, a ZarBatchRequest models the full input set with modes and collision handling; ZarItemResult/ZarItemStatus carry outcomes (Completed, Skipped, Failed, ...).

Archive-Container Stage (7z)

SevenZip (ZArchiveSharp.Pipeline) is the library half of the archive-container stage: FindTool locates the external binary (explicit path, then the standard Windows install location, then 7z/7zz on PATH) and Extract runs it via ProcessRunner (x for full paths, -bsp1 for progress, exit 0/1 accepted). PickIsoCandidate selects the first .iso out of the extracted tree. The CLI owns the orchestration (temp dir, collision-resolved moves, extract-archive terminal vs. continue-to-.zar, source cleanup), since the ISO→.zar leg needs the CLI-side XISO bridge.

Exit Codes (zarchive.exe Contract)

ZarchiveCli.Run is the callable form of the zarchive.exe input_path [output_path] contract: directory input packs, file input extracts, outputs default to <stem>.zar / <stem>_extracted, existing pack outputs are refused, incomplete outputs are deleted.

using ZArchiveSharp.Pipeline;

int code = ZarchiveCli.Run(
    args: ["C:\\mydata", "C:\\mydata.zar"],
    options: options,
    log: Console.WriteLine);   // receives the native stdout chatter

Exit codes (same negative values main() returns; -2, -5–-9 unused upstream too):

Code	Constant	Meaning
0	Ok	Success
-1	BadUsage	Too many paths; input neither file nor directory
-3	OutputNotDirectory	Extract output exists and is not a directory
-4	OutputDirectoryNotCreated	Could not create output directory
-10	NotFound	Archive not found; pack output not a regular file
-11	Refused	Archive failed to open; pack output already exists
-12	ExtractionFailed	Corrupt archive or extract I/O
-13	PackFailed	Archive structure error
-14	ArchiveEntryFailed	Duplicate or bad path
-15	InputNotReadable	Could not open an input file
-16	PackOutputFailed	Output I/O error

Automation matching on these codes keeps working unchanged.

Stdout Chatter

With log supplied, ZarchiveCli reproduces native chatter:

• Pack: Outputting to: ... then Adding <path> per file (OS separators in the display line)
• Extract: Extracting to: ... then per-entry lines with the native leading-/ quirk, directories included, preorder — even on mid-archive failure

Three Intentional Deviations

Where native behavior is a bug, ZArchiveSharp deviates (all tested):

1. An unopenable extract output throws (native prints Unable to write file: then keeps writing into the dead stream)
2. A mid-file input read error fails the pack with -16 (native treats a short read as EOF and silently packs a truncated file)
3. Error-string paths use / on every OS (native pathEntry.string() prints \ on Windows; only the Adding display line converts)

Exception Types

Exception	When
ZarArchiveOpenException	Archive failed to open/validate (maps to -11)
ZarInputOpenException	Pack could not open an input file (maps to -15)
ZarEntryCreateException	Duplicate or bad archive path (maps to -14)
IOException	Output I/O errors (maps to -16); also Fail-policy collisions
InvalidOperationException	Archive structure errors (maps to -13)
OperationCanceledException	Cancelled — propagated, never swallowed

Config and File Discovery

• ZarSettings — AOT-safe JSON settings: defaults, merge-forward load, per-user save location
• ProcessableFiles.Find(dir, ZarProcessMode.Auto) — ARCHIVE/ISO file sets, non-recursive scan, ordinal sort

ProcessRunner

ProcessRunner is the seam for external tools (e.g., 7z): (\d+)% progress parsing with a 10 FPS throttle, exit 0/1 treated as ok, anything else throws with the last output line, WinError 740 mapped to an elevation message, missing binaries get an AV-hint. Since v1.2.0 stderr is drained after exit (a late-only failure line is still reported) and the child is polled, so one that closed stdout but keeps running is killed on cancellation.