compression: add compression-threads option

[ernetas]

Layers exported as zstd are not compressed in parallel today. zstd.NewWriter is created with klauspost's default concurrency (GOMAXPROCS), but in streaming mode that only pipelines match finding with entropy coding and writing; the blocks themselves are encoded sequentially because they share the match history. A large layer at a high compression level takes minutes while the other cores sit idle.

klauspost/compress (already vendored at v1.19.2) supports real parallel stream compression with WithConcurrentBlocks(true): the input is split into large jobs (4× window size, 16–32MB) that are compressed simultaneously, each non-first job receives an overlap prefix from the previous job, and the output is flushed in order as a valid single-frame stream — the same design as zstd -T<n>.

This PR exposes that as a compression-threads=<n> attribute next to compression-level. It is parsed by the shared compression.ParseAttributes, so the image, OCI, registry cache and local cache exporters all accept it:

• 0: use all CPUs (GOMAXPROCS), like zstd -T0
• 1: the sequential encoder
• n ≥ 2: n parallel jobs
• unset: unchanged behaviour

Only zstd honours it for now; like compression-level with compression=uncompressed, other types ignore it. Negative or non-integer values are rejected when the attributes are parsed.

Determinism

Job boundaries are fixed by size and each job depends only on its own input and its overlap prefix, so the output does not depend on the number of threads or on how the tar stream is chunked into writes — only on whether parallel mode is used at all. TestZstdCompressThreads asserts this on a 40MB input spanning several jobs: threads=2 ≡ threads=4 ≡ threads=0, one write ≡ 4KB writes, and threads=1 ≡ unset.

The option is opt-in and the default output is byte-for-byte unchanged, because enabling parallel jobs changes the produced blob digests.

Numbers

256MB of synthetic text-like data, Apple M1 Pro (10 cores), BenchmarkZstdCompressThreads:

level	default	threads=1	threads=2	threads=4	threads=8	ratio default → parallel
3	493 MB/s	309 MB/s	593 MB/s	1057 MB/s	1652 MB/s	20.112% → 20.111%
12	83 MB/s	76 MB/s	147 MB/s	282 MB/s	384 MB/s	17.106% → 17.105%

Memory grows with the thread count: each in-flight job buffers 16–32MB of input and up to ~2n jobs can be in flight per layer writer, on top of buildkit already compressing the layers of a chain concurrently. That is why this is opt-in rather than the new default.

Not in this PR

• gzip/estargz stay as they are; the option is deliberately generic so a parallel gzip implementation could honour it later.
• Switching the default to parallel. That changes digests and memory use, so it deserves its own discussion.

[tonistiigi]

I don't see how a user is ever supposed to know what value provides good performance and doesn't OOM the machine at the same time. I support making sure we use all cores, but for that we need controls that allow using X cores to compress Y layers in parallel. If it is X cores for one layer, where all layers still run in parallel, then it is unpredictable.

See also #6841