DNM: upsert v2 hydration optimizations and payload-separated state

[DAlperin]

Motivation

DNM. Draft benchmark branch only. Upsert v2 hydration spends substantial work allocating and moving wide payloads through its spill and merge path. This branch combines the optimization experiments and payload-separated upsert prototype so CI publishes images for controlled comparisons. Optimization commits remain separate for review.

Description

• Bound serialized column chunks to 2 MiB, except indivisible oversized rows. Size merge reservations and output checks from observed row width.
• Bound pool insertion debt under concurrent enforcement. Encode admission-denied inserts directly to extents.
• Add enable_upsert_async_reads: offload upsert drain reads to the blocking executor, with at most eight queued or running reads per pool and cancellation-safe permit ownership. This flag controls drains and probes independently of merge scheduling.
• Add enable_upsert_async_merges: use Differential's pollable chunk merge pipeline and batcher with a vendored fueled spine for source-stash insertion and sealing, feedback batching, and feedback trace compaction. Pending reads preserve merge phases, remaining fuel, and published input batches. Materialize continues to use its existing Differential trace interfaces. A private timestamp/frontier adapter connects maintenance to the newer DD/Timely dependency without changing stored timestamps or Timely operators. The DD polling APIs are pinned to 3659671246e0b5635a8128cc27d65f12ecf7dda5 in the fork. Only the spine scheduling implementation remains vendored. DNM-only dependency-policy exceptions allow the two pinned Git sources and duplicate DD/Timely versions. Merge reads yield to Timely and share a 256 MiB decoded-input budget across workers and sources. Trace replacement publishes atomically after completion and respects aggregated reader compaction holds. This path requires chunked upsert with payload separation disabled.
• Add enable_column_chunk_direct_compressed_output: serialize compressed generations directly to extents, avoiding resident admission and the intermediate serialized allocation. Younger identity-coded generations keep the existing path.
• Add enable_upsert_payload_stash: keep row bytes in shared pool-backed blocks and merge columnar keys, times, differences, and payload handles. Manifests retain referenced blocks through compaction and restashing. Feedback compares keys and fingerprints before resolving exact payload bytes, so equal updates cancel across physical allocations without a persistent interner. The bounded comparison cache reuses blocks across argument positions and within-block comparisons. Source encoding publishes only winners after offset consolidation, and output decoding groups reads by block within bounded probe windows.

The flags default off in production and on in the test configuration. The payload path uses Materialize's existing Timely and Differential dependencies. The standalone local proxy-join harness remains separate and is not required to build these images.

This is a prototype, not a complete out-of-core runtime. Manifests and chunk summaries remain resident. Payload merge comparisons use synchronous reads into a two-block cache outside shared read admission. Probe histories and other temporary metadata still need bounds, payload decoded-read budgets are per dataflow, and partially live blocks retain dead bytes. Oversized rows retain an inline fallback. The design document describes the shared operator model and remaining boundaries.

Verification

Regression coverage exercises byte bounds, oversized rows, concurrent admission with stalled enforcement, resident and spilled round trips, async cancellation and concurrency limits, and direct extent accounting and codec equivalence. Shared payload tests cover ownership, admission, exact equality across independent locators and forced fingerprint collisions, timestamp compaction, block reclamation, and cache reuse across swapped arguments. A deterministic local microbenchmark compares unique keys, cancellation, replacement churn, forced collisions, and timestamp advancement against a canonical-handle control. It verifies exact output and block retirement, and records merge/advance timing and payload decompression counts separately.

The upsert operator comparison exercises paged, chunked, resumable, and payload-separated execution under forced spilling. Resumable batching is also compared against synchronous sealing across retractions, times, and re-stashing. Trace tests verify reader holds and that pending compaction keeps both input batches visible. Spine tests pause irregular fueled rollups to check visibility and continuation state. DD owns the chunk and merger suspension tests. Timestamp-adapter tests cover incomparable frontiers, lattice operations, and path-summary composition and overflow. It covers inserts, highest-offset selection, deletes, persisted feedback, resume filtering, ineligible restashing, errors, multiple payload blocks, and oversized rows.

Hydration comparisons include the original 12 million key seed and a skewed 192.36 million row seed across 105 sources on eight workers, with 1,900 byte payloads and a 0.2 pool budget fraction. The replacement resumable profile pins commit 09ed476925278025aed3aa35df08fa07e5d5fd10, keeps drain offload enabled, and compares the vendored Differential merge driver on/off twice in reversed order using the retained large seed. This third queue entry replaces the earlier fully vendored driver and uses the pinned DD polling APIs. The on/off comparison includes async admission and resumable maintenance as a unit. The existing runs retain their pinned images. Measurements retain hydration time, CPU, memory, extent traffic, image identity, and replica restart evidence. Resumable merge performance gains are not yet established.

Release note: None (draft performance experiment).