TCP T1/T2: native-client benchmarks, the transport comparison, and a working comparison job

[alex-clickhouse]

Epic T1/T2: benchmarks for the native client, the cross-transport comparison, and the PR comparison job that runs them.

Ratios below are time relative to HTTP, so lower is faster — the same convention as BenchmarkDotNet's own Ratio column.

What this adds

Native-protocol benchmarks (tcp-regression, 41 methods). Each class prices the tiers a caller chooses between, not one number per operation:

Class	Compares
TcpSelectColumn	16 expressions, block tier, mirroring SelectColumn
TcpReadTiers	block spans / typed indexer / boxed GetValue / object[] / POCO
TcpProjectedRead	stored type / ReadAs<T> indexer / ReadAs<T>.Values
TcpCompositeRead	composite interface / indexer / Values / row tier, over 5 composites
TcpStringRead	IStringColumn.Offsets / GetBytes / decoded strings
TcpInsertShapes	columnar / object[] / POCO
TcpArrayWriteShape	dense offsets against one array per row
TcpConnectionCost	warm ping / warm scalar / client-per-operation
TcpPoolConcurrency	64 queries serially against widths 1, 8, 32
TcpCompression	none / lz4 / zstd, read and insert

Cross-transport benchmarks (cross, 14 methods): TransportRead, TransportReadPoco, TransportInsert, TransportLatency. Each pairs an HTTP arm with its native counterpart on identical data in one process, with the codec and the server's insert buffering equalised so that a ratio reports the transport and not a setting that lands on one side only.

A working PR comparison job. benchmark-compare.yml has been failing to compile since the Common assembly split: the outer dotnet run left ClickHouseDriverVersion unpinned, so it restored the default release from nuget.org and could not compile against any API added since. It now pins the version and source, publishes 9000, and defaults baseline_ref to the PR's own base branch — a tcp/** PR sits on a stack whose base is not main, so comparing against main cannot compile.

Category selection from the diff. A change under ClickHouse.Driver.Tcp/ runs the native set, ClickHouse.Driver/ the HTTP set, either Compression/ directory the codec sweep, and the harness itself runs everything. Both transports moving adds the cross table. A push to main runs the regression sets; the nightly build runs all of them.

TCP against HTTP

One ClickHouse 26.6.1.1193 container serving both 8123 and 9000, so both transports hit the same data and CPU budget. 3 warmup + 30 iterations × 2 launches, on 4 logical and 2 physical cores. Loopback, so this measures client and protocol work and none of the transfer saving that compression exists for. The read sections use 2M rows; the insert section uses 5M, for the reason given there.

Every time below is the fastest of 120 iterations: 30 × 2 launches, run twice. This host interferes in bursts, so an iteration can land far above its neighbours. Means and medians do not survive that; best-case iterations do, agreeing between the two runs to within 0.03% to 6.79% on every one of the 18 cross-transport arms. Interference is additive, so the fastest iteration is the arm's own cost and the tail is the box. Allocation does not depend on this and is quoted as BenchmarkDotNet reports it.

Every arm below runs uncompressed on both sides. The transports do not default to the same
codec (HTTP gzip or ZSTD, native LZ4), so at their defaults the codec difference reports itself as a
transport difference, and over loopback a codec costs CPU while saving nothing. An earlier revision
of this PR ran the read arms at both clients' defaults; that inflated the native client's advantage,
and the corrected figures are below. What compression is actually worth is measured on a real
network further down.

Everything on loopback was re-measured on 2026-09-11, at bdf1cdb8. Two things make that worth saying rather than silently replacing numbers:

• The driver changed underneath the branch. The 2026-09-09 rebase pulled in main as of that date, which includes the box-free typed column slots of Box-free ADO read path: typed column slots (stacked on #449) #499. That is a real change to the HTTP read path and it moves the allocation columns below by three orders of magnitude on fixed-width data. It is not noise, and the row-reads section says what it did.
• The host is not the host the earlier figures came from. The box was reconfigured between the two sessions. Allocation reproduces to three significant figures wherever the code did not change — every native arm, HTTP's string-heavy read, the whole insert allocation model — so the arms are doing the same work. Times are not stable across sessions: bulk arms run 15% to 86% slower than the 2026-08-30 session, while TransportLatency runs 13% to 31% faster. A single cause that slows bulk throughput and speeds up round trips is not obvious, and this re-run cannot identify one. Treat absolute times as this box on this day.

The WAN and compression figures further down are the one part not re-measured, because they need a Cloud service; they are attributed where they appear.

Row reads — TransportRead

Http and TcpRows both consume every value; TcpBlocks is the columnar ceiling.

Shape	HTTP	TCP rows	TCP blocks	HTTP alloc	TCP rows alloc	TCP blocks alloc
1 × UInt64	68.6 ms	79.6 ms (1.16)	11.7 ms (0.17)	15.5 KB	109,397 KB (7,051)	22.2 KB (1.43)
UInt64+String+Float64	255.7 ms	274.9 ms (1.07)	172.4 ms (0.67)	76,580 KB	264,111 KB (3.45)	76,611 KB (1.00)
3 × String	352.6 ms	518.9 ms (1.47)	521.2 ms (1.48)	232,049 KB	325,840 KB (1.40)	232,080 KB (1.00)

Equalising the codec removed the native row tier's read advantage. At the two clients' defaults an earlier revision of this PR measured TcpRows at 0.66 / 0.72 / 1.02; uncompressed it reads 1.16 / 1.07 / 1.47. HTTP had been paying ZSTD on the response body while the native side paid LZ4, and that gap was most of the earlier win. On this hardware the native row tier is not faster than the ADO reader.

The block tier's time advantage is real and is where the value is: 0.17 on fixed-width data, because nothing is boxed and nothing is copied.

Its allocation advantage on fixed-width data is gone, and HTTP now wins it. 15.5 KB against the block tier's 22.2 KB, where the earlier session measured HTTP at 46,890 KB. #499 gave GetFieldValue<T> typed column slots that return the value without boxing, so HTTP's reader no longer allocates per row on this shape — 2M rows cost it fixed buffers and nothing else. The same change shows up on the mixed shape, where HTTP drops from 170,330 KB to 76,580 KB and lands within 31 KB of the block tier: both sides now allocate the 2M strings and neither boxes the integer or the float. On three string columns nothing changed, because strings were never the boxing (232,048 KB then, 232,049 KB now). So the block tier is now a time argument, not an allocation one.

The native row tier still allocates a fresh object[] per row — 109,397 KB on the narrow shape in both sessions, 56 B/row. Against HTTP's fixed buffers that is 7,051×, where the earlier session read 2.33×; the multiple moved because HTTP's denominator collapsed, not because the row tier changed. QueryAsync yields a fresh object[] per row because a consumer is free to keep it, which is the row API's shape rather than the protocol's, and it is not fixable without changing the contract.

On string-heavy data both TCP tiers are slower than HTTP, 1.47 and 1.48. The block arm calls .Values on the string columns, materialising 6M strings, and pays worse GC for it (Gen1 7,000 per 1,000 ops against HTTP's zero, Gen2 zero on all three) because three 2M-element arrays live for the whole block and get promoted, while HTTP's per-row strings die in Gen0.

That is what IStringColumn is for. TcpStringRead reads the same strings for 33.5 ms and 12.17 KB against 46.1 ms and 19,538 KB: 0.73 on time, and 1/1,600 of the allocation. So TCP's read advantage on strings depends entirely on whether the caller decodes them.

POCO reads — TransportReadPoco

Arm	Fastest	Ratio	Allocated	Alloc ratio
HTTP QueryAsync<T>	277.5 ms	1.00	151.08 MB	1.00
TCP QueryAsync<T>	235.0 ms	0.85	151.11 MB	1.00
TCP blocks → POCO by hand	198.8 ms	0.72	151.11 MB	1.00

All three allocate within 1% of each other, and the figures are identical to the earlier session's — 151.08 MB over 2M rows is 79 B/row, which is the POCO (40 B) plus one fresh string per row (~32 B) and essentially nothing else. The objects are the allocation and no transport can avoid them. Neither path boxes: QueryAsync<T> materialises straight from the stream on both transports.

What the shared allocation floor does not do is equalise the time. The earlier session put all three within 3% of each other and concluded the floor left nothing for a transport to win; re-measured, the native reader is 15% ahead and hand-building from blocks 28% ahead. The allocation is identical to the byte, so whatever moved did not change what these arms allocate, and the gap is far larger than anything the two runs within this session disagree about (0.16% to 4.28% per arm). Beyond that the cause is open: #499 does not reach QueryAsync<T>, but the rebase brought three weeks of main with it and this re-run cannot separate that from the host. An earlier revision of this PR reported 0.70 here and the PR then withdrew it as a codec artefact; at an equalised codec it reads 0.72, so that withdrawal was wrong. What is fair to say is that the ordering is stable and the margin is not: on this box a POCO consumer should expect the native transport to be somewhere between level and a quarter faster, and should not plan around a specific figure.

Hand-building from blocks remains the worst of the three on GC pressure — Gen1 4,000 per 1,000 ops against zero for both QueryAsync<T> arms — because the intermediate column arrays survive the block.

A caller who wants materially better than this has to stop building objects, which is what the 0.17 on narrow blocks shows.

Latency — TransportLatency

500 sequential SELECT 1 on warm connections: HTTP 572.7 ms (1.15 ms/query, 7.19 MB) against TCP 513.4 ms (1.03 ms/query, 4.64 MB), ratio 0.90, allocation 0.65. TCP saves about 0.12 ms of fixed per-request cost.

This is the section that moved most against the earlier run, and in the opposite direction to everything else: both arms are faster than they were (824.6 and 590.6 ms), so the native client's per-request advantage narrows from 0.72 to 0.90 while its allocation ratio holds at 0.65 against 0.66. The allocation figures rule out the arms having changed. Whatever this host did between the two sessions, it helped round trips and hurt bulk transfer, so the size of the latency win is the least portable number in this document.

Inserts — TransportInsert

5,000,000 rows, because a smaller insert measures the fixed cost of an insert as much as the serialization the arms compare. Fitting both sizes gives each arm 4.7 to 8.6 ms of fixed cost, which is a fifth of a 500k-row measurement, so the ratios there compress toward 1 (TcpColumnar reads 0.65 at 500k against 0.60 at 5M, TcpRows 1.07 against 1.04, TcpPoco 0.91 against 0.94). 5M costs about 1 GB of source rows held for the run, which is the reason not to go further.

The native arms send this insert as 100 wire blocks of 50,000 rows, which is MaxRowsPerBlock at its default, and convert rows to columns one block at a time.

Two things are turned off, because each lands on the transports unequally and would report itself as a protocol difference. async_insert = 0: a server with async inserts on holds an insert's response until its buffer flushes, waiting from async_insert_busy_timeout_min_ms (50 ms) upward, and the native arms never paid it. No request compression: the transports do not default to the same codec (HTTP ZSTD, native LZ4), and the codec costs more than the serialization the arms exist to compare. Over loopback there is no bandwidth to save and TcpCompression owns the codec axis, so nothing of value is dropped.

Arm	Fastest	Ratio	ns/row	Allocated
HttpRows	385.8 ms	1.00	75.8	30.1 KB
TcpRows	400.6 ms	1.04	78.4	519.9 KB
HttpPoco	392.5 ms	1.02	76.9	30.4 KB †
TcpPoco	362.6 ms	0.94	71.5	518.3 KB
TcpColumnar	233.2 ms	0.60	45.3	517.8 KB
HttpRowsDefaultBatching	536.1 ms	1.39	106.3	487.2 KB

The ns/row column is the slope between the two sizes, so it excludes the fixed cost. † HttpPoco's allocation does not hold still between runs; the lower of the two is quoted and the spread is below.

The native protocol's insert advantage still needs the columnar shape to appear. TcpColumnar is the fastest thing here at 0.60, and hand the same client rows and the advantage disappears — but it no longer reverses. The earlier session put TcpRows behind HTTP at 1.35; re-measured it is 1.04, which is parity, and TcpPoco holds at 0.94 exactly. The claim that the row tier is slower than HTTP on inserts does not survive re-measurement; the claim that it gives up the columnar win does.

The transpose is still what the row tiers pay, and the two sizes price it twice over: as totals, 400.6 − 233.2 = 167.4 ms for 15M values, or 11.2 ns per value; as slopes, 78.4 − 45.3 = 33.1 ns per row, which is the same 11.0 ns per value. HTTP's RowBinary is row-major, so it matches the input layout and encodes straight into the request body while sending; the native block format is column-major, so every column must be fully materialised before a byte of it goes out, and the transpose runs inside the schema => … callback, wedged between two round trips with nothing to overlap.

The POCO tier beats the object[] tier on the native side, 362.6 against 400.6 ms — 2.5 ns per value, where the earlier session measured 7.2 — because the compiled typed accessors read fields in place instead of unboxing scattered values. On HTTP the same swap is a wash on the clock (392.5 against 385.8 ms), as it was before. A probe in the earlier session tagging each insert with a query_id put the HTTP POCO path at 239 ms of client CPU against the object[] path's 297 ms; that probe was not repeated, and the wall-clock wash it explains still reproduces.

Allocation is per unit of framing, and the transports frame differently — and the framing model is the part that reproduces to the digit. A native insert allocates about 13 KB fixed plus 5.05 KB per wire block, which the two sizes pin down: 10 blocks cost 63 to 65 KB and 100 cost 518 to 520 KB, against the earlier session's 13 KB and 5.1 KB per block. HTTP allocates about 20 KB fixed plus 9.3 KB per request, so one request costs 30 KB and the fifty of HttpRowsDefaultBatching cost 487 KB. Neither figure includes row handling, because the source objects are built once in GlobalSetup, and no arm triggers a single GC collection. Neither HTTP path boxes during the operation: the POCO path uses the typed Action<T, ExtendedBinaryWriter> writers, and InsertOptions.Format defaults to RowBinary, so the boxed getters serve only RowBinaryWithDefaults and failure diagnostics.

HttpPoco's allocation was the one arm the earlier session could not reproduce, at 30.9 KB in one run against 358.9 KB in the other. The anomaly recurred, so it is not a one-off: 95.41 KB in this session's first run against 30.36 KB in its second, while HttpRows beside it held at 30.29 and 30.12 KB. Smaller spread than before and the same shape — one arm, on the HTTP side, whose allocation jumps by a multiple between runs while its neighbour is stable to 0.6%. Still unexplained, and now twice observed rather than once.

InsertOptions.BatchSize: the default 100,000 sends fifty requests for 5M rows, costing 536.1 against 385.8 ms — about 3.1 ms per extra request, which is the round trip and little else — plus 16.2× the allocation in per-request buffers. An earlier revision of this PR reported a 4× time penalty here; that was fifty exposed async_insert waits and the claim stays withdrawn.

Compression, and the columnar layout, on a real network

Loopback cannot answer whether a codec pays for itself, so this was measured against a ClickHouse Cloud service (26.4.1.2212): 500k rows, median of 3 after a warm-up, rows taken from ClickBench hits so the payload is not a compressor-friendly sequence. This is characterisation from a one-off probe on 2026-08-30, not a committed benchmark, and it is the one section the 2026-09-11 re-run did not repeat — it needs that service, and its arms run for 0.4 s to 30 s each on a path where transfer dominates client work by an order of magnitude, so nothing the client does moves them. The arms run sequentially over a WAN, so read differences under ~15% are noise.

Reads:

Shape	HTTP	HTTP compressed	Native	Native LZ4	Native ZSTD
10 × hits, mixed	30,203 ms	3,103 ms (0.10)	26,776 ms	5,863 ms (0.22)	5,294 ms (0.20)
10 × hits, numeric	2,332 ms	245 ms (0.10)	2,499 ms	397 ms (0.16)	506 ms (0.20)
3 × synthetic	2,004 ms	692 ms (0.35)	2,792 ms	1,303 ms (0.47)	1,398 ms (0.50)

Inserts:

Shape	HTTP	HTTP gzip	HTTP ZSTD	Native	Native LZ4	Native ZSTD
10 × hits, mixed	19,279 ms	3,772 ms (0.20)	3,059 ms (0.16)	16,847 ms	4,314 ms (0.26)	3,348 ms (0.20)
10 × hits, numeric	1,865 ms	830 ms (0.45)	463 ms (0.25)	2,784 ms	687 ms (0.25)	415 ms (0.15)
3 × synthetic	2,458 ms	802 ms (0.33)	791 ms (0.32)	1,953 ms	1,208 ms (0.62)	448 ms (0.23)

Compression is the dominant factor and everything else is secondary. It takes 0.10 to 0.26 of the uncompressed time. This is the number the loopback suite structurally cannot produce, and it settles the direction of T1a: compression on by default is right.

ZSTD, not LZ4, is the better codec on this path — it wins 4 of 6 arms, including both large payloads and every insert. LZ4 holds only on the two small, highly compressible reads. One WAN probe is not enough to change a shipped default, but it is evidence T1a did not have.

HTTP's compressed reads beat the native client's on every shape, by 1.6× to 2.1×. Uncompressed the two are level or favour the native client (0.89 on the mixed hits shape), so the gap is specific to the compressed read path, and it is not explained by bytes: the layout table below has the native format sending fewer bytes on this data. Compression improves HTTP's mixed-shape read by 9.7× against a 6.49× byte reduction, while it improves the native client's by only 5.1× against its own 7.05×. The native compressed read is the one arm whose speedup falls short of its byte reduction, so something in that path is not converting the saving. Per-block framing is the leading suspect — a codec context, a CityHash checksum and a frame header per block rather than once per response — but this is unverified, and it is worth a look before the client ships.

Inserts are near parity on real data: HTTP is 21% ahead on the mixed shape, the native client 10% ahead on the numeric one. The native client's large insert win shows up only on the synthetic shape.

The columnar layout does not compress better than RowBinary on realistic data

An earlier revision of this PR claimed it did, on the strength of a synthetic shape, and that claim is withdrawn. Comparing FORMAT RowBinary against FORMAT Native for the same 1M rows under ZSTD-3, compressed in 1 MB frames so neither layout gets cross-block context the native protocol would never have:

Shape	RowBinary	Native	Columnar sends
1 × WatchID	1.00×	1.00×	1.00× (incompressible control)
10 × hits, mixed	6.49×	7.05×	1.09× fewer bytes
3 × hits String	7.70×	8.04×	1.04× fewer bytes
10 × hits, numeric	30.09×	24.39×	1.23× more bytes
3 × synthetic	4.91×	10.73×	2.19× fewer bytes

These are server-side format sizes, so no client change can move them, and they are unaffected by the re-run.

The synthetic row is the outlier, and it is the one the withdrawn claim rested on. Its columns are number, number % 100 and number / 7: three short-period sequences that compress far better once grouped by column. Real hits columns are high-cardinality — WatchID and UserID are near-random, URL and Title are varied text — so grouping them by column unlocks little the compressor was not already finding.

On ten narrow numeric columns row-major wins outright, for a reason that is a property of real traffic rather than of either format: many hits repeat an identical (CounterID, RegionID, OS, UserAgent, Resolution*, Flash*, Net*) tuple. Row-major keeps each duplicate row contiguous, so the compressor matches one long run per repeat; column-major scatters that tuple across ten distant regions and can only find per-column runs.

The frame size barely mattered — whole-stream compression reads 6.98× against the chunked 6.49× on the mixed shape — so the distortion in the withdrawn claim was the synthetic data, not the compression window.

Per-type decode — SelectColumn against TcpSelectColumn

Same 16 expressions, 500k rows, both drained without touching a value. These are no longer unequal work in the way the earlier revision of this section said they were. That text had HTTP's Read() boxing every value into a reused object[]; #499 replaced that row buffer with typed column slots, so a drain that calls no accessor now allocates nothing on the fixed-width types — 17.5 KB for 500k rows, flat across all ten of them. The allocation columns are therefore comparable on those types and are given below.

Type	HTTP	TCP	Ratio	HTTP alloc	TCP alloc
DateTime	31.9 ms	5.2 ms	0.16	18.0 KB	18.0 KB
UInt32	22.9 ms	4.4 ms	0.19	17.5 KB	12.3 KB
Date32	22.5 ms	4.5 ms	0.20	17.5 KB	13.8 KB
Int32	22.7 ms	4.7 ms	0.21	17.5 KB	12.2 KB
Nullable(Int32)	25.7 ms	5.5 ms	0.21	18.1 KB	17.5 KB
Date	22.0 ms	4.7 ms	0.21	17.5 KB	16.9 KB
Float32	21.5 ms	5.1 ms	0.24	17.5 KB	12.3 KB
Float64	35.7 ms	10.3 ms	0.29	17.5 KB	12.3 KB
Int64	34.9 ms	10.1 ms	0.29	17.5 KB	12.2 KB
UInt64	34.9 ms	10.1 ms	0.29	17.5 KB	12.3 KB
Decimal256	70.4 ms	20.5 ms	0.29	14.62 MB	31.2 KB
Array	72.1 ms	21.7 ms	0.30	22.91 MB	17.4 KB
Decimal64	47.5 ms	16.0 ms	0.34	14.62 MB	20.0 KB
Tuple	77.7 ms	39.1 ms	0.50	45.03 MB	37.3 KB
String	53.2 ms	31.4 ms	0.59	22.14 MB	12.7 KB
Decimal128	58.3 ms	128.6 ms	2.21	14.62 MB	170.37 MB

This is the section whose ratios reproduce best. Every absolute time is 20% to 40% above the earlier session's, and the ratios still land within 0.04 of it on 15 of 16 types — String 0.51 → 0.59 is the only one that moved further. Decimal128 reproduces at 2.21 against 2.19.

TCP scales with width — Int32 4.7 ms to Int64 10.1 ms is 2.1× — which HTTP does not.

On the variable-length types HTTP still allocates per value and TCP still does not, but for different reasons on each side, and neither is a protocol win. HTTP's 14.62 MB on the decimals is one decimal object per row (30 B/row) built during Read(), which typed slots do not cover; TCP's flat 12 to 37 KB is the drain loop declining to materialise strings, arrays and tuples at all. TransportRead is still the honest allocation comparison, because there both sides consume every value.

What these turned up

Decimal128 reads allocate ~357 B/row, and this reproduces exactly. 128.6 ms and 170.37 MB over 500k rows, against its own Decimal64 at 16.0 ms / 20.0 KB and Decimal256 at 20.5 ms / 31.2 KB — the same 174,459 KB the earlier session measured, to the digit. Gen0 is 6,000 per 1,000 ops where every other TCP type is zero. It is the one type in the sweep where TCP loses to HTTP, in both sessions. ClickHouseTcpDecimal(Int128 mantissa, int scale) (Numerics/ClickHouseTcpDecimal.cs:39) is documented as sign-extending to 256 bits but reaches Int256 through Int256.FromBigInteger(mantissa), and the implicit Int128 → BigInteger conversion heap-allocates once per row. The Int256(ulong, ulong, ulong, ulong) constructor next to it costs nothing. Not fixed here — this PR only measures.

#499's typed column slots reach the ADO reader and the per-type sweep, and stop there. The three places this suite can see it are HTTP's fixed-width reads (46,890 KB → 15.5 KB on 2M rows), HTTP's mixed-shape read (170,330 KB → 76,580 KB, the integer and the float stopping boxing while the string stays) and the drain loop in SelectColumn (per-value → 17.5 KB flat). It does not touch QueryAsync<T>, which materialises POCOs straight from the stream and already allocated only the objects; TransportReadPoco's 151.08 MB is identical across the two sessions. Nor does it touch the decimals, arrays, tuples or strings in the sweep, which still build one object per value.

Not TCP, and not a defect: the HTTP insert cost is async_insert. A server with async_insert = 1 and wait_for_async_insert = 1 holds an insert's response until its buffer flushes, waiting from async_insert_busy_timeout_min_ms (50 ms) upward. On 26.6.1.1193 a one-row HTTP insert costs 52 ms against 2.0 ms with the setting off, where an empty SELECT 1 round trip is 2.2 ms. It reproduces with curl, on Null/MergeTree/Memory alike and for VALUES/FORMAT CSV alike, and a buffered ExecuteNonQueryAsync literal pays it too — so it is the server's policy, not the driver's path. INSERT ... SELECT costs 2.4 ms, which is the line it falls on: async inserts apply only when the data comes from the client. Worth documenting rather than fixing, since BatchSize multiplies it and async_insert = 1 is the ClickHouse Cloud default. (Measured 2026-08-30; not repeated.)

RowBinary costs the server about three times what the native block format does. For the same 109 MiB, the same 5M rows and the same ENGINE Null target, the server burns 150 to 180 ms of CPU parsing RowBinary against 55 to 68 ms reading native blocks, and on the native path it spends most of its time waiting for the client rather than parsing. So HTTP has a server-side floor the client cannot get under, and the native protocol's insert advantage is partly the server's work, not only the client's. Measured by tagging every insert with a query_id and reading query_duration_ms, ProfileEvents['UserTimeMicroseconds'] and ProfileEvents['NetworkReceiveElapsedMicroseconds'] out of system.query_log. (Measured 2026-08-30; not repeated, and note that it predicts a larger native insert win than the re-measured TcpRows 1.04 shows.)

Where the row tier's cost is, and where it is not. The native row tier costs 11.2 ns per value above the columnar arm, 2.5 ns when the values come off POCO fields, and neither figure has a confirmed cause. Both are smaller than the earlier session's 14.4 and 7.2. Two candidates are ruled out:

• Type.IsInstanceOfType, called once per value in the gather that UntypedRowColumns.CreateBuilder compiles, is not it: removing it moved TcpRows by less than its standard deviation.
• Cache locality in the transpose is not it either, and that one was settled by a change rather than an experiment. TCP N9: POCO row insert — InsertRowsAsync<T> over a compiled per-column gather #559 replaced the whole-insert gather with one that fills a 50,000-row window per wire block, which is what the locality argument asks for. Measured back to back in the 2026-08-30 session, the tip before that change and the tip with it agreed on every arm: TcpRows 356.4 against 347.5 ms, TcpPoco 263.5 against 241.2, TcpColumnar 129.4 against 131.1. Those are that session's numbers — the same arms read 400.6, 362.6 and 233.2 here — but the comparison was same-session and back-to-back, which is what makes it valid. At three columns the row array is walked three times either way, and a 50,000-row window is still 200,000 scattered heap objects, about 5 MB of them. Whether locality pays on wide rows is a question three columns cannot answer.

That change is visible in this suite only as allocation: 5.05 KB per wire block is unchanged by it, but the default block size moved, so the same 5M-row insert allocates 519 KB as 100 blocks where it allocated 40 KB as 5. Its real subject, the peak memory of the conversion buffers, is not something a timing table shows.

Caveats

• Loopback for everything committed. Compression is priced at its cost with none of its benefit, so TcpCompression cannot rank codecs; its own docs say so. The Cloud figures above come from a one-off probe that is not part of the suite, and T1a still owns making that repeatable.
• This host is not a measurement instrument, and the re-run shows it is not even a stable one. Two physical cores run the client and the server together, and the host interferes in bursts. That is why the figures are best-case iterations, and why an arm's mean sits 15% to 40% above its best case. Within one session best-case iterations are tight: 0.03% to 6.79% across the 18 cross-transport arms. Across the two sessions they are not, and not uniformly — bulk arms 15% to 86% slower, latency 13% to 31% faster, with allocation identical wherever the code did not change.
• Ratios are more portable than times, but they are not fixed either. The per-type sweep holds within 0.04 on 14 of 16 types across sessions, and TcpPoco's insert ratio held at 0.94 exactly. Three did not: TcpRows on inserts 1.35 → 1.04, TcpBlocks on string reads 1.79 → 1.48, and the POCO read arms 0.97 → 0.85 and 0.72. Read any single ratio here as good to about ±0.1 and only the large ones as load-bearing.
• The comparison job reports, it does not gate. Nothing decides which rows matter: on a 2-core runner the same code on both sides reads as −21% to +11% at 3 iterations. Filed as T5 — a row should be flagged only when the ratio's interval clears RatioSD, and the job should say how many rows it suppressed.
• tcp-investigation is defined and unused, kept as the slot symmetric with http-investigation.
• BenchmarkDotNet's MinIterationTime advisory fires on the sub-100 ms arms. Under RunStrategy.Monitoring there is no invocation amortisation to increase; the measurements are still five orders of magnitude above timer resolution.

Verification

• Both Program guards were failing on this branch, in both modes, and both are fixed. The 2026-09-09 rebase brought three benchmark classes from main that predate this branch's conventions, and each guard caught one half of that:
  • Categorization. AdoReadPathBenchmark (Box-free ADO read path: typed column slots (stacked on #449) #499), ArrayInsertElementBoxing (perf(types): per-value allocations on UUID, Array and Decimal (reland #550) #571) and WideIntegerInsert (perf(types): two per-value allocations on the write path (reland #556) #572) arrived carrying no category, and the guard exits 1 rather than let them be silently skipped by every --anyCategories run — so the harness ran nothing at all, in either mode. All three pin one issue or one past optimization on the HTTP path and are now http-investigation.
  • Method baseline. AdoReadPathBenchmark.UntypedAccessor also arrived with a literal [Benchmark(Baseline = true)], which the second guard rejects in comparison mode — so benchmark-compare.yml, the mode every PR actually runs, stayed broken after the categories were fixed. It now uses BenchmarkModes.MethodBaseline, keeping it the method baseline locally and standing aside for the job baseline in a comparison run. Reported by Cursor Bugbot.
• Verified by running --list flat both ways — local, and with BENCHMARK_COMPARISON defined: exit 0 and 120 benchmarks each time, neither guard reporting anything.
• Category counts, from --list flat --anyCategories: http-regression 40, tcp-regression 41, cross 14, http-investigation 17, tcp-investigation 0, compression 8. They sum to the 120 the flat list reports, so every benchmark is reachable from exactly one category.
• Every loopback figure above was re-measured on 2026-09-11 at bdf1cdb8, twice, at 30 × 2 launches per run. Best-case iterations agree between the two runs to within 0.03% to 6.79% on the 18 cross-transport arms, and to within 3% on the per-type sweep and the string tiers.
• The insert ns/row slopes and the allocation model come from fitting 500k against 5M. The native model reproduces the earlier session's to the digit: 5.05 KB per wire block against 5.1, ~13 KB fixed against 13, 10 blocks at 63 to 65 KB and 100 at 518 to 520 KB.
• Where a figure changed for a reason other than the host, the reason is named: Box-free ADO read path: typed column slots (stacked on #449) #499 for HTTP's read allocation, and nothing else.
• Where the earlier session's claim did not survive, it is withdrawn in place rather than quietly replaced: the row tier being behind HTTP on inserts, the POCO arms being at parity, the block tier's allocation advantage on fixed-width reads, and the 0.70 POCO figure's attribution to the codec. HttpPoco's unstable allocation went the other way — it recurred, so that caveat is kept and strengthened.
• The async_insert finding, the server-CPU comparison, the client-CPU query_id probe and the Cloud/WAN section are from the 2026-08-30 session and were not repeated; each is marked where it appears.
• The path-to-category mapping has 12 cases covering each project, the Compression/ subdirectory rule, the harness escape hatch, and the docs/tests/examples fallbacks.
• Every new class ran end to end against a real server; three arms were reworked after their first results showed they measured something other than what their docs claimed.

Stacked on #595. Nothing here depends on the examples, so the diff to review is the commits on top of tcp/epic-t3-examples.

🤖 Generated with Claude Code

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Comparison groups ignore benchmark params

Medium Severity

AddLogicalGroupRules is called with only BenchmarkLogicalGroupRule.ByMethod, so every parameter combination of a method shares one logical group. Most new classes are parametrized (Shape, Codec, Degree, ElementsPerRow), so the comparison job can treat one param’s baseline job as the ratio baseline for the others and mix the parameter effect into the PR-versus-baseline signal.

 

^{Reviewed by Cursor Bugbot for commit 635c2ff. Configure here.}

[Copilot]

Pull request overview

Adds native TCP and cross-transport benchmarks, category-based execution, and updated comparison workflows.

Changes:

• Adds TCP read, insert, compression, latency, and concurrency benchmarks.
• Categorizes existing HTTP benchmarks and supports package-to-package comparisons.
• Updates benchmark documentation and GitHub Actions automation.

Reviewed changes

Copilot reviewed 38 out of 38 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
CONTRIBUTING.md	Documents benchmark categories and execution.
ClickHouse.Driver.Benchmark/TransportReadPoco.cs	Compares POCO reads across transports.
ClickHouse.Driver.Benchmark/TransportRead.cs	Compares transport read tiers.
ClickHouse.Driver.Benchmark/TransportLatency.cs	Measures warm request latency.
ClickHouse.Driver.Benchmark/TransportInsert.cs	Compares insert shapes across transports.
ClickHouse.Driver.Benchmark/TcpStringRead.cs	Benchmarks native string access tiers.
ClickHouse.Driver.Benchmark/TcpSelectColumn.cs	Adds native per-type read benchmarks.
ClickHouse.Driver.Benchmark/TcpReadTiers.cs	Compares native materialization tiers.
ClickHouse.Driver.Benchmark/TcpProjectedRead.cs	Measures projected column reads.
ClickHouse.Driver.Benchmark/TcpPoolConcurrency.cs	Measures pool concurrency scaling.
ClickHouse.Driver.Benchmark/TcpInsertShapes.cs	Compares native insert shapes.
ClickHouse.Driver.Benchmark/TcpConnectionCost.cs	Measures connection lifecycle costs.
ClickHouse.Driver.Benchmark/TcpCompression.cs	Benchmarks native compression codecs.
ClickHouse.Driver.Benchmark/TcpCompositeRead.cs	Measures composite access paths.
ClickHouse.Driver.Benchmark/TcpArrayWriteShape.cs	Compares dense and jagged arrays.
ClickHouse.Driver.Benchmark/SelectColumn.cs	Categorizes HTTP column benchmarks.
ClickHouse.Driver.Benchmark/ResponseDecompressionBenchmark.cs	Categorizes decompression benchmarks.
ClickHouse.Driver.Benchmark/ReadValueBenchmark.cs	Categorizes reads and adjusts baselines.
ClickHouse.Driver.Benchmark/Program.cs	Validates categories and comparison baselines.
ClickHouse.Driver.Benchmark/PocoReadBenchmark.cs	Categorizes POCO reads.
ClickHouse.Driver.Benchmark/PocoInsertColumn.cs	Categorizes POCO column inserts.
ClickHouse.Driver.Benchmark/PocoInsertBenchmark.cs	Categorizes POCO inserts.
ClickHouse.Driver.Benchmark/MultidimArrayInsert.cs	Categorizes array investigation benchmarks.
ClickHouse.Driver.Benchmark/InsertCompressionBreakdownBenchmark.cs	Categorizes compression breakdowns.
ClickHouse.Driver.Benchmark/DynamicReadBenchmark.cs	Categorizes dynamic-read investigation.
ClickHouse.Driver.Benchmark/ComparisonConfig.cs	Configures job-based comparison ratios.
ClickHouse.Driver.Benchmark/ClickHouse.Driver.Benchmark.csproj	Adds TCP references and comparison symbols.
ClickHouse.Driver.Benchmark/BulkInsertColumn.cs	Categorizes bulk inserts.
ClickHouse.Driver.Benchmark/BinaryInsertStreamingBenchmark.cs	Categorizes streaming investigation.
ClickHouse.Driver.Benchmark/BinaryInsertObjectArrayBenchmark.cs	Categorizes object-array investigation.
ClickHouse.Driver.Benchmark/BinaryInsertCompressionBenchmark.cs	Categorizes insert compression.
ClickHouse.Driver.Benchmark/BenchmarkServer.cs	Centralizes HTTP/TCP endpoints.
ClickHouse.Driver.Benchmark/BenchmarkModes.cs	Switches baseline behavior by build mode.
ClickHouse.Driver.Benchmark/BenchmarkCategories.cs	Defines benchmark categories.
ClickHouse.Driver.Benchmark/BatchQueryLineBenchmark.cs	Categorizes batch investigation.
.github/workflows/nightly.yml	Adds the nightly full benchmark suite.
.github/workflows/benchmark.yml	Adds TCP and category selection.
.github/workflows/benchmark-compare.yml	Selects baselines/categories and runs comparisons.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.

There are 3 total unresolved issues (including 1 from previous review).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 9368b70. Configure here.}

[cursor]

Nightly workflow path is invalid

High Severity

The new benchmark-full job sets uses to $/.github/workflows/benchmark.yml, which is not a valid reusable-workflow path. GitHub Actions cannot resolve it, so the nightly full suite never starts.

 

^{Reviewed by Cursor Bugbot for commit 9368b70. Configure here.}