[improve][test] Create the IoT gateways' producers concurrently in a random order

[lhotari]

Motivation

In the Pulsar Performance Testing Framework's high-rate and max-rate IoT telemetry scenarios, the gateways precreate their producers one at a time, in the order of the gateways. Each gateway has a Pulsar client of its own, which first connects to the service URL for its lookup and then to the topic's broker. In the test cluster, the service URL and the topics' owner are the same broker.

So the broker accepted the gateways' connections in strict alternation: a lookup connection, then a data connection, and so on. The broker assigns the connections that it accepts to its I/O threads in turn, so every data connection landed on every other I/O thread:

• With the default numIOThreads (32 on the test host), 16 of the 32 I/O threads served all the traffic, and the other 16 were idle at about 1 %.
• With 8 I/O threads, 4 served it.

Real gateways connect independently and many at the same time, for example when they reconnect after a broker restart. The measurements therefore came from a connection layout that real clients don't produce.

Modifications

• gateways.producer.precreateConcurrency (32): with precreate: true, the gateways create their producers in a random order, up to that many at a time. The order is the same for every run of a scenario.
  • precreateConcurrency: 1 creates them one at a time in the order of the gateways, as before. It reproduces runs made before this change.
  • Validation: an explicit value must be at least 1. A scenario or a saved resolved configuration without the setting gets 32.
• Failures: after a producer creation fails, no more are started, and the first failure is thrown once the started ones have completed. All the created producers are closed with the others.
• Docs: the scenario docs explain the setting. They also say that concurrent connections spread the data connections over the I/O threads on average, not evenly, and that the lazy path (precreate: false) still creates producers one at a time.
• Tests: a test of the creation order, and a test of the validation.

Measurements

The iot-telemetry-max-rate scenario at 128 B: 500 gateways publish unbatched messages to one topic without a rate limit, with up to 100,000 in flight, and 20 consumers receive them.

• Host: Intel i9-9980HK, 8 cores / 16 threads, fixed at 2.4 GHz.
• Disk: the bookies' journal on a tmpfs, and the write barriers off ([improve][test] Add disk-limit options and A/B comparison charts to the Pulsar Performance Testing Framework #26800).
• Runs: each combination of one at a time or concurrent, and the broker's numIOThreads 32 (the default on this host) or 8 (the default that [improve][broker] Default numIOThreads to half the available processors, at least 8 #26825 proposes), run twice, interleaved.
• Per-thread CPU: sampled from the host's /proc for 15 s of each measurement.
• Correctness: every run delivered every message without duplicates or ordering violations.

Max rate 128 B, means of 2 (min–max)	One at a time (before)	Concurrent (this change)	Change
numIOThreads 32: I/O threads above 5 % of a core	16 of 32	32 of 32
numIOThreads 32: throughput	105,263 msg/s (104.4k–106.2k)	91,660 msg/s (90.1k–93.2k)	−12.9 %
numIOThreads 32: broker CPU per million messages	55.9 s	63.1 s	+12.9 %
numIOThreads 8: I/O threads above 5 % of a core	4 of 8	8 of 8
numIOThreads 8: throughput	119,432 msg/s (119.1k–119.7k)	101,925 msg/s (101.7k–102.2k)	−14.7 %
numIOThreads 8: broker CPU per million messages	40.7 s	49.2 s	+20.9 %

• The load spreads over every I/O thread, as with independently connecting clients.
• Throughput falls by 13–15 %, and the broker's CPU per message rises. Half of the I/O threads doing all the work was more efficient than all of them sharing it: busier event loops handle more work per wakeup, and the managed-ledger thread, which hands every publish receipt to a producer's event loop, wakes sleeping loops less often. The earlier layout flattered the scenario's results.
• Comparing with earlier runs: this changes the scenarios' results, so compare runs made with the same setting, and use precreateConcurrency: 1 to compare with runs made before this change.

The broker's I/O threads, numIOThreads 32

The broker's I/O threads, numIOThreads 8

Throughput, numIOThreads 32: one at a time (A, before) and concurrent (B, this change), with the same axes

Verifying this change

• Make sure that the change passes the CI checks.

This change added tests and can be verified as follows:

• TelemetryProducerTest:
  • The creation order: in gateway order for a concurrency of 1, and a repeatable random order otherwise.
  • With controlled futures: at most precreateConcurrency creations at a time; no new creations after a failure; every started creation completes before the first failure is thrown.
• IotScenarioTest checks that precreateConcurrency below 1 is rejected and that an omitted one is 32.
• ScenarioFilesTest resolves the scenarios with the new setting.

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

• Dependencies (add or upgrade a dependency)
• The public API
• The schema
• The default values of configurations
• The threading model
• The binary protocol
• The REST endpoints
• The admin CLI options
• The metrics
• Anything that affects deployment

This change was prepared with the assistance of Claude Code (claude-opus-5-5); I have reviewed and verified it.