[mikehorgan-dell]

Summary

Choosing the s3-rdma driver doesn't prove that payloads moved over RDMA. An operation can be below the threshold, too large for one buffer, declined by the server, or sent over HTTP after RDMA preparation failed. The driver already kept counts, but they left the engine only as one log line at driver close. The CLI read nothing about the driver, so a run that sent every payload over HTTP produced a normal-looking summary.

This adds a generic driver statistics contract: a storage driver can report named counters and gauges about itself. The engine carries them through the metrics rows, fleet totals, terminal retention, the stored results and the Prometheus endpoint. The s3-rdma driver is the first to report them.

• JSON: rows of /metrics/json and /metrics/fleet/json gain an optional driver_stats object.

  • A node row carries its driver's counters and gauges.
  • A fleet row sums its nodes and adds reporting, the number of nodes in the sum, and partial.
  • When nodes report different drivers there are no sums: the object carries only partial: true and the drivers seen.
  • The published metrics schema 5 now describes the object, with an example.

• s3-rdma statistics:

  • RDMA requests sent and the outcome of each: transferred, declined (served over HTTP, or failed), failed, protocol error, timed out;
  • payload requests sent over HTTP, by reason;
  • pooled and per-operation buffer registrations;
  • registration, allocation and token failures;
  • the buffer pool's levels.

• Run summary: each step in which the driver sent payload requests gets one block, for example RDMA Data Path — Read: 237030 transfers reported over RDMA, 0 payload requests sent over HTTP. It adds one line for each problem count above zero. The summary warns when a step moved no payload over RDMA, and when a run with --s3-driver rdma has no statistics. It shows counts only, no percentage.

• Stored results: each step's final statistics go to spt_run_params.json under driverStats. The CLI reads them while it monitors the run, as each step ends. It stores them only when they are complete; otherwise it records why in driverStatsError. It never stores partial statistics.

• Prometheus: while a step runs, the entry node's /metrics exports the step's fleet total:

  • spt_driver_<counter>_total counters;
  • spt_driver_<gauge> gauges;
  • spt_driver_reporting and spt_driver_partial;
  • a driver label.

  A counter never falls below a value it showed, and the series end with the step.

Changes

• Engine core (spt-base):
  • StorageDriver gains a default method that returns a statistics snapshot, null by default.
  • LoadStepContextImpl registers one supplier per driver.
  • MetricsContextImpl collects it inside the existing 100 ms snapshot guard, and on a forced refresh.
  • DistributedMetricsContextImpl builds DriverStatsTotal under the fleet rules.
  • TerminalStepEntry and MetricsManagerImpl keep a finished step's final statistics, including for a mixed step whose carrier context ends first, and for a step whose driver counted work but completed nothing.
  • MetricsJsonResponder writes driver_stats, and PrometheusMetricsExporterImpl, through DriverStatsCollector, exports the fleet total.
  • MetricsContextImpl and DistributedMetricsContextImpl refresh one at a time. MetricsManagerImpl.unregister takes a step's final values with the new MetricsContext.terminalSnapshot(). A refresh already in flight finishes first, and no later one replaces the result, so a scrape or poll overlapping the step's end can no longer leave earlier counts in the retained entry marked complete (final review R1, 46034e07).
• s3-rdma:
  • RdmaPathStats splits the declined outcome and adds request, registration and failure counters.
  • RdmaBufferPool reports its counters and levels in one read under its monitor, plus a peak.
  • RdmaDriverStats maps them to the public names.
• CLI:
  • capture and validation while the run is monitored (run_driver_stats.go);
  • storage in spt_run_params.json;
  • driverstats model and RDMA view;
  • summary block and warnings; spt replay captures the statistics too.
• Docs:
  • cli/docs/S3_RDMA.md: every statistic, what it counts and what it must not be read as;
  • engine METRICS.md: the driver_stats object and the Prometheus series;
  • metrics-schema-v5.json and a new example;
  • CHANGELOG.md under Unreleased › Added and › Fixed.
• Merge: the branch started from 5.16.0, and main at 5bed9172 is merged into it in 74fbb0a6. In the metrics classes the snapshot and terminal-entry constructors now carry both main's steady-state result and the driver statistics. Each side's earlier constructor is kept and delegates to the new one.

Compatibility

• Additive. A driver that reports nothing changes nothing: no driver_stats key, no summary line, no stored field.
• metrics_schema stays 5, and driver_stats is an optional field of schema 5, which is unreleased.
• Results stored by an older CLI, and rows from an older engine, load and render as before.
• New default methods StorageDriver.driverStats() and MetricsContext.terminalSnapshot(), and default accessors on AllMetricsSnapshot and DistributedAllMetricsSnapshot. Existing public constructors of the snapshot classes and of TerminalStepEntry are kept.
• The RDMA data path summary log line keeps its format.

Notes

• Weighted and pipeline steps run one driver per operation. While such a step runs, each driver has its own row. After it ends, the row kept carries the statistics of one driver, not their sum. The CLI generates no such step with a reporting driver. This is documented in S3_RDMA.md.
• Pre-existing behaviors left unchanged, documented where they matter:
  • with --s3-driver rdma, the CLI passes /dev/infiniband to the container, so --rdma-fallback cannot start on a host without it;
  • starting the driver loads the host's RDMA kernel modules on demand.
• Performance: the maintainer accepted the Class B evidence for this merge: the fake-transport harness, and an A/B on real S3-RDMA hardware. Throughput was at parity. Engine CPU per GiB/s was +2.9% in both medians, inside main's own 8–10% spread. A full performance run on RDMA hardware comes before the 5.17 release, and performance is not described as qualified until then.
• The real-RDMA runs are single-host, because only one RDMA client was available. Fleet summing doesn't depend on the data path and was qualified on three hosts.