cluster: a saturated dispatch queue has no documented retryable class — clients cannot back off correctly; add one and a counter #371

Version / build tested against

origin/main @ 1ff3551

Deployment mode

Origin, single node (local)

Engine(s) involved

cluster, pgwire

Summary

When the dispatch queue is saturated, a client cannot tell that condition apart from a
generic failure, and the server treats it as one. The bridge WFQ returns
Error::Dispatch { detail: "core <id>: total WFQ capacity exhausted" }
(nodedb/src/bridge/dispatch/dispatcher.rs), and Error::Dispatch is listed in
is_unclassified_failure (nodedb/src/error_classify.rs:298), so a caller cannot match
it by variant and has no retry contract.

There is already a house pattern to copy. VShardAdmissionCapacityExceeded classifies
to rate_exceeded("vshard_admission", ...) (nodedb/src/error_classify.rs:120), which is
a matchable retryable class. The WFQ and Calvin scheduler dispatch capacity paths should
follow the same shape, plus a counter. The Calvin scheduler channel capacity is
driver/config.rs:11 (default 512).

I know PR #355 was closed because its approach on the server side was wrong, and that
the fix is being done in-tree so a busy dispatch keeps the transaction pending and
re-dispatches it. This issue is only about the client-visible surface: when the pending
path is not taken or a refusal still reaches the client, the code must be a retryable
one, and it must be documented.

Expected behavior

• A saturated dispatch queue surfaces as a retryable class, either the existing 53200
  or a dedicated code, and the docs say which.
• The class is distinct from generic internal errors, so a client can back off instead
  of aborting work.
• A counter exists for it (nodedb_dispatch_capacity_busy_total or similar), so
  operators can see pressure instead of guessing.

Actual behavior

Today a client sees a catalog of error classes where capacity pressure is not
separately documented, and there is no counter for it. A loader therefore paces
conservatively or retries blind. I hit this while draining 66k edges: the correct
action after a refusal was always "wait and resend", but nothing on the wire said so.

What actually happened? (check all that are true)

• Acknowledged/committed data was lost, corrupted, or silently wrong
• The server crashed, hung, or failed to start
• A security or isolation boundary was crossed
• Core functionality is broken with no acceptable workaround
• A workaround exists (blind retry with conservative pacing)

Proposed severity

SEV-3, small surface change, meaningful for every bulk writer.

Reproducibility

Under sustained write pressure.

Last known-good version / commit (if a regression)

Not a regression.

Environment & logs

Linux x86_64, single-node origin, trust mode. Related: #338 covers the SQLSTATE mapper
coverage for routed errors; this issue is the capacity-specific case inside that family.

Before submitting

• I searched existing issues and this is not a duplicate (Routed response-shaper errors flatten to XX000; the numeric SQLSTATE mapper covers 20 of 81 codes #338 is adjacent, not the same).
• I reproduced this on current main.
• This is not a security vulnerability.

Consolidated from #372 (item 2)

#372 asked for the same counter this issue asks for. That request now lives here; #372 stays open for its other item, the per-engine write counter (nodedb_graph_edges_written_total).

From #372:

2. A counter for dispatch capacity busy, which pairs with the signal request in the
   P4 issue above (nodedb_dispatch_capacity_busy_total).

Status on fix/calvin-dispatch-capacity: Error::DispatchCapacity maps to 57P03 (SERVER_OVERLOAD), and a per-vShard nodedb_calvin_dispatch_deferred_total exists (calvin/scheduler/metrics_flow.rs:18). Still missing: a process-wide counter for capacity refusals that reach clients, and the docs (nodedb-docs/docs/reference/error-codes.rdx:50 lists SERVER_OVERLOAD as 7000; the code says 1403, and 57P03 is not documented).