fix(cluster): treat a full dispatch queue as retryable capacity and pace the startup rebuild

[EnRaiha]

Why

A full per-tenant bridge queue returned Error::Dispatch; the Calvin scheduler
logged it at ERROR and completed the transaction as failed, releasing its locks.
During the startup rebuild the epoch is re-driven, so a full queue became a spin:
33,435 dispatch failed ERRORs in 60 s, CPU pinned, pgwire starved. It
reproduced on the production server with a build derived from main: the node came
up and never served.

What

Commit	Change
df9c3b818	Error::DispatchCapacityBusy { tenant_id, inflight, cap } — retryable by type. The dispatcher returns it for the per-tenant cap; the epoch dispatch demotes the log to debug, counts it (dispatch_busy_count), and paces the catch-up drain with busy_backoff (5 ms doubling to a 250 ms cap, deterministic jitter)
ae60781c4	the same handling at every dispatch site: CalvinResolve, commit-resolution, write-version record, and the active epoch dispatch (note_dispatch_busy / note_dispatch_busy_shared)
c5a61d025	RAFT_READY_STALL_TIMEOUT 30 s → 300 s: with a large apply backlog the metadata group needs minutes, and 30 s turned a slow boot into a restart loop
eb4f0b295	DATA_GROUP_RECOVERY_TIMEOUT 60 s → 600 s: restores the value the production config carried before the bound became a hard-coded constant
dbfccd46c	drop an issue reference from a touched comment (house preflight)

The gateway maps the new variant to BUSY, so a client still sees a retryable
overload rather than an internal failure, and the tracker entry is cancelled on
the deferred path.

How to test

• full_queue_is_a_retryable_capacity_condition — new; fails on main (the error
  was Error::Dispatch), passes with the fix
• full_queue_returns_error — unchanged: a full queue is still an error
• busy_backoff_is_bounded_and_jittered — bound ≤ 257 ms, jitter varies with
  vShard and epoch

Validation

• Production deploy of a build carrying these commits: serving, zero
  storm-class ERRORs across the monitored window, SELECT 1 returns, the
  knowledge-graph store is intact (70,978 rows).
• Before the fix, the same workload on the same server produced the 33k-error
  storm and no service.
• The startup path was verified with the raised bounds on a backlogged recovery;
  the previous hard-coded bounds aborted the start with
  metadata group applied no entry for 30s and
  data raft group recovery timeout after 60s.

Notes

• Check C5 (500 non-test lines per changed file) is green without a bypass:
  this branch splits the busy accounting into core/busy.rs and the active
  dispatch into core/active_dispatch.rs (dispatch.rs 543 → 387,
  scheduler.rs 508 → 459 non-test lines), and drops variant docs in
  error/types.rs that restated their display messages, returning it below its
  base count (601 → 586). The enum itself cannot go under 500 without changing
  the public API, so C5 treats it as the pre-existing over-limit file it is:
  no growth.
• The two raised bounds should become configuration again: issue feat(config): make the startup stall and data-group recovery bounds configurable #354.
• Files carrying pre-existing issue numbers elsewhere in the tree are untouched.

Fixes #352
Fixes #353

Tradeoffs

Tradeoff: error/types.rs cannot reach 500 non-test lines while its enum stays one type, so C5 treats it as a pre-existing over-limit file that must not grow; the refactor keeps it below its base instead.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[farhan-syah]

Closing. On a capacity-busy dispatch, static_dispatch.rs and active_dispatch.rs still call on_txn_complete. That releases the locks and marks the position applied, but the transaction never ran on that replica. Demoting the log to debug makes that loss invisible instead of fixing it. The two startup bounds are progress/stall bounds. The failures cited (no applied entry for 30 s; 261 of 262 entries) are stalls, not slow replays. No [server] data_group_recovery_timeout_ms field ever existed in the repo history. The maintainers will fix this in-tree: a busy dispatch keeps the transaction pending and re-dispatches it.