Scheduler: in-repo election, failover, and catch-up test coverage

[jcohen-hdb]

Closes #1866.

Summary

Adds unit coverage for the scheduler engine's election/heartbeat/failover/catch-up orchestration (electRole, becomeLeader, heartbeat, failoverCheck, runCatchUp, executeJob) — previously backed in this repo only by pure-helper tests, with true-network validation living in the harper-pro cluster harness.

Approach — same-process simulated peer (the deploy-tracking-peer-branch.test.ts pattern): true multi-node replication isn't available in this repo (replication ships as a separate component), but the engine's entire knowledge of other nodes flows through exactly two observables — server.nodes (normally populated by the replication component) and rows in hdb_scheduler_state (normally replicated). The test supplies both, against the real engine module and a real store, no sinon. Staleness is data-driven (a lease heartbeat stamped past the threshold is a dead leader), so nothing waits out production intervals; heartbeat/failover ticks are driven through three new @internal pass-through seams (electionSettledForTests, runHeartbeatForTests, runFailoverCheckForTests — same pattern as setCoolingFunctionForTests).

Scenarios (a single node walked through its lifecycle): sticky-election deferral to a fresh peer lease → no premature promotion → stale-lease promotion with catch-up firing the missed occurrence (catchUp: true asserted) → lease renewal → step-down when a heartbeat finds a fresh competing lease → cold-start election win → roster-driven cold-start election loss (the one outcome that only passes if nodeRoster()/pickNextLeader wiring is correct, not just the pure function) → new-job baseline without immediate fire → registration replacement (upgrading a previously assertion-free test).

Where to focus review

• The three engine seams are intended as pure pass-throughs with zero production behavior change (one held reference to an already-created election promise, reset in stopSchedulerEngine).
• The lifecycle tests are deliberately sequential (mocha serial order); each later test states its precondition.
• What this deliberately does NOT cover: true concurrent-write lease races and network-partition behavior — those remain in the harper-pro cluster harness (2-node kill/failover test), which is planned as a companion contribution there.

A Harper-domain review pass ran pre-PR; its findings (inert roster stub, state-row cleanup, one overclaiming comment) are addressed in the second commit. Note: npm run build on current main reports pre-existing type errors in agent/bestPractices.ts and resources/DatabaseTransaction.ts — unrelated to this diff, which adds no TS errors.

Refs #1828, #951.

Generated by an LLM (Claude Fable 5) pairing with @jcohen-hdb.

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request introduces in-repo test coverage for the scheduler engine's election, heartbeat, and failover orchestration, adding test-only hooks to resources/scheduler/engine.ts and a comprehensive test suite in unitTests/resources/scheduler/engineElection.test.js. The feedback highlights two important improvements: first, addressing a potential race condition where an in-flight election promise could resolve after the engine is stopped, leaking timers; and second, replacing a flaky real-time delay in the heartbeat test with a deterministic backdated timestamp.