Native: one schedule's persistent database error stalls every schedule, the repair pass and readiness #955

Found while answering General's read-only question for Workflow #875 (native schedule bounds). Native 3.0.x, source 923d45b3.

Mechanism

• The maintenance loop (execution.rs TimerScheduler::start, every 250 ms when DW_RUST_MAINTENANCE_LOOP=1, which the image sets) runs fire_due_timers()?, then fire_due_schedules()?, then the repair pass.
• fire_due_schedules (schedule_execution.rs) fires up to 100 due schedules ordered by next_fire_at, id.
  • A non-database error goes to record_failure, which advances next_fire_at past now. Good: that's bounded.
  • A RuntimeError::Database error returns from the whole tick (Err(error @ RuntimeError::Database(_)) => return Err(error)). The schedule keeps its stored occurrence, by design, so a transient error retries from durable state. The unit test failed_due_admission_rolls_back_start_and_retains_original_occurrence_and_quota pins that.
• The loop sets healthy=false on any failed tick, and schema_ready() (/api/ready) requires healthy.

Consequence if the error is persistent

If one due schedule's start fails with a database-class error every time, that schedule stays at the head of the due ordering. Every tick on every node then:

1. fails at that schedule, so later due schedules in the batch never fire;
2. skips the repair pass (run deadlines, task repair, activity and service-call timeouts), which runs after schedules;
3. leaves /api/ready answering 503, so the whole server drops out of rotation.

Timers still fire, because they run first.

Not yet established

I haven't found a natural input that makes a schedule start fail persistently with a database-class error (rather than a validation refusal). The existing test reaches the path only with an injected SQLite trigger. Candidates worth checking: column length or truncation limits for generated workflow ids or memo/search-attribute values on MySQL/PostgreSQL, and constraint conflicts with existing instance ids.

Expected

A persistent failure in one schedule must not stop other schedules, the repair pass or readiness. For example: retry a database-class failure a bounded number of times, then record it via record_failure and continue the batch, keeping the overall tick error only for a real storage outage. Behavior for transient errors (retry from durable state, occurrence kept) should stay as it is.

Not in 3.0.1.