Skip to content

[factory] Readiness reconciliation can consume ~95% duty cycle when fallback hydration exceeds its interval #284

Description

@khaliqgant

Problem

The readiness reconcile interval defaults to 60 seconds, while a measured GitHub readiness sweep took 1,173,071 ms (19.5 minutes) because legacy _index.json rows omit labels and Factory hydrates roughly 1,400 issues individually.

The implementation does not overlap sweeps, which is good: #scheduleReadinessReconcile() refuses an existing timer/in-flight sweep, and it schedules the next timer only from the prior sweep finalizer (src/orchestrator/factory.ts:1417-1436 at a63cf96). The discovery lease is a second fence.

However, that scheduling shape means the effective cycle under the measured fallback is approximately:

1,173,071 ms work + 60,000 ms delay = 1,233,071 ms

or about 95.1% reconciliation duty cycle. It is not literal overlap and not exactly zero-gap back-to-back execution, but it is effectively near-continuous source-of-truth scanning. The intended relationship — events as the fast path and reconciliation as a periodic correctness backstop — becomes operationally inverted while the legacy index fallback is active.

Operator symptom

The daemon spends nearly all wall time in issue enumeration/hydration, Relayfile load stays elevated, and the next reconcile begins one minute after a 19.5-minute sweep completes. This can mask the event-driven architecture and makes overload/backoff behavior harder to distinguish from a dispatch outage.

Scope

Do not solve this in the cloud-host cutover. cloud#3052 / adapter-github 0.5.5 restores labels for newly written index rows; the legacy-row backfill is tracked separately. Once index fidelity is restored, remeasure before changing scheduling semantics.

Acceptance

  • Export/alert on reconcile duration and duty cycle (or an equivalent operator-visible saturation signal).
  • Prove long reconciles remain non-overlapping.
  • After the legacy labels backfill, attach a new measured duration across the configured repo set.
  • If duration can still exceed the intended interval, define adaptive scheduling/backoff without weakening the source-of-truth correctness sweep.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions