synchronizers: a transient start failure is FAILED and retried, never FATAL (#7248)

[delchev]

Fixes #7248.

The defect

A .listener / listener.ts artefact whose subscription fails while the embedded broker is still starting is retried exactly once, in the same synchronization pass. On the next pass the START phase finds it FAILED and registers FATAL; from the pass after that, SynchronizationProcessor.parseDefinitions strips every FATAL artefact from the set before any phase runs. Nothing brings the listener back until the file's bytes change, and since a topic discards what it delivers to nobody, every message published to that destination afterwards is silently lost.

FATAL - "this artefact is broken, stop trying" - was being reached by a transient failure of a collaborator that is expected to be briefly unavailable at exactly that moment (with a shared message store the broker may not take its lease for far longer than the first second of boot, and EmbeddedBrokerMessagingConfig already promises that a listener started before mastery "fails and is retried"). Same root cause as #7217 (the client-Java path, fixed in #7220), different module, different mechanism for giving up.

The same pattern elsewhere

Asked to check the other artefacts, I grepped every registerState(..., FATAL, ...) in the synchronizers. Four sites, all the same shape - a second failure of a collaborator promoted to FATAL:

Synchronizer	Collaborator that may be late	Where it went FATAL
ListenerSynchronizer	the embedded broker; the handler file	START on a FAILED listener (pass 2)
JobSynchronizer	the Quartz scheduler / its store	START on a FAILED job (pass 2)
CamelSynchronizer	a bean or data source a route needs	START on a FAILED route - within the very first pass: the outer catch records a refused create FAILED and returns false, the in-pass cross-retry reaches START, and the guard fires
SchemasSynchronizer	the .datasource a schema names, possibly in a project published on a later pass	the second unresolved data source lookup

Every other synchronizer with a START phase already records FAILED and keeps retrying; registerFatals is defined and called by nothing.

The fix

The issue's first, most surgical option, applied to all four: a failure of a collaborator is FAILED, retried every pass, and never FATAL - the same philosophy case BROKEN: in parseDefinitions already applies one level down (a definition that failed to parse is re-parsed every pass because "the original failure may have been transient"), and the shape ViewsSynchronizer adopted in #6942.

• Listener / Job: the CREATE branch records a start that threw as FAILED with its cause and returns false (it used to register CREATED with running = false - the issue's first "smaller thing" - and the job branch registered CREATED without even the cause). The START phase drops the FAILED -> FATAL guard, retries the start while the artefact is FAILED or not running, heals it to CREATED on success (so the Registry and Problems views stop showing a stale error) and stays FAILED + return false while refused.
• Camel: the START guard is dropped; the existing addToProcessor + CREATED then is the retry.
• Schemas: the FATAL promotion block is dropped; the existing FAILED + return false is the retry.
• ListenersManager.startListener throws for a missing handler instead of logging one ERROR and returning normally - the issue's second "smaller thing": the normal return had the synchronizer record the artefact CREATED and running with nothing subscribed. A handler is also something a later pass may publish, so FAILED is the right reading for it too.

The second layer the IT found: an idle instance never ran a retry at all. processSynchronizers runs its phases only when the pass carries a NEW or MODIFIED artefact (or is the boot pass), and isSynchronizationNeeded admits a pass only on a registry change - so with the FATAL guard gone, a FAILED listener was still retried only when someone published something else. The processor now gives every FAILED artefact one START attempt per pass: on any pass that runs without new or modified artefacts (retryFailed), and on an idle instance every DIRIGIBLE_SYNCHRONIZER_FAILED_RETRY_INTERVAL_SECONDS (default 30) through a private gate (isFailedRetryDue). The public isSynchronizationNeeded keeps meaning "the registry changed" - the test framework's waitForStableSynchronization polls it for a ten-second quiet window, which is also why the retry cadence is its own key and not the ten-second job tick. No cross-retry loop on that path: a permanently failing artefact costs one refused call per interval, and registerState logs a repeat of the same error at DEBUG instead of stack-tracing at ERROR each time (the per-pass "Error occured during synchronization" summary line remains - one line per interval per broken artefact, the operator signal the single boot-time ERROR never was).

Per-tenant correctness needed one more line. MultitenantBaseSynchronizer completes one shared artefact once per tenant and resets only the lifecycle between tenants, not running - so a START gate on running alone lets the first tenant to subscribe flip the flag and skip every tenant after it (a pre-existing weakness of the START phase this retry would have inherited). The gate is therefore FAILED || !running; both managers are idempotent per tenant (LISTENERS.containsKey, scheduleJob's checkExists), which is what makes the repeated attempt safe.

Cost, stated: a permanently failing artefact now stays FAILED instead of being stripped, so it occupies the in-pass cross-retry loop (DIRIGIBLE_SYNCHRONIZER_CROSS_RETRY_COUNT x _INTERVAL_MILLIS, 10 x 10 s by default) on every pass - exactly the cost a FAILED view already carries since #6942, and the cost the UPDATE phase's if (FAILED) return false was always paying between the first and the second pass. Artefacts already persisted as FATAL by an earlier version stay stripped until republished (the file's bytes must change once).

The .claude/docs/synchronizer-model.md guide gains a paragraph on this.

Verification

• ListenerArtefactSubscriptionRetryIT (new, HTTP-level, in the smoke set) - the end-to-end guard, modelled on JavaListenerSubscriptionRetryIT. A @MockitoSpyBean ActiveMQConnectionArtifactsFactory refuses the connection whose exception handler carries the probe's handler path (so the platform's own messaging is untouched), a .listener on a topic is deployed, and the artefact row is asserted across passes: FAILED and not running after a forced pass 1 (the refusal actually happened), still FAILED after a forced pass 2 (the pass that used to register FATAL), then - with the refusal lifted and nothing forced or published from there on - CREATED and running within the idle instance's own retry (the passes the artefact used to be stripped from, and which used to run no phase at all). Then a real round trip through the JS handler. Cross-retry count/interval and the FAILED retry interval are shortened for the class and restored. Verified to fail without the processor change (the artefact stayed FAILED with no start attempted on the later passes).
• ListenerSynchronizerRetryTest (4), JobSynchronizerRetryTest (3), CamelSynchronizerRetryTest (2), SchemasSynchronizerRetryTest (1) - new unit tests in the style of ViewsSynchronizerRetryTest, each asserting the refused start reads FAILED not CREATED, the second refusal stays FAILED not FATAL, and the healed artefact reads CREATED with no stale error. ListenersManagerTest.testStartListenerOnMissingListener now asserts the throw.
• Unit suites of engine-listeners, engine-jobs, engine-camel, data-structures, core-initializers, commons-config: green.
• mvn formatter:format on the changed modules, formatter cache wiped, mvn -T 1C formatter:validate: BUILD SUCCESS.

Not verified: a real multi-tenant instance with a second provisioned tenant (the per-tenant gate is reasoned from MultitenantBaseSynchronizer.completeInternal and covered by unit tests only), and the Job / Camel / Schema retries end to end (unit-tested at the synchronizer boundary).

🤖 Generated with Claude Code