From 492246f45aac6385ec14f9f229865fbff04426f4 Mon Sep 17 00:00:00 2001 From: Iliyan Velichkov Date: Fri, 11 Sep 2026 15:17:46 +0300 Subject: [PATCH 1/2] registry: a deep registry write schedules a synchronization pass (#7303) LocalRegistryWatcher watched /registry/public recursively and handed every event to List, an interface with zero implementations - so each instance started a thread, opened a watch service, walked the whole registry once and kept polling it, to notify nobody. Next to it sits SynchronizationWatcher, which decides whether a pass runs at all and registers the registry root NON-recursively: a file written several folders deep produces no event, so nothing is ever scheduled for it. Only a publish (SynchronizationWatcherPublisherHandler forces a pass) and the external-folder copy (#7299) reported themselves; anything else was invisible for the life of the process - the shape of #7192, a partial client-Java generation that stayed installed with every source on disk. So the SPI is dropped and the watcher is given the one job it was next to all along: it marks the registry modified on every create/modify/delete under it, which makes any deep writer self-healing. The six fan-out loops and the boot-time walk go with the interface - a pass reconciles the whole registry, so a per-file notification is not needed and the walk was only there to feed one. DIRIGIBLE_REGISTRY_LOCAL_IGNORED_FOLDERS now means something observable: those top-level folders are neither watched nor reported. This does not retire RegistryMutationTracker. A pass is now scheduled while a multi-file write is still arriving, and only the bracket tells that pass to defer its cleanup instead of reaping artefacts whose sources have not landed yet - a component that writes the registry outside the publisher pipeline still brackets the write, it just no longer has to remember to announce it. Two details that are load-bearing. A new folder is REGISTERED before it is reported: a file written into it before the registration produces no event of its own and is only covered because the pass the report schedules walks the subtree afterwards. And an OVERFLOW now reports instead of being skipped - events were dropped, so what changed is unknown, which is when a pass is most needed. Co-Authored-By: Claude Opus 5 (1M context) --- .claude/docs/synchronizer-model.md | 4 +- CLAUDE_FEATURES.md | 1 - .../watcher/LocalRegistryWatcher.java | 247 +++++------------- .../watcher/LocalRegistryWatcherHandler.java | 61 ----- .../LocalRegistryWatcherShutdownTest.java | 4 +- .../watcher/LocalRegistryWatcherTest.java | 145 ++++++++++ .../tests/api/RegistryDeepWriteSyncIT.java | 135 ++++++++++ 7 files changed, 355 insertions(+), 242 deletions(-) delete mode 100644 components/core/core-registry/src/main/java/org/eclipse/dirigible/components/registry/watcher/LocalRegistryWatcherHandler.java create mode 100644 components/core/core-registry/src/test/java/org/eclipse/dirigible/components/registry/watcher/LocalRegistryWatcherTest.java create mode 100644 tests/tests-integrations/src/main/java/org/eclipse/dirigible/integration/tests/api/RegistryDeepWriteSyncIT.java diff --git a/.claude/docs/synchronizer-model.md b/.claude/docs/synchronizer-model.md index d3c5ff67011..8f64feff15b 100644 --- a/.claude/docs/synchronizer-model.md +++ b/.claude/docs/synchronizer-model.md @@ -12,6 +12,8 @@ Existing synchronizer implementations (grep `extends BaseSynchronizer` / `extend JS/TS user code is **not** synchronized — it is loaded on demand by `engine-javascript` (`JavascriptEndpoint` at `/services/js/...`, `/public/js/...`) via `DirigibleJavascriptCodeRunner` backed by Graalium/GraalVM polyglot. The `api-*` Java modules under `components/api/` register the JS-callable APIs (`@dirigible/db`, `@dirigible/http`, etc.) into the GraalJS context — pre-built TS/JS bundles for those APIs live in `components/api/api-modules-javascript/src/main/resources/META-INF/dirigible/modules/`. -**A pass only runs when something says the registry changed — and the file-system watcher is shallow.** `SynchronizationProcessor.processSynchronizers()` returns immediately unless `SynchronizationWatcher.isModified()`, and that watcher registers **`/registry/public` itself, non-recursively**: a write several folders deep produces no event on Linux at all. A publish is fine — `SynchronizationWatcherPublisherHandler.afterPublish` forces a pass — but anything else that writes the registry must do the same, and must additionally bracket the write with `RegistryMutationTracker` so a pass looking into a half-applied copy does not reap artefacts whose sources are still arriving. `RecursiveFolderWatcher` (`DIRIGIBLE_REGISTRY_EXTERNAL_FOLDER`) did neither, so a boot that lost the race against the copy installed a partial client-Java generation that was never rebuilt — controllers answering 404 for the life of the process with all their sources on disk (#7192). `RegistryMutationTracker`'s javadoc states the rule: bracket the write, do not enumerate callers. +**A pass only runs when something says the registry changed, and a deep write says it through `LocalRegistryWatcher`.** `SynchronizationProcessor.processSynchronizers()` returns immediately unless `SynchronizationWatcher.isModified()`, and that watcher registers **`/registry/public` itself, non-recursively**: a write several folders deep produces no event on Linux at all. A publish was fine — `SynchronizationWatcherPublisherHandler.afterPublish` forces a pass — but anything else that wrote the registry was invisible for the life of the process, which is how a boot that lost the race against the external-folder copy installed a partial client-Java generation that was never rebuilt: controllers answering 404 with all their sources on disk (#7192, fixed at that writer by #7299). `LocalRegistryWatcher` (`core-registry`) now closes the gap generically — it watches `/registry/public` **recursively**, registers folders as they appear, and marks the registry modified on every create/modify/delete, so whatever writes the registry, a pass follows (#7303). Its `DIRIGIBLE_REGISTRY_LOCAL_IGNORED_FOLDERS` top-level folders are neither watched nor reported. + +This does **not** retire `RegistryMutationTracker`: a pass is now scheduled while a multi-file write is still arriving, and only the bracket tells that pass to defer its cleanup instead of reaping artefacts whose sources have not landed yet. So a component that writes the registry outside the publisher pipeline still **brackets the write with `RegistryMutationTracker`** — it just no longer has to remember to announce it. `RegistryMutationTracker`'s javadoc states the rule: bracket the write, do not enumerate callers. **A failure of a collaborator records `FAILED` and is retried; `FATAL` is not a synchronizer's answer to it** (#7248). A `.listener` whose subscription the embedded broker refused while it was still taking its store lease, a `.job` the scheduler could not schedule yet, a `.camel` route missing a bean, a `.schema` naming a `.datasource` published on a later pass - each of these is transient, and each of the four synchronizers used to promote the second failure to `FATAL`, after which `SynchronizationProcessor.parseDefinitions` strips the artefact from every later pass until the file's bytes change (a topic subscription lost to a boot race stayed lost, with every message discarded). The shape now, in all four: a failed start registers `FAILED` **with its cause** (never `CREATED` while nothing runs) and returns `false`, which hands the artefact to the in-pass cross-retry loop (`DIRIGIBLE_SYNCHRONIZER_CROSS_RETRY_COUNT` x `_INTERVAL_MILLIS`, so a permanently failing artefact costs that loop every pass - the same cost a `FAILED` view carries since #6942); the `START` phase retries it and heals it to `CREATED` on success. **A pass runs its phases only when it carries a `NEW`/`MODIFIED` artefact** (`isSynchronizationNeeded` answers "the registry changed", and the test framework waits on that answer for a quiet period - do not widen it), so the processor additionally gives every `FAILED` artefact ONE `START` attempt per pass on an idle instance every `DIRIGIBLE_SYNCHRONIZER_FAILED_RETRY_INTERVAL_SECONDS` (30) via a private gate (`isFailedRetryDue` / `retryFailed`) - no cross-retry loop, and a repeat of the same error logs at DEBUG rather than stack-tracing at ERROR on every attempt. Two multitenant details: `MultitenantBaseSynchronizer` completes one shared artefact once per tenant and resets only the lifecycle between tenants, so a retry gate must include `FAILED` and not rely on `running` alone (the first tenant to subscribe flips it and would skip the rest), and the managers' idempotence per tenant (`ListenersManager.LISTENERS.containsKey`, `JobsManager.scheduleJob`'s `checkExists`) is what makes the repeated attempt safe. `ListenersManager.startListener` throws for a missing handler rather than returning normally - a normal return read as CREATED and running with nothing subscribed. Artefacts already persisted as `FATAL` by an earlier version stay stripped until republished; `case BROKEN:` in `parseDefinitions` is the same philosophy one level down (a definition that failed to parse is re-parsed every pass). diff --git a/CLAUDE_FEATURES.md b/CLAUDE_FEATURES.md index 597784ed1f4..b5f09b8f2ad 100644 --- a/CLAUDE_FEATURES.md +++ b/CLAUDE_FEATURES.md @@ -434,7 +434,6 @@ Interfaces that are either explicit SPIs (intended for extension), or load-beari | `UserAccessVerifier` | Plug-in role / access verification. | | `CustomSecurityConfigurator` | Add Spring Security configuration without forking `BasicSecurityConfig`. | | `SynchronizationWalkerCallback` | Callback used by the initializer / registry walker. | -| `LocalRegistryWatcherHandler` | Hook into local filesystem changes under the registry. | | `DataSourceLifecycleListener` | React to datasource registration / lifecycle. | #### CMS (`components/engine/engine-cms/`) diff --git a/components/core/core-registry/src/main/java/org/eclipse/dirigible/components/registry/watcher/LocalRegistryWatcher.java b/components/core/core-registry/src/main/java/org/eclipse/dirigible/components/registry/watcher/LocalRegistryWatcher.java index 4d558f7b73b..e3b243f510e 100644 --- a/components/core/core-registry/src/main/java/org/eclipse/dirigible/components/registry/watcher/LocalRegistryWatcher.java +++ b/components/core/core-registry/src/main/java/org/eclipse/dirigible/components/registry/watcher/LocalRegistryWatcher.java @@ -29,7 +29,6 @@ import java.util.Arrays; import java.util.Collections; import java.util.HashMap; -import java.util.List; import java.util.Map; import java.util.Set; import java.util.concurrent.ExecutorService; @@ -38,17 +37,42 @@ import java.util.stream.Collectors; import org.eclipse.dirigible.commons.config.DirigibleConfig; +import org.eclipse.dirigible.components.base.synchronizer.SynchronizationWatcher; import org.eclipse.dirigible.repository.api.IRepository; import org.eclipse.dirigible.repository.api.IRepositoryStructure; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.springframework.beans.factory.DisposableBean; -import org.springframework.beans.factory.annotation.Autowired; import org.springframework.context.annotation.Scope; import org.springframework.stereotype.Component; /** - * The Class LocalRegistryWatcher. + * Watches {@code /registry/public} recursively and marks the registry modified whenever + * something in it changes, so that the next synchronization pass actually runs. + * + *

+ * {@link SynchronizationWatcher} - the thing {@code SynchronizationProcessor} asks before it does + * anything at all - registers the registry root and nothing below it, so a file written several + * folders deep produces no event and no pass is ever scheduled for it. A publish is covered + * ({@code SynchronizationWatcherPublisherHandler} forces a pass) and so is the external-folder copy + * ({@link RecursiveFolderWatcher} brackets and forces its own writes), but a writer that does + * neither used to be invisible for the life of the process - the shape of #7192, where a partial + * client-Java generation stayed installed with all its sources on disk. This watcher closes that + * gap generically: whatever writes the registry, the write is seen and a pass follows. + * + *

+ * It does not make + * {@link org.eclipse.dirigible.components.base.registry.RegistryMutationTracker} optional. A + * pass is now scheduled while a multi-file write is still arriving, and only the bracket tells that + * pass to defer its cleanup instead of reaping artefacts whose sources have not landed yet. A + * component that writes the registry outside the publisher pipeline still brackets the write - what + * it no longer has to do is remember to announce it. + * + *

+ * Marking is deliberately all this does. {@link SynchronizationWatcher#force()} sets a flag; which + * pass runs, when, and over what is the processor's decision, so a copy of a thousand files costs a + * thousand flag writes and one pass rather than a pass per file. The folders named by + * {@code DIRIGIBLE_REGISTRY_LOCAL_IGNORED_FOLDERS} (top level only) are neither watched nor marked. */ @Component @Scope("singleton") @@ -106,19 +130,18 @@ public class LocalRegistryWatcher implements DisposableBean { /** The repository. */ private final IRepository repository; - /** The handlers. */ - private final List handlers; + /** Told that the registry changed, so a synchronization pass is scheduled. */ + private final SynchronizationWatcher synchronizationWatcher; /** * Instantiates a new local registry watcher. * * @param repository the repository - * @param handlers the handlers + * @param synchronizationWatcher the synchronization watcher */ - @Autowired - public LocalRegistryWatcher(IRepository repository, List handlers) { + public LocalRegistryWatcher(IRepository repository, SynchronizationWatcher synchronizationWatcher) { this.repository = repository; - this.handlers = handlers; + this.synchronizationWatcher = synchronizationWatcher; } /** @@ -149,15 +172,12 @@ public synchronized void initialize() { this.watchService = FileSystems.getDefault() .newWatchService(); - // Initial sync before start watching - initialSync(); - // Register watchers recursively registerAll(sourceDir); // Start actual watching this.startWatching(); - } catch (IOException | InterruptedException e) { + } catch (IOException e) { logger.error("Error during initializing the Local Registry Watcher", e); } }); @@ -197,35 +217,6 @@ private static String sanitizeFolderName(String folderName) { .replace("\n", ""); } - /** - * Perform initial sync of all files and folders. - * - * @throws IOException Signals that an I/O exception has occurred. - */ - private void initialSync() throws IOException { - logger.info("Performing initial sync..."); - Files.walkFileTree(sourceDir, new SimpleFileVisitor<>() { - @Override - public FileVisitResult preVisitDirectory(Path dir, BasicFileAttributes attrs) throws IOException { - if (isIgnored(dir)) { - logger.debug("Skipping ignored directory: {}", dir); - return FileVisitResult.SKIP_SUBTREE; - } - directoryRegistered(dir); - return FileVisitResult.CONTINUE; - } - - @Override - public FileVisitResult visitFile(Path file, BasicFileAttributes attrs) throws IOException { - if (!isIgnored(file)) { - fileRegistered(file); - } - return FileVisitResult.CONTINUE; - } - }); - logger.info("Initial sync complete."); - } - /** * Checks if is ignored. * @@ -289,10 +280,9 @@ private void register(Path dir) throws IOException { * Start watching. * * @throws IOException Signals that an I/O exception has occurred. - * @throws InterruptedException the interrupted exception */ - public void startWatching() throws IOException, InterruptedException { - logger.info("Recursively watching: " + sourceDir); + private void startWatching() throws IOException { + logger.info("Recursively watching: {}", sourceDir); watching = true; watchThread = Thread.currentThread(); @@ -352,43 +342,33 @@ private void watchLoop() throws IOException { for (WatchEvent event : key.pollEvents()) { WatchEvent.Kind kind = event.kind(); - if (kind == OVERFLOW) + if (kind == OVERFLOW) { + // Events were dropped, so what changed is unknown - which is exactly when a pass + // is most needed. Reconciling the whole registry is what a pass does anyway. + registryChanged(dir, "overflow"); continue; + } Path name = (Path) event.context(); Path sourcePath = dir.resolve(name); - if (kind == ENTRY_CREATE) { - if (Files.isDirectory(sourcePath)) { - // Register new directory + if (kind == ENTRY_CREATE && Files.isDirectory(sourcePath)) { + // A folder and everything already inside it. Register FIRST, report second: a + // file written into it before the registration produces no event of its own, and + // is only covered because the pass this report schedules walks the subtree after + // that file has landed. Reporting first would leave exactly that window open. + try { registerAll(sourcePath); - // Also sync its contents - try { - Files.walkFileTree(sourcePath, new SimpleFileVisitor<>() { - @Override - public FileVisitResult visitFile(Path file, BasicFileAttributes attrs) throws IOException { - fileCreated(file); - return FileVisitResult.CONTINUE; - } - - @Override - public FileVisitResult preVisitDirectory(Path dir, BasicFileAttributes attrs) throws IOException { - directoryCreated(dir); - return FileVisitResult.CONTINUE; - } - }); - } catch (IOException e) { - logger.error("Failed to sync new folder: " + sourcePath, e); - } - } else { - fileCreated(sourcePath); + } catch (IOException e) { + logger.error("Failed to watch the new registry folder: " + sourcePath, e); } - } else if (kind == ENTRY_MODIFY) { - if (!Files.isDirectory(sourcePath)) { - fileModified(sourcePath); - } - } else if (kind == ENTRY_DELETE) { - fileDeleted(sourcePath); + registryChanged(sourcePath, "created"); + } else if (kind == ENTRY_MODIFY && Files.isDirectory(sourcePath)) { + // A directory's own timestamp moves whenever a child is added or removed, and that + // child's event is reported in its own right - reporting this one too is noise. + logger.debug("Ignoring the modification of the directory: {}", sourcePath); + } else { + registryChanged(sourcePath, kind == ENTRY_CREATE ? "created" : kind == ENTRY_DELETE ? "deleted" : "modified"); } } @@ -396,118 +376,31 @@ public FileVisitResult preVisitDirectory(Path dir, BasicFileAttributes attrs) th if (!valid) { keyToPathMap.remove(key); if (keyToPathMap.isEmpty()) { - break; + // The registry root itself is gone, so there is nothing left to register against. + // Say so: from here on a deep write schedules no pass until the watcher is + // re-initialized, which is the very failure this watcher exists to prevent. + logger.warn("Nothing left to watch under [{}] - the Local Registry Watcher is stopping." + + " Registry changes will no longer schedule a synchronization pass.", sourceDir); + return; } } } } /** - * Directory registered. - * - * @param path the path - */ - private void directoryRegistered(Path path) { - if (!Files.isDirectory(path) || isIgnored(path)) { - return; - } - for (LocalRegistryWatcherHandler handler : handlers) { - try { - handler.directoryRegistered(path); - } catch (Exception e) { - logger.error("Failed to handle registration of a directory: " + path, e); - } - } - } - - /** - * File registered. - * - * @param path the path - */ - private void fileRegistered(Path path) { - if (Files.isDirectory(path) || isIgnored(path)) { - return; - } - for (LocalRegistryWatcherHandler handler : handlers) { - try { - handler.fileRegistered(path); - } catch (Exception e) { - logger.error("Failed to handle registration of a file: " + path, e); - } - } - } - - /** - * Directory created. - * - * @param path the path - */ - private void directoryCreated(Path path) { - if (!Files.isDirectory(path) || isIgnored(path)) { - return; - } - for (LocalRegistryWatcherHandler handler : handlers) { - try { - handler.directoryCreated(path); - } catch (Exception e) { - logger.error("Failed to handle creation of a directory: " + path, e); - } - } - } - - /** - * File created. - * - * @param path the path - */ - private void fileCreated(Path path) { - if (Files.isDirectory(path) || isIgnored(path)) { - return; - } - for (LocalRegistryWatcherHandler handler : handlers) { - try { - handler.fileCreated(path); - } catch (Exception e) { - logger.error("Failed to handle creation of a file: " + path, e); - } - } - } - - /** - * File modified. + * Reports a change under the registry, which marks the registry modified so the next + * synchronization pass runs. Ignored folders are not reported. * - * @param path the path + * @param path the path that changed + * @param change what happened to it, for the log */ - private void fileModified(Path path) { - if (Files.isDirectory(path) || isIgnored(path)) { + private void registryChanged(Path path, String change) { + if (isIgnored(path)) { + logger.debug("Ignoring the {} entry: {}", change, path); return; } - for (LocalRegistryWatcherHandler handler : handlers) { - try { - handler.fileModified(path); - } catch (Exception e) { - logger.error("Failed to handle modification of a file: " + path, e); - } - } - } - - /** - * File deleted. - * - * @param path the path - */ - private void fileDeleted(Path path) { - if (Files.isDirectory(path) || isIgnored(path)) { - return; - } - for (LocalRegistryWatcherHandler handler : handlers) { - try { - handler.fileDeleted(path); - } catch (Exception e) { - logger.error("Failed to handle deletion of a file: " + path, e); - } - } + logger.debug("Registry entry {}: [{}] - scheduling a synchronization pass", change, path); + synchronizationWatcher.force(); } /** diff --git a/components/core/core-registry/src/main/java/org/eclipse/dirigible/components/registry/watcher/LocalRegistryWatcherHandler.java b/components/core/core-registry/src/main/java/org/eclipse/dirigible/components/registry/watcher/LocalRegistryWatcherHandler.java deleted file mode 100644 index afbada1d4f2..00000000000 --- a/components/core/core-registry/src/main/java/org/eclipse/dirigible/components/registry/watcher/LocalRegistryWatcherHandler.java +++ /dev/null @@ -1,61 +0,0 @@ -/* - * Copyright (c) 2010-2026 Eclipse Dirigible contributors - * - * All rights reserved. This program and the accompanying materials are made available under the - * terms of the Eclipse Public License v2.0 which accompanies this distribution, and is available at - * http://www.eclipse.org/legal/epl-v20.html - * - * SPDX-FileCopyrightText: Eclipse Dirigible contributors SPDX-License-Identifier: EPL-2.0 - */ -package org.eclipse.dirigible.components.registry.watcher; - -import java.nio.file.Path; - -/** - * The Interface LocalRegistryWatcherHandler. - */ -public interface LocalRegistryWatcherHandler { - - /** - * Directory registered. - * - * @param path the path - */ - public void directoryRegistered(Path path); - - /** - * Directory created. - * - * @param path the path - */ - public void directoryCreated(Path path); - - /** - * File registered. - * - * @param path the path - */ - public void fileRegistered(Path path); - - /** - * File created. - * - * @param path the path - */ - public void fileCreated(Path path); - - /** - * File modified. - * - * @param path the path - */ - public void fileModified(Path path); - - /** - * File deleted. - * - * @param path the path - */ - public void fileDeleted(Path path); - -} diff --git a/components/core/core-registry/src/test/java/org/eclipse/dirigible/components/registry/watcher/LocalRegistryWatcherShutdownTest.java b/components/core/core-registry/src/test/java/org/eclipse/dirigible/components/registry/watcher/LocalRegistryWatcherShutdownTest.java index 05a3f72865d..43ccfb08e91 100644 --- a/components/core/core-registry/src/test/java/org/eclipse/dirigible/components/registry/watcher/LocalRegistryWatcherShutdownTest.java +++ b/components/core/core-registry/src/test/java/org/eclipse/dirigible/components/registry/watcher/LocalRegistryWatcherShutdownTest.java @@ -25,11 +25,11 @@ import java.nio.file.WatchKey; import java.nio.file.WatchService; import java.time.Duration; -import java.util.List; import java.util.concurrent.CountDownLatch; import java.util.concurrent.TimeUnit; import java.util.concurrent.atomic.AtomicBoolean; +import org.eclipse.dirigible.components.base.synchronizer.SynchronizationWatcher; import org.eclipse.dirigible.repository.api.IRepository; import org.junit.jupiter.api.Test; import org.junit.jupiter.api.io.TempDir; @@ -78,7 +78,7 @@ void destroyReturnsWhileTheWatchLoopIsBlockedInTake(@TempDir Path root) throws E IRepository repository = mock(IRepository.class); when(repository.getInternalResourcePath(anyString())).thenReturn(registryPublic.toString()); - LocalRegistryWatcher watcher = new LocalRegistryWatcher(repository, List.of()); + LocalRegistryWatcher watcher = new LocalRegistryWatcher(repository, mock(SynchronizationWatcher.class)); watcher.initialize(); awaitWatching(watcher); diff --git a/components/core/core-registry/src/test/java/org/eclipse/dirigible/components/registry/watcher/LocalRegistryWatcherTest.java b/components/core/core-registry/src/test/java/org/eclipse/dirigible/components/registry/watcher/LocalRegistryWatcherTest.java new file mode 100644 index 00000000000..fb37b270d9c --- /dev/null +++ b/components/core/core-registry/src/test/java/org/eclipse/dirigible/components/registry/watcher/LocalRegistryWatcherTest.java @@ -0,0 +1,145 @@ +/* + * Copyright (c) 2010-2026 Eclipse Dirigible contributors + * + * All rights reserved. This program and the accompanying materials are made available under the + * terms of the Eclipse Public License v2.0 which accompanies this distribution, and is available at + * http://www.eclipse.org/legal/epl-v20.html + * + * SPDX-FileCopyrightText: Eclipse Dirigible contributors SPDX-License-Identifier: EPL-2.0 + */ +package org.eclipse.dirigible.components.registry.watcher; + +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.mockito.ArgumentMatchers.anyString; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.timeout; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.verifyNoInteractions; +import static org.mockito.Mockito.when; + +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; + +import org.eclipse.dirigible.commons.config.Configuration; +import org.eclipse.dirigible.components.base.synchronizer.SynchronizationWatcher; +import org.eclipse.dirigible.repository.api.IRepository; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +/** + * The registry watcher that everything depends on registers the registry root and nothing below it, + * so a write several folders deep schedules no synchronization pass at all - the shape of #7192. + * This watcher is the one that sees deep writes, and its whole job is to report them. + */ +class LocalRegistryWatcherTest { + + /** A change picked up by the watch service: up to ~10s where the JDK falls back to polling. */ + private static final long WATCH_TIMEOUT_MILLIS = 60_000; + + /** How long to wait for the watch loop to be up before changing anything under it. */ + private static final long START_TIMEOUT_MILLIS = 30_000; + + private final SynchronizationWatcher synchronizationWatcher = mock(SynchronizationWatcher.class); + + private LocalRegistryWatcher watcher; + + @AfterEach + void stopWatching() { + Configuration.remove("DIRIGIBLE_REGISTRY_LOCAL_IGNORED_FOLDERS"); + if (watcher != null) { + // the watch service holds handles on the temp folders - Windows refuses to delete those + watcher.destroy(); + } + } + + @Test + void aFileWrittenDeepInTheRegistrySchedulesASynchronizationPass(@TempDir Path root) throws IOException { + Path registry = startWatching(root, "project/deep"); + + Files.writeString(registry.resolve("project") + .resolve("deep") + .resolve("artefact.txt"), + "content"); + + verify(synchronizationWatcher, timeout(WATCH_TIMEOUT_MILLIS)).force(); + } + + /** + * A folder is registered as it appears, or the project a publish or a copy drops in would be the + * one thing the watcher never watches. + */ + @Test + void aFolderCreatedAfterStartupIsWatchedTooAndItsContentSchedulesAPass(@TempDir Path root) throws IOException { + Path registry = startWatching(root, "existing"); + + Path fresh = Files.createDirectories(registry.resolve("fresh-project") + .resolve("deep")); + verify(synchronizationWatcher, timeout(WATCH_TIMEOUT_MILLIS)).force(); + + Files.writeString(fresh.resolve("artefact.txt"), "content"); + verify(synchronizationWatcher, timeout(WATCH_TIMEOUT_MILLIS).atLeast(2)).force(); + } + + /** A deletion leaves runtime state behind just as a creation leaves it missing. */ + @Test + void aDeletedArtefactSchedulesASynchronizationPass(@TempDir Path root) throws IOException { + Path registry = startWatching(root, "project"); + Path artefact = registry.resolve("project") + .resolve("artefact.txt"); + Files.writeString(artefact, "content"); + verify(synchronizationWatcher, timeout(WATCH_TIMEOUT_MILLIS)).force(); + + Files.delete(artefact); + + verify(synchronizationWatcher, timeout(WATCH_TIMEOUT_MILLIS).atLeast(2)).force(); + } + + /** An ignored top-level folder is neither watched nor reported - that is what the key is for. */ + @Test + void aChangeInAnIgnoredFolderSchedulesNothing(@TempDir Path root) throws IOException, InterruptedException { + Configuration.set("DIRIGIBLE_REGISTRY_LOCAL_IGNORED_FOLDERS", "ignored"); + Path registry = startWatching(root, "ignored/deep"); + + Files.writeString(registry.resolve("ignored") + .resolve("deep") + .resolve("artefact.txt"), + "content"); + // nothing to wait for, so give the watch service the time it would have needed to report it + Thread.sleep(5_000); + + verifyNoInteractions(synchronizationWatcher); + } + + private Path startWatching(Path root, String existingFolder) throws IOException { + Path registry = root.resolve("registry") + .resolve("public"); + Files.createDirectories(registry.resolve(existingFolder)); + + IRepository repository = mock(IRepository.class); + when(repository.getInternalResourcePath(anyString())).thenReturn(registry.toString()); + + watcher = new LocalRegistryWatcher(repository, synchronizationWatcher); + watcher.initialize(); + awaitWatching(); + return registry; + } + + private void awaitWatching() { + long deadline = System.currentTimeMillis() + START_TIMEOUT_MILLIS; + while (!watcher.isWatching()) { + if (System.currentTimeMillis() > deadline) { + throw new AssertionError("The watcher did not start watching within " + START_TIMEOUT_MILLIS + " ms"); + } + try { + Thread.sleep(50); + } catch (InterruptedException e) { + Thread.currentThread() + .interrupt(); + throw new AssertionError("Interrupted while waiting for the watcher to start", e); + } + } + assertTrue(watcher.isWatching()); + } +} diff --git a/tests/tests-integrations/src/main/java/org/eclipse/dirigible/integration/tests/api/RegistryDeepWriteSyncIT.java b/tests/tests-integrations/src/main/java/org/eclipse/dirigible/integration/tests/api/RegistryDeepWriteSyncIT.java new file mode 100644 index 00000000000..a41c65896b2 --- /dev/null +++ b/tests/tests-integrations/src/main/java/org/eclipse/dirigible/integration/tests/api/RegistryDeepWriteSyncIT.java @@ -0,0 +1,135 @@ +/* + * Copyright (c) 2010-2026 Eclipse Dirigible contributors + * + * All rights reserved. This program and the accompanying materials are made available under the + * terms of the Eclipse Public License v2.0 which accompanies this distribution, and is available at + * http://www.eclipse.org/legal/epl-v20.html + * + * SPDX-FileCopyrightText: Eclipse Dirigible contributors SPDX-License-Identifier: EPL-2.0 + */ +package org.eclipse.dirigible.integration.tests.api; + +import static io.restassured.RestAssured.given; +import static org.hamcrest.Matchers.containsString; + +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.concurrent.TimeUnit; + +import org.apache.commons.io.FileUtils; +import org.awaitility.Awaitility; +import org.eclipse.dirigible.components.base.synchronizer.SynchronizationWatcher; +import org.eclipse.dirigible.components.initializers.synchronizer.SynchronizationProcessor; +import org.eclipse.dirigible.repository.api.IRepository; +import org.eclipse.dirigible.repository.api.IRepositoryStructure; +import org.eclipse.dirigible.tests.base.IntegrationTest; +import org.eclipse.dirigible.tests.framework.restassured.RestAssuredExecutor; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.Test; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.test.annotation.DirtiesContext; + +/** + * A write several folders deep into the registry reaches the runtime on its own - whoever made it + * (issue #7303). + * + *

+ * {@code SynchronizationWatcher}, which decides whether a synchronization pass runs at all, + * registers the registry root and nothing below it, so only a publish (which forces a pass) and the + * external-folder copy (which forces its own, #7299) were ever seen. {@code LocalRegistryWatcher} + * watches the registry recursively and marks it modified, which is what makes any other writer - a + * git clone straight into the registry, a tool copying a project in - self-healing too. + * + *

+ * Nothing here publishes, forces a pass, or mounts an external folder: the platform has to notice + * by itself, so the assertion waits for the scheduled pass instead. On master this test fails by + * timing out on a 404 with the source sitting on disk. + */ +@DirtiesContext(classMode = DirtiesContext.ClassMode.AFTER_CLASS) +class RegistryDeepWriteSyncIT extends IntegrationTest { + + private static final String PROJECT = "registry-deep-write-it"; + + /** + * The folder the source is written into, INSIDE the project. A file added directly under the + * project folder moves the mtime of a direct child of the registry root, which the root watcher + * notices on the polling watch service macOS falls back to - the test would then pass for the wrong + * reason. + */ + private static final String FOLDER = "rdwit"; + + private static final String ENDPOINT = "/services/java/" + PROJECT + "/" + FOLDER + "/DeepWrite"; + + /** The synchronization job fires on its own schedule (10s by default), then javac runs. */ + private static final long AWAIT_SECONDS = 180; + + @Autowired + private RestAssuredExecutor restAssuredExecutor; + + @Autowired + private IRepository repository; + + @Autowired + private SynchronizationWatcher synchronizationWatcher; + + @Autowired + private SynchronizationProcessor synchronizationProcessor; + + @AfterEach + void removeTheProject() throws IOException { + FileUtils.deleteDirectory(projectFolder().toFile()); + } + + @Test + void a_source_written_deep_into_the_registry_is_compiled_and_served() throws IOException { + Files.createDirectories(sourceFolder()); + // Settle first, or the assertion proves nothing: the project folder is a direct child of the + // registry root, which the root watcher DOES see, and the pass that schedules would pick the + // source up on its own. Once no change is pending and no pass is running, nothing else can + // schedule one - so the write below is the only possible cause of the next pass. + awaitAnIdlePlatform(); + + Files.writeString(sourceFolder().resolve("DeepWrite.java"), handlerSource()); + + restAssuredExecutor.execute(() -> given().when() + .get(ENDPOINT) + .then() + .statusCode(200) + .body(containsString("hello from a deep registry write")), + AWAIT_SECONDS); + } + + private void awaitAnIdlePlatform() { + Awaitility.await() + .atMost(2, TimeUnit.MINUTES) + .pollInterval(1, TimeUnit.SECONDS) + .until(() -> !synchronizationWatcher.isModified() && !synchronizationProcessor.isSynchronizationRunning()); + } + + private Path sourceFolder() { + return projectFolder().resolve(FOLDER); + } + + private Path projectFolder() { + return Path.of(repository.getInternalResourcePath(IRepositoryStructure.PATH_REGISTRY_PUBLIC)) + .resolve(PROJECT); + } + + private static String handlerSource() { + return """ + package %s; + import jakarta.servlet.http.HttpServletRequest; + import jakarta.servlet.http.HttpServletResponse; + import org.eclipse.dirigible.engine.java.handler.JavaHandler; + public class DeepWrite implements JavaHandler { + @Override + public void handle(HttpServletRequest request, HttpServletResponse response) throws Exception { + response.setContentType("application/json"); + response.getWriter().write("{\\"message\\": \\"hello from a deep registry write\\"}"); + } + } + """.formatted(FOLDER); + } + +} From a1e80394a95f0ffb30650c11429a37632f35c6cf Mon Sep 17 00:00:00 2001 From: Iliyan Velichkov Date: Fri, 11 Sep 2026 16:06:44 +0300 Subject: [PATCH 2/2] registry: the watcher test asserts the write was reported, not how often (#7303) How many events one write produces is the platform's business: inotify reports a created file as ENTRY_CREATE and again as ENTRY_MODIFY, while the polling watch service macOS falls back to reports it once. The assertions were written against the macOS count, so on the Linux CI leg two forces met a verification wanting exactly one and the two single-write cases failed with TooManyActualInvocations after burning the full 60s timeout. Marking the registry modified is idempotent, so the count carries nothing worth pinning down - every assertion is now atLeastOnce(), and the steps of a sequence are separated by clearing the recorded calls instead of by counting them. Co-Authored-By: Claude Opus 5 (1M context) --- .../watcher/LocalRegistryWatcherTest.java | 28 +++++++++++++++---- 1 file changed, 23 insertions(+), 5 deletions(-) diff --git a/components/core/core-registry/src/test/java/org/eclipse/dirigible/components/registry/watcher/LocalRegistryWatcherTest.java b/components/core/core-registry/src/test/java/org/eclipse/dirigible/components/registry/watcher/LocalRegistryWatcherTest.java index fb37b270d9c..73489790784 100644 --- a/components/core/core-registry/src/test/java/org/eclipse/dirigible/components/registry/watcher/LocalRegistryWatcherTest.java +++ b/components/core/core-registry/src/test/java/org/eclipse/dirigible/components/registry/watcher/LocalRegistryWatcherTest.java @@ -11,6 +11,7 @@ import static org.junit.jupiter.api.Assertions.assertTrue; import static org.mockito.ArgumentMatchers.anyString; +import static org.mockito.Mockito.clearInvocations; import static org.mockito.Mockito.mock; import static org.mockito.Mockito.timeout; import static org.mockito.Mockito.verify; @@ -27,6 +28,7 @@ import org.junit.jupiter.api.AfterEach; import org.junit.jupiter.api.Test; import org.junit.jupiter.api.io.TempDir; +import org.mockito.verification.VerificationMode; /** * The registry watcher that everything depends on registers the registry root and nothing below it, @@ -38,6 +40,7 @@ class LocalRegistryWatcherTest { /** A change picked up by the watch service: up to ~10s where the JDK falls back to polling. */ private static final long WATCH_TIMEOUT_MILLIS = 60_000; + /** How long to wait for the watch loop to be up before changing anything under it. */ private static final long START_TIMEOUT_MILLIS = 30_000; @@ -63,7 +66,7 @@ void aFileWrittenDeepInTheRegistrySchedulesASynchronizationPass(@TempDir Path ro .resolve("artefact.txt"), "content"); - verify(synchronizationWatcher, timeout(WATCH_TIMEOUT_MILLIS)).force(); + verify(synchronizationWatcher, reported()).force(); } /** @@ -76,10 +79,11 @@ void aFolderCreatedAfterStartupIsWatchedTooAndItsContentSchedulesAPass(@TempDir Path fresh = Files.createDirectories(registry.resolve("fresh-project") .resolve("deep")); - verify(synchronizationWatcher, timeout(WATCH_TIMEOUT_MILLIS)).force(); + verify(synchronizationWatcher, reported()).force(); + clearInvocations(synchronizationWatcher); Files.writeString(fresh.resolve("artefact.txt"), "content"); - verify(synchronizationWatcher, timeout(WATCH_TIMEOUT_MILLIS).atLeast(2)).force(); + verify(synchronizationWatcher, reported()).force(); } /** A deletion leaves runtime state behind just as a creation leaves it missing. */ @@ -89,11 +93,12 @@ void aDeletedArtefactSchedulesASynchronizationPass(@TempDir Path root) throws IO Path artefact = registry.resolve("project") .resolve("artefact.txt"); Files.writeString(artefact, "content"); - verify(synchronizationWatcher, timeout(WATCH_TIMEOUT_MILLIS)).force(); + verify(synchronizationWatcher, reported()).force(); + clearInvocations(synchronizationWatcher); Files.delete(artefact); - verify(synchronizationWatcher, timeout(WATCH_TIMEOUT_MILLIS).atLeast(2)).force(); + verify(synchronizationWatcher, reported()).force(); } /** An ignored top-level folder is neither watched nor reported - that is what the key is for. */ @@ -126,6 +131,19 @@ private Path startWatching(Path root, String existingFolder) throws IOException return registry; } + /** + * The verification every assertion here uses: the write was reported at least once. How many + * events one write produces is the platform's business - inotify reports a created file as + * ENTRY_CREATE and again as ENTRY_MODIFY, while the polling watch service macOS falls back to + * reports it once - and marking the registry modified is idempotent, so the count carries nothing + * worth pinning down. The steps of a sequence are separated by clearing the recorded calls instead. + * + * @return the verification mode + */ + private static VerificationMode reported() { + return timeout(WATCH_TIMEOUT_MILLIS).atLeastOnce(); + } + private void awaitWatching() { long deadline = System.currentTimeMillis() + START_TIMEOUT_MILLIS; while (!watcher.isWatching()) {