OAK-12089: Add Lucene 9 index provider (oak-search-lucene-ng) by bhabegger · Pull Request #2817 · apache/jackrabbit-oak

bhabegger · 2026-03-26T15:07:12Z

Summary

Introduces oak-search-lucene-ng, a new Oak module that provides a Lucene 9 based index engine. Indexes opt in explicitly via type=lucene9; all existing indexes are unaffected.

New oak-search-luceneNg module: index editor, query index, tracker, index node, storage (Oak JCR node-based directory), and OSGi wiring
Full query parity with the legacy lucene engine: property restrictions, path/type filters, fulltext, sorting, and excerpts
Facet parity for all three ACL modes: insecure, statistical (TapeSampling), and secure (per-document access check) — Lucene 9 API adaptations and null-safe MatchingDocs.bits handling
LuceneNgFacetCommonTest extends the shared FacetCommonTest suite for end-to-end JCR-level facet coverage
AbstractIndexComparisonTest inlined into oak-search test-jar; oak-search-test module removed
README documents feature parity vs legacy Lucene and Elastic

thomasmueller

Just some bikeshedding: I wonder if we should use the term "Lucene 9" at all.

thomasmueller · 2026-03-30T14:06:43Z

@@ -0,0 +1,26 @@
+# oak-search-luceneNg
+
+Lucene 9 index provider for Oak (`type="lucene9"`).


What about:

Suggested change

Lucene 9 index provider for Oak (`type="lucene9"`).

Lucene NG index provider for Oak (`type="lucene-ng"`).

Hoping we can upgrade to Lucene 10 without having to change the type. I do assume the index storage version won't change, or that there is an option to add compatibility.

If not, and we do need to reindex for lucene 10, we can still add "lucene-ng-10" if that should be needed.

So the idea is that the code should be less sensitive to Lucene upgrades. However, I wasn't really sure about having a fixed name and the in some future having to have a lucene-ng-ng ;) Here we could interpret it as lucene since 9 and still use that for lucene 10, 11 up to and imaginary lucene 12 that breaks compatibility again (which might never happen as likely the APIs are more stable now).

Personally, I fine with lucene-ng as I myself had doubts.

Also the NG in the code for now is fine since we do have 2 versions, but at some point my expectation is that the legacy code be removed and the NG no longer be new generation and be refactored with proper non NG names. So in the code I'm fine, in the type however, this sticks more. That's mostly what bother me with ng in the type.

@thomasmueller What about lucene2026 ? Less tied to 9 doesn't have explicit ng ?

Let's vote here for this: https://github.com/orgs/oak-indexing/discussions/1289

reschke · 2026-08-10T18:29:18Z

[ERROR] [ERROR] Some problems were encountered while processing the POMs:
[FATAL] Non-resolvable parent POM for org.apache.jackrabbit:oak-search-lucene-ng:1.93-SNAPSHOT: The following artifacts could not be resolved: org.apache.jackrabbit:oak-parent:pom:1.93-SNAPSHOT (absent): Could not find artifact org.apache.jackrabbit:oak-parent:pom:1.93-SNAPSHOT and 'parent.relativePath' points at wrong local POM @ line 23, column 13

Introduces oak-search-luceneNg, a new Oak module providing a Lucene 9 based index engine under type=lucene9, with full parity to the legacy lucene implementation for property queries, fulltext, sorting, excerpts, and facets (insecure, statistical, and secure ACL modes). Key changes: - New oak-search-luceneNg module: index editor, query index, tracker, index node, storage, and OSGi wiring - Facet parity: LuceneNgSecure/StatisticalSortedSetDocValuesFacetCounts ported to Lucene 9 APIs with null-safe MatchingDocs.bits handling - LuceneNgFacetCommonTest extends FacetCommonTest for JCR-level coverage - AbstractIndexComparisonTest inlined into oak-search test-jar; oak-search-test module removed - getRootBuilder removed from ContextAwareCallback and IndexUpdate - leaf OSGi property removed from LuceneIndexProviderService - README documents feature parity vs legacy Lucene and Elastic Made-with: Cursor

amit-jain

I think there are some architectural gaps which we should address, some are even mentioned in the readme as well and could be addressed with these changes.

Mainly we should reuse the FulltextIndexEditor and FulltextIndexTracker from the oak-search module which the elastic module also reuses and extends.

amit-jain · 2026-08-24T06:33:49Z

+
+    public IndexSearcherHolder(NodeState storageState, String indexName) throws IOException {
+        this.indexName = indexName;
+        this.directory = new OakDirectory(storageState.builder(), indexName, true);


NRT handling is an important part ans is also missing i think

I heard a lot about NRT being mostly abused and I think we could have a first go without it. I would like to test an already big change. Let's make sure everything works without NRT and then add it back in a second step.

I heard a lot about NRT being mostly abused

Ok I am not aware of the abuse, can you share more details.
Also, no problem in doing as a second step but I just highlighted things missing.

Some users put nrt just because they can, not because they need it.

Let's keep this like this for now. I would really like to test this first "core" implementation on indices where it already has sufficient features and then extend the features to cover the other cases.

Limitations are anyway documented here in the README

Let's keep this like this for now. I would really like to test this first "core" implementation on indices where it already has sufficient features and then extend the features to cover the other cases.

This functionality was added afair as users complained adding a doc and not being able to search on it.

Some users put nrt just because they can, not because they need it.

And does it create a problem? But for users requiring it, not having this it creates a problem

Also, adding this with the extended common oak-search extension might have some quirks (new overloaded methods etc) which might be cleaner in a new impl as this.
BTW i am just looking at this from a parity pov and do not have a personal view on it. @thomasmueller can best suggest if we are ok not adding it initially.

And does it create a problem? But for users requiring it, not having this it creates a problem
That's very general question that fits better around a coffee or a beer ;)

But understood that there are cases where this current version will not work. And it never has been the intent of this first PR to be feature complete, only to be a first step towards it.

Also, adding this with the extended common oak-search extension might have some quirks (new overloaded methods etc) which might be cleaner in a new impl as this.

I'm not sure what you meant exactly by this. If it is to generalize and simplify the code, I'm all in. Just note that, this PR was voluntarily built in a way to keep running production code not using this feature completely untouched. (Notice only tests have been adjusted to validate feature iso-behavior).

amit-jain · 2026-08-25T06:49:24Z

Another important missing piece that is needed is the local fs index file support with the IndexCopier etc class which then get called with https://github.com/apache/jackrabbit-oak/blob/trunk/oak-lucene/src/main/java/org/apache/jackrabbit/oak/plugins/index/lucene/reader/DefaultIndexReaderFactory.java#L97

…eManager/FulltextIndexTracker Replaces the hand-rolled LuceneNgIndexNode/LuceneNgIndexTracker lifecycle/locking with the shared oak-search IndexNode/IndexNodeManager/FulltextIndexTracker framework, mirroring ElasticIndexNode/ElasticIndexNodeManager/ElasticIndexTracker. Fixes the documented IndexSearcherHolder.getFacetReaderState() vs close() race: releaseResources() (where the searcher is actually closed) now cannot run until every acquire()-held read lock has been released, because IndexNodeManager.close() holds its write lock across the whole closed=true flip. LuceneNgIndexNode now implements IndexNode directly (no more AcquiredNode inner class). LuceneNgIndexTracker.acquireIndexNode(String) returns LuceneNgIndexNode directly. New LuceneNgIndexNodeManager (luceneNg.internal) wraps one generation of a node per the Elastic pattern. Two deviations from a naive port, needed for correctness: - LuceneNgIndexTracker overrides isUpdateNeeded() to diff the whole subtree instead of relying on FulltextIndexTracker's default (:status/:index-definition), since this module's editor never writes either of those nodes -- Lucene segment files live directly under the index definition node itself. - LuceneNgIndexNode.release() cannot call the inherited protected IndexNodeManager.release() directly (it isn't an IndexNodeManager subclass); added a package-private LuceneNgIndexNodeManager.releaseNode() wrapper. Also updates callers to compile/behave correctly against the new API: - LuceneNgIndex.java, internal/LuceneNgCursor.java: AcquiredNode -> LuceneNgIndexNode. - LuceneNgIndexProviderService.java: FulltextIndexTracker.close() is package-private to oak-search and unreachable from here; deactivate() now drives tracker.update(EMPTY_NODE) instead, which closes every tracked IndexNodeManager through the same public API. - LuceneNgQueryIndexProvider.java (not in the original file list, but required to compile and to avoid a real regression): getQueryIndexes() now enumerates lucene9 indexes directly off the given NodeState rather than tracker.getIndexNodePaths(), since the shared tracker only caches paths already opened at least once and does not itself do full-repository discovery on update(). Adapts LuceneNgIndexTrackerTest/LuceneNgIndexNodeTest per the task brief, plus mechanical fixes (type renames, tracker.close() -> tracker.update(EMPTY_NODE)) in IntegrationTest, LuceneNgCursorBatchingTest and LuceneNgIndexTest so the module keeps compiling. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

bhabegger · 2026-08-26T07:27:06Z

Another important missing piece that is needed is the local fs index file support with the IndexCopier etc class which then get called with https://github.com/apache/jackrabbit-oak/blob/trunk/oak-lucene/src/main/java/org/apache/jackrabbit/oak/plugins/index/lucene/reader/DefaultIndexReaderFactory.java#L97

Can you remind me what this is for ? Is this a must for this PR or can it be moved to later ?

amit-jain · 2026-08-26T08:16:35Z

Another important missing piece that is needed is the local fs index file support with the IndexCopier etc class which then get called with https://github.com/apache/jackrabbit-oak/blob/trunk/oak-lucene/src/main/java/org/apache/jackrabbit/oak/plugins/index/lucene/reader/DefaultIndexReaderFactory.java#L97

Can you remind me what this is for ? Is this a must for this PR or can it be moved to later ?

https://jackrabbit.apache.org/oak/docs/query/lucene.html#copyonread

bhabegger · 2026-08-26T10:11:02Z

Another important missing piece that is needed is the local fs index file support with the IndexCopier etc class which then get called with https://github.com/apache/jackrabbit-oak/blob/trunk/oak-lucene/src/main/java/org/apache/jackrabbit/oak/plugins/index/lucene/reader/DefaultIndexReaderFactory.java#L97

Can you remind me what this is for ? Is this a must for this PR or can it be moved to later ?

https://jackrabbit.apache.org/oak/docs/query/lucene.html#copyonread

Ok, so this is in fact a performance optimization in the case the NodeStore is remote (many cases I agree). This doesn't block functional testing though (unless performance is so degraded that this becomes just not usable at all even restricted to testing). I would accept this for now, and leave open to solve the performance issue in maybe some way different than how it was managed in the legacy.

Ok, with you to differ ?

…eak, and adopt shared FulltextIndex/FulltextIndexPlanner for LuceneNgIndex query planning

This was referenced Mar 26, 2026

OAK-12089: add multi-target routing and per-target catch-up lifecycle #2818

Draft

feat: Add Lucene 9 integration - Phase 1 (Write Path) + Phase 2 Step 1 (Query Support) bhabegger/jackrabbit-oak#51

Closed

bhabegger force-pushed the oak-12089-lucene9-core branch from e4bc5ad to dbccf97 Compare March 26, 2026 15:11

bhabegger marked this pull request as draft March 26, 2026 15:13

bhabegger force-pushed the oak-12089-lucene9-core branch from 28c304c to 66fb544 Compare March 27, 2026 16:59

bhabegger changed the title ~~feat: add lucene9 index provider as safe opt-in target~~ feat: add Lucene 9 index provider (oak-search-luceneNg) Mar 27, 2026

bhabegger marked this pull request as ready for review March 30, 2026 05:10

thomasmueller reviewed Mar 30, 2026

View reviewed changes

Comment thread oak-search-luceneNg/README.md Outdated

Comment thread oak-search-luceneNg/README.md Outdated

thomasmueller reviewed Mar 30, 2026

View reviewed changes

bhabegger changed the title ~~feat: add Lucene 9 index provider (oak-search-luceneNg)~~ OAK-12089: Add Lucene 9 index provider (oak-search-luceneNg) Apr 10, 2026

bhabegger force-pushed the oak-12089-lucene9-core branch from f0a7199 to de83bda Compare June 5, 2026 07:59

bhabegger changed the base branch from OAK-12089 to trunk June 5, 2026 08:12

bhabegger changed the base branch from trunk to OAK-12089 June 5, 2026 08:13

bhabegger force-pushed the oak-12089-lucene9-core branch from 7cbee48 to 5b36b34 Compare August 10, 2026 09:24

bhabegger changed the title ~~OAK-12089: Add Lucene 9 index provider (oak-search-luceneNg)~~ OAK-12089: Add Lucene 9 index provider (oak-search-lucene-ng) Aug 10, 2026

bhabegger force-pushed the oak-12089-lucene9-core branch from 5b36b34 to c335365 Compare August 12, 2026 05:59

amit-jain requested changes Aug 24, 2026

View reviewed changes

bhabegger force-pushed the oak-12089-lucene9-core branch from 7c648cd to a82839a Compare August 26, 2026 07:14

bhabegger force-pushed the oak-12089-lucene9-core branch from 8c5cc0b to 402d45c Compare August 28, 2026 06:39

OAK-12089: fix property-scoped fulltext, close write-path Directory l…

605a3a6

…eak, and adopt shared FulltextIndex/FulltextIndexPlanner for LuceneNgIndex query planning

bhabegger force-pushed the oak-12089-lucene9-core branch from 402d45c to 605a3a6 Compare August 28, 2026 10:41

		@@ -0,0 +1,26 @@
		# oak-search-luceneNg

		Lucene 9 index provider for Oak (`type="lucene9"`).

	Lucene 9 index provider for Oak (`type="lucene9"`).
	Lucene NG index provider for Oak (`type="lucene-ng"`).

Uh oh!

Conversation

bhabegger commented Mar 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Uh oh!

Uh oh!

Uh oh!

thomasmueller left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bhabegger Apr 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bhabegger Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

reschke commented Aug 10, 2026

Uh oh!

amit-jain left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bhabegger Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

amit-jain Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

amit-jain commented Aug 25, 2026

Uh oh!

bhabegger commented Aug 26, 2026

Uh oh!

amit-jain commented Aug 26, 2026

Uh oh!

bhabegger commented Aug 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

bhabegger commented Mar 26, 2026 •

edited

Loading

bhabegger Apr 1, 2026 •

edited

Loading

bhabegger Aug 25, 2026 •

edited

Loading

bhabegger Aug 26, 2026 •

edited

Loading

amit-jain Aug 26, 2026 •

edited

Loading