Extract an ingestion adapter seam inside elasticgraph-indexer#1301
Open
jwils wants to merge 1 commit into
Open
Extract an ingestion adapter seam inside elasticgraph-indexer#1301jwils wants to merge 1 commit into
jwils wants to merge 1 commit into
Conversation
jwils
requested review from
BrianSigafoos-SQ,
bsorbo,
ellisandrews-toast,
jwondrusch,
marcdaniels-toast,
myronmarston and
rossroberts-toast
as code owners
July 7, 2026 21:21
This was referenced Jul 7, 2026
jwils
force-pushed
the
joshuaw/ingestion-adapter-seam
branch
from
July 7, 2026 21:47
1f731a7 to
6e17704
Compare
jwils
force-pushed
the
joshuaw/apply-indexer-extensions
branch
from
July 8, 2026 02:10
8070692 to
97060ac
Compare
jwils
force-pushed
the
joshuaw/ingestion-adapter-seam
branch
from
July 8, 2026 02:10
6e17704 to
a9d37cc
Compare
jwils
added a commit
that referenced
this pull request
Jul 8, 2026
## Why To support ingestion in any data format while keeping format-specific logic in its own gem (`elasticgraph-json_ingestion`, the upcoming `elasticgraph-proto_ingestion`), the indexer needs an extension mechanism analogous to the GraphQL one: schema definition extensions register runtime extensions in the schema artifacts, and the runtime component applies them at boot. This PR adds the runtime metadata storage for that. ## What - Add `indexer_extension_modules` to `RuntimeMetadata::Schema`, mirroring `graphql_extension_modules` - Add a `RuntimeMetadata::IndexerExtension` wrapper (twin of `GraphQLExtension`) - Nothing registers or consumes these yet, and empty lists are pruned from dumped YAML, so no schema artifacts change ## Verification - `script/lint`, `script/type_check` - `script/run_gem_specs elasticgraph-schema_artifacts` (100% coverage) - `script/run_gem_specs elasticgraph-schema_definition` - `bundle exec rake schema_artifacts:check` (no artifact changes) ## Stack Current PR is marked with `->`. - -> [#1298 Add indexer_extension_modules to runtime metadata](#1298) - [#1299 Add register_indexer_extension schema definition API](#1299) - [#1300 Apply indexer extension modules when the Indexer boots](#1300) - [#1301 Extract an ingestion adapter seam inside elasticgraph-indexer](#1301) - [#1302 Move JSON ingestion into elasticgraph-json_ingestion via an indexer extension](#1302)
jwils
added a commit
that referenced
this pull request
Jul 8, 2026
## Why Ingestion format gems need a way to ensure their indexer-side behavior is active whenever a schema is defined with their schema definition extension--the same guarantee `register_graphql_extension` provides for GraphQL extensions (e.g. how `elasticgraph-apollo` registers its engine extension). ## What - Add `register_indexer_extension` to the schema definition API, mirroring `register_graphql_extension` - Registered extensions are dumped into the `indexer_extension_modules` runtime metadata added in #1298 ## Verification - `script/lint`, `script/type_check` - `script/run_gem_specs elasticgraph-schema_definition` (100% coverage) ## Stack Current PR is marked with `->`. - [#1298 Add indexer_extension_modules to runtime metadata](#1298) - -> [#1299 Add register_indexer_extension schema definition API](#1299) - [#1300 Apply indexer extension modules when the Indexer boots](#1300) - [#1301 Extract an ingestion adapter seam inside elasticgraph-indexer](#1301) - [#1302 Move JSON ingestion into elasticgraph-json_ingestion via an indexer extension](#1302)
jwils
force-pushed
the
joshuaw/ingestion-adapter-seam
branch
from
July 8, 2026 16:15
a9d37cc to
73bb297
Compare
jwils
force-pushed
the
joshuaw/apply-indexer-extensions
branch
2 times, most recently
from
July 8, 2026 20:52
4ece4bc to
ae59e99
Compare
jwils
added a commit
that referenced
this pull request
Jul 8, 2026
## Why With runtime metadata storage (#1298) and the registration API (#1299) in place, `elasticgraph-indexer` needs to actually apply registered extensions--introducing the indexer extension concept, mirroring how `elasticgraph-graphql` applies GraphQL extension modules from both settings YAML and runtime metadata. ## What - `Indexer#initialize` extends configured extension modules onto the instance: first from the new `indexer.extension_modules` setting (mirroring `graphql.extension_modules`), then from the runtime metadata's `indexer_extension_modules` - Regenerated config schema artifacts for the new setting - Nothing registers an indexer extension yet, so this is a no-op for all existing schemas ## Verification - `script/lint`, `script/type_check` - `script/run_gem_specs elasticgraph-indexer` (100% coverage) - `script/run_gem_specs elasticgraph-local` ## Stack Current PR is marked with `->`. - [#1298 Add indexer_extension_modules to runtime metadata](#1298) - [#1299 Add register_indexer_extension schema definition API](#1299) - -> [#1300 Apply indexer extension modules when the Indexer boots](#1300) - [#1301 Extract an ingestion adapter seam inside elasticgraph-indexer](#1301) - [#1302 Move JSON ingestion into elasticgraph-json_ingestion via an indexer extension](#1302)
Operation::Factory no longer performs JSON-schema validation itself; instead it routes each event to an ingestion adapter, which validates the event and supplies the version-appropriate record preparer. The existing JSON validation logic moves behind the seam as IngestionAdapter::JSONEvents, so behavior is unchanged (the existing validation specs pass as-is). Multiple adapters dispatch by handles_event?, with a single available adapter receiving all events.
jwils
force-pushed
the
joshuaw/ingestion-adapter-seam
branch
from
July 8, 2026 21:25
73bb297 to
9b5aa5c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Operation::Factoryperforms JSON-schema validation and version-based record-preparer selection inline, which couples the indexer core to the JSON ingestion format. To support other formats (e.g. protobuf), that logic needs to live behind a format-neutral seam.What
Indexer::IngestionAdapter: an interface (handles_event?,validate_event) plusValidationResult/Failurevalue typesOperation::Factoryroutes each event to an adapter: the first whosehandles_event?returns true, with a sole available adapter receiving all events (so its more specific failure messages are used); events no adapter recognizes becomeFailedEventErrorsIngestionAdapter::JSONEvents, behind the seam but still withinelasticgraph-indexer(it moves out in the next PR)Verification
script/lint,script/type_checkscript/run_gem_specs elasticgraph-indexer(100% line + branch coverage)script/run_gem_specsfor indexer_lambda, warehouse_lambda, localStack
Current PR is marked with
->.