Source enum values from external proto enums and reference external enum types#1286
Open
jwils wants to merge 3 commits into
Open
Source enum values from external proto enums and reference external enum types#1286jwils wants to merge 3 commits into
jwils wants to merge 3 commits into
Conversation
jwils
requested review from
BrianSigafoos-SQ,
bsorbo,
ellisandrews-toast,
jwondrusch,
marcdaniels-toast,
myronmarston and
rossroberts-toast
as code owners
July 1, 2026 20:55
jwils
force-pushed
the
joshuaw/protobuf-schema-generation
branch
4 times, most recently
from
July 5, 2026 15:39
474788d to
bcc30c1
Compare
jwils
force-pushed
the
joshuaw/protobuf-external-types
branch
from
July 5, 2026 15:44
723577d to
362dde9
Compare
jwils
force-pushed
the
joshuaw/protobuf-schema-generation
branch
2 times, most recently
from
July 7, 2026 15:14
722f460 to
1764684
Compare
jwils
force-pushed
the
joshuaw/protobuf-external-types
branch
from
July 7, 2026 15:14
362dde9 to
b140cc6
Compare
jwils
force-pushed
the
joshuaw/protobuf-schema-generation
branch
from
July 7, 2026 15:27
1764684 to
ba76771
Compare
jwils
force-pushed
the
joshuaw/protobuf-external-types
branch
from
July 7, 2026 15:27
b140cc6 to
61d2c86
Compare
jwils
force-pushed
the
joshuaw/protobuf-schema-generation
branch
from
July 7, 2026 16:22
ba76771 to
5725228
Compare
jwils
force-pushed
the
joshuaw/protobuf-external-types
branch
from
July 7, 2026 16:22
61d2c86 to
06b8a2b
Compare
jwils
force-pushed
the
joshuaw/protobuf-schema-generation
branch
from
July 7, 2026 16:23
5725228 to
f70244d
Compare
jwils
force-pushed
the
joshuaw/protobuf-external-types
branch
from
July 7, 2026 16:23
06b8a2b to
a2e1ff0
Compare
jwils
force-pushed
the
joshuaw/protobuf-schema-generation
branch
from
July 7, 2026 16:50
f70244d to
653a94f
Compare
jwils
force-pushed
the
joshuaw/protobuf-external-types
branch
from
July 7, 2026 16:50
a2e1ff0 to
cd2b49f
Compare
jwils
force-pushed
the
joshuaw/protobuf-schema-generation
branch
from
July 7, 2026 18:01
653a94f to
313e89c
Compare
jwils
force-pushed
the
joshuaw/protobuf-external-types
branch
from
July 7, 2026 18:02
cd2b49f to
f3f5f66
Compare
jwils
force-pushed
the
joshuaw/protobuf-schema-generation
branch
from
July 8, 2026 22:59
313e89c to
405c9bd
Compare
jwils
force-pushed
the
joshuaw/protobuf-external-types
branch
from
July 8, 2026 22:59
f3f5f66 to
8cd283f
Compare
jwils
force-pushed
the
joshuaw/protobuf-schema-generation
branch
from
July 8, 2026 23:21
405c9bd to
a4edbdf
Compare
jwils
force-pushed
the
joshuaw/protobuf-external-types
branch
from
July 9, 2026 16:00
0a35bef to
8f32460
Compare
jwils
force-pushed
the
joshuaw/proto-syntax-headers
branch
2 times, most recently
from
July 9, 2026 17:29
3396a37 to
98fac03
Compare
jwils
force-pushed
the
joshuaw/protobuf-external-types
branch
2 times, most recently
from
July 9, 2026 18:13
158b6f8 to
ed900f2
Compare
jwils
force-pushed
the
joshuaw/proto-syntax-headers
branch
from
July 9, 2026 18:13
98fac03 to
b3029f8
Compare
jwils
force-pushed
the
joshuaw/protobuf-external-types
branch
from
July 10, 2026 15:44
ed900f2 to
3c7024a
Compare
jwils
force-pushed
the
joshuaw/proto-syntax-headers
branch
from
July 10, 2026 15:44
b3029f8 to
d837b3a
Compare
jwils
force-pushed
the
joshuaw/protobuf-external-types
branch
from
July 10, 2026 15:52
3c7024a to
dd523aa
Compare
jwils
force-pushed
the
joshuaw/proto-syntax-headers
branch
2 times, most recently
from
July 11, 2026 13:38
6f1b1b1 to
d4471ca
Compare
jwils
force-pushed
the
joshuaw/protobuf-external-types
branch
from
July 11, 2026 13:40
dd523aa to
593342e
Compare
jwils
force-pushed
the
joshuaw/proto-syntax-headers
branch
from
July 11, 2026 13:54
d4471ca to
5eb990f
Compare
jwils
force-pushed
the
joshuaw/protobuf-external-types
branch
from
July 11, 2026 13:54
593342e to
ec4819c
Compare
jwils
force-pushed
the
joshuaw/proto-syntax-headers
branch
from
July 11, 2026 14:00
5eb990f to
d40ff2e
Compare
jwils
force-pushed
the
joshuaw/protobuf-external-types
branch
from
July 11, 2026 14:00
ec4819c to
7a122a7
Compare
jwils
force-pushed
the
joshuaw/proto-syntax-headers
branch
from
July 11, 2026 14:14
d40ff2e to
40be848
Compare
jwils
force-pushed
the
joshuaw/protobuf-external-types
branch
2 times, most recently
from
July 11, 2026 14:24
4d94fd9 to
529dcb5
Compare
jwils
force-pushed
the
joshuaw/proto-syntax-headers
branch
from
July 15, 2026 23:52
7b9a2b7 to
c8b5e5e
Compare
jwils
force-pushed
the
joshuaw/protobuf-external-types
branch
2 times, most recently
from
July 16, 2026 14:21
170323f to
5f29c5a
Compare
jwils
force-pushed
the
joshuaw/proto-syntax-headers
branch
from
July 16, 2026 14:21
c8b5e5e to
020eb17
Compare
If a project already has canonical protobuf enum definitions, maintaining the value list in two places invites drift. `external_proto_enum` on an enum type sources the generated proto enum's values from an existing proto enum class (anything exposing `.enums`), honoring `exclusions`, `expected_extras`, and `name_transform` options. Multiple sources may be registered and must resolve to the same value set. Configuration lives on the enum type itself (per review feedback on the earlier global-hash `proto_enum_mappings` API), so options are real keyword arguments and no extension state or hash normalization is needed.
Some schemas already have canonical protobuf enum definitions, and generating duplicate enums in `schema.proto` makes consumers deal with parallel types. Passing `proto:` and `import:` to `external_proto_enum` imports the named proto file and references the existing enum type directly instead of generating a local definition. A referenced enum must have exactly one option-free source whose values match the ElasticGraph enum's values, so transformed or curated enums stay generated locally. Since the API lives on enum types, non-enum external references are impossible by construction rather than needing validation.
jwils
force-pushed
the
joshuaw/proto-syntax-headers
branch
from
July 17, 2026 15:05
020eb17 to
9a7765f
Compare
jwils
force-pushed
the
joshuaw/protobuf-external-types
branch
from
July 17, 2026 15:05
5f29c5a to
6fc64b7
Compare
jwils
added a commit
that referenced
this pull request
Jul 20, 2026
## Why - continue the pluggable ingestion serializer proposal after pulling JSON Schema into its own gem - revive the earlier protobuf prototype from #1056 on top of the new serializer extension points ## What - fill in the `elasticgraph-proto_ingestion` extension gem with core generation: `schema_artifacts:dump` emits a `proto3` `schema.proto` covering the schema's indexed types - map built-in ElasticGraph scalars to proto types, with `t.protobuf type:` for custom scalars (resolved via `type_ref.with_reverted_override` so built-ins renamed with `type_name_overrides` keep working) - generate messages for object/interface/union types and enums (with a zero-valued `*_UNSPECIFIED` entry), escaping proto reserved words and wrapping lists of lists so the output stays valid - keep `schema.proto` on public GraphQL field names; validate proto package names - hold extension state on a `ProtoIngestionState` container behind a single `proto_ingestion_state` reader (matching #1281) Field and enum value numbers are assigned sequentially in definition order in this PR; the stacked follow-up adds the `proto_field_numbers.yaml` sidecar that keeps them wire-stable across schema evolution. ## Stacked follow-ups 1. this PR — core `schema.proto` generation 2. wire-stable field/enum value numbers via a `proto_field_numbers.yaml` sidecar 3. `syntax: :proto2` support and custom file-level `headers:` 4. #1286 — enum value sourcing from existing proto enums + external proto type references ## Verification - `script/run_gem_specs elasticgraph-proto_ingestion` (100% line + branch coverage at this commit) - `script/type_check`, `script/lint`, `script/spellcheck` - `script/quick_build` green at the stack head (whose tree is identical to the previously reviewed single-PR revision) ## References - #1059 - #1056 - #1079 ## Update — 2026-07-10 - The current stack is #1080 → #1304 → #1306 → #1305 → #1286. - Proto extension state now uses a mutable Struct, and keyword package-name segments are validated without being rewritten. - Lists of lists now raise an actionable schema error instead of generating wrapper messages; this supersedes the earlier wrapping note above. --- ## Update — 2026-07-15 - Interface and union messages now wrap concrete subtype messages in a `oneof`, matching the JSON Schema `oneOf` representation. - Concrete subtype messages omit the redundant `__typename` discriminator. - Proto type rendering is now stateless: the generator selects the reachable type graph up front, then each extended type renders itself without mutating shared traversal state. - No-block extension coverage now completes the definitions so the fixture remains valid under CI GraphQL-schema validation. --- ## Update — 2026-07-19 - Replaced keyword suffixing with fully qualified local message and enum references, preserving source type and field names while disambiguating contextual protobuf words and built-in scalar names. - Removed the now-unnecessary keyword collision tracking and verified a generated schema containing contextual names with `protoc` 35.1.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Some schemas already have canonical protobuf enum definitions. Generating duplicate enums in
schema.protomakes Java consumers deal with parallel types, and maintaining enum value lists in two places invites drift.What
Adds
external_proto_enumon enum types, per the API inversion suggested in review — configuration lives on the enum type itself instead of a global hash imported from application code:.enums), honoringexclusions,expected_extras, andname_transformkeyword args. Multiple sources may be registered and must agree.proto:andimport:goes further —schema.protoimports the named file and references the existing enum type instead of generating a local duplicate. A referenced enum must have exactly one option-free source whose values match the ElasticGraph enum's values, so transformed or curated enums stay generated locally.Because the API lives on enum types and options are real keyword arguments, several things from the previous revision disappear entirely: the global-hash normalization and string/symbol key handling, the extension state field, and two validation error cases that are now impossible by construction (an external reference without a value source, and referencing a non-enum type).
Risk Assessment
Low — only affects the unreleased
elasticgraph-proto_ingestionextension, and both features are opt-in with eager validation.References