Skip to content

Source enum values from external proto enums and reference external enum types#1286

Open
jwils wants to merge 3 commits into
joshuaw/proto-syntax-headersfrom
joshuaw/protobuf-external-types
Open

Source enum values from external proto enums and reference external enum types#1286
jwils wants to merge 3 commits into
joshuaw/proto-syntax-headersfrom
joshuaw/protobuf-external-types

Conversation

@jwils

@jwils jwils commented Jul 1, 2026

Copy link
Copy Markdown
Collaborator

Why

Some schemas already have canonical protobuf enum definitions. Generating duplicate enums in schema.proto makes Java consumers deal with parallel types, and maintaining enum value lists in two places invites drift.

What

Adds external_proto_enum on enum types, per the API inversion suggested in review — configuration lives on the enum type itself instead of a global hash imported from application code:

schema.enum_type "Currency" do |t|
  t.values "USD", "CAD"
  t.external_proto_enum MyApp::Protos::Currency,
    exclusions: [:UNKNOWN_DO_NOT_USE],
    name_transform: ->(name) { name.sub(/\ACURRENCY_/, "") }
end
  • Commit 1 — value sourcing: the generated proto enum's values come from the proto enum class (anything exposing .enums), honoring exclusions, expected_extras, and name_transform keyword args. Multiple sources may be registered and must agree.
  • Commit 2 — external references: passing proto: and import: goes further — schema.proto imports the named file and references the existing enum type instead of generating a local duplicate. A referenced enum must have exactly one option-free source whose values match the ElasticGraph enum's values, so transformed or curated enums stay generated locally.

Because the API lives on enum types and options are real keyword arguments, several things from the previous revision disappear entirely: the global-hash normalization and string/symbol key handling, the extension state field, and two validation error cases that are now impossible by construction (an external reference without a value source, and referencing a non-enum type).

Risk Assessment

Low — only affects the unreleased elasticgraph-proto_ingestion extension, and both features are opt-in with eager validation.

References

@jwils
jwils force-pushed the joshuaw/protobuf-schema-generation branch 4 times, most recently from 474788d to bcc30c1 Compare July 5, 2026 15:39
@jwils
jwils force-pushed the joshuaw/protobuf-external-types branch from 723577d to 362dde9 Compare July 5, 2026 15:44
@jwils
jwils force-pushed the joshuaw/protobuf-schema-generation branch 2 times, most recently from 722f460 to 1764684 Compare July 7, 2026 15:14
@jwils
jwils force-pushed the joshuaw/protobuf-external-types branch from 362dde9 to b140cc6 Compare July 7, 2026 15:14
@jwils
jwils force-pushed the joshuaw/protobuf-schema-generation branch from 1764684 to ba76771 Compare July 7, 2026 15:27
@jwils
jwils force-pushed the joshuaw/protobuf-external-types branch from b140cc6 to 61d2c86 Compare July 7, 2026 15:27
@jwils
jwils force-pushed the joshuaw/protobuf-schema-generation branch from ba76771 to 5725228 Compare July 7, 2026 16:22
@jwils
jwils force-pushed the joshuaw/protobuf-external-types branch from 61d2c86 to 06b8a2b Compare July 7, 2026 16:22
@jwils
jwils force-pushed the joshuaw/protobuf-schema-generation branch from 5725228 to f70244d Compare July 7, 2026 16:23
@jwils
jwils force-pushed the joshuaw/protobuf-external-types branch from 06b8a2b to a2e1ff0 Compare July 7, 2026 16:23
@jwils
jwils force-pushed the joshuaw/protobuf-schema-generation branch from f70244d to 653a94f Compare July 7, 2026 16:50
@jwils
jwils force-pushed the joshuaw/protobuf-external-types branch from a2e1ff0 to cd2b49f Compare July 7, 2026 16:50
@jwils
jwils force-pushed the joshuaw/protobuf-schema-generation branch from 653a94f to 313e89c Compare July 7, 2026 18:01
@jwils
jwils force-pushed the joshuaw/protobuf-external-types branch from cd2b49f to f3f5f66 Compare July 7, 2026 18:02
@jwils
jwils force-pushed the joshuaw/protobuf-schema-generation branch from 313e89c to 405c9bd Compare July 8, 2026 22:59
@jwils
jwils force-pushed the joshuaw/protobuf-external-types branch from f3f5f66 to 8cd283f Compare July 8, 2026 22:59
@jwils jwils changed the title Reference external proto enum types Source enum values from existing proto enums and reference external enum types Jul 8, 2026
@jwils
jwils force-pushed the joshuaw/protobuf-schema-generation branch from 405c9bd to a4edbdf Compare July 8, 2026 23:21
@jwils jwils changed the title Source enum values from existing proto enums and reference external enum types Source enum values from external proto enums and reference external enum types Jul 9, 2026
@jwils
jwils force-pushed the joshuaw/protobuf-external-types branch from 0a35bef to 8f32460 Compare July 9, 2026 16:00
@jwils
jwils force-pushed the joshuaw/proto-syntax-headers branch 2 times, most recently from 3396a37 to 98fac03 Compare July 9, 2026 17:29
@jwils
jwils force-pushed the joshuaw/protobuf-external-types branch 2 times, most recently from 158b6f8 to ed900f2 Compare July 9, 2026 18:13
@jwils
jwils force-pushed the joshuaw/proto-syntax-headers branch from 98fac03 to b3029f8 Compare July 9, 2026 18:13
@jwils
jwils force-pushed the joshuaw/protobuf-external-types branch from ed900f2 to 3c7024a Compare July 10, 2026 15:44
@jwils
jwils force-pushed the joshuaw/proto-syntax-headers branch from b3029f8 to d837b3a Compare July 10, 2026 15:44
@jwils
jwils force-pushed the joshuaw/protobuf-external-types branch from 3c7024a to dd523aa Compare July 10, 2026 15:52
@jwils
jwils force-pushed the joshuaw/proto-syntax-headers branch 2 times, most recently from 6f1b1b1 to d4471ca Compare July 11, 2026 13:38
@jwils
jwils force-pushed the joshuaw/protobuf-external-types branch from dd523aa to 593342e Compare July 11, 2026 13:40
@jwils
jwils force-pushed the joshuaw/proto-syntax-headers branch from d4471ca to 5eb990f Compare July 11, 2026 13:54
@jwils
jwils force-pushed the joshuaw/protobuf-external-types branch from 593342e to ec4819c Compare July 11, 2026 13:54
@jwils
jwils force-pushed the joshuaw/proto-syntax-headers branch from 5eb990f to d40ff2e Compare July 11, 2026 14:00
@jwils
jwils force-pushed the joshuaw/protobuf-external-types branch from ec4819c to 7a122a7 Compare July 11, 2026 14:00
@jwils
jwils force-pushed the joshuaw/proto-syntax-headers branch from d40ff2e to 40be848 Compare July 11, 2026 14:14
@jwils
jwils force-pushed the joshuaw/protobuf-external-types branch 2 times, most recently from 4d94fd9 to 529dcb5 Compare July 11, 2026 14:24
@jwils
jwils force-pushed the joshuaw/proto-syntax-headers branch from 7b9a2b7 to c8b5e5e Compare July 15, 2026 23:52
@jwils
jwils force-pushed the joshuaw/protobuf-external-types branch 2 times, most recently from 170323f to 5f29c5a Compare July 16, 2026 14:21
@jwils
jwils force-pushed the joshuaw/proto-syntax-headers branch from c8b5e5e to 020eb17 Compare July 16, 2026 14:21
jwils added 3 commits July 17, 2026 10:03
If a project already has canonical protobuf enum definitions, maintaining
the value list in two places invites drift. `external_proto_enum` on an
enum type sources the generated proto enum's values from an existing proto
enum class (anything exposing `.enums`), honoring `exclusions`,
`expected_extras`, and `name_transform` options. Multiple sources may be
registered and must resolve to the same value set.

Configuration lives on the enum type itself (per review feedback on the
earlier global-hash `proto_enum_mappings` API), so options are real
keyword arguments and no extension state or hash normalization is needed.
Some schemas already have canonical protobuf enum definitions, and
generating duplicate enums in `schema.proto` makes consumers deal with
parallel types. Passing `proto:` and `import:` to `external_proto_enum`
imports the named proto file and references the existing enum type
directly instead of generating a local definition.

A referenced enum must have exactly one option-free source whose values
match the ElasticGraph enum's values, so transformed or curated enums stay
generated locally. Since the API lives on enum types, non-enum external
references are impossible by construction rather than needing validation.
@jwils
jwils force-pushed the joshuaw/proto-syntax-headers branch from 020eb17 to 9a7765f Compare July 17, 2026 15:05
@jwils
jwils force-pushed the joshuaw/protobuf-external-types branch from 5f29c5a to 6fc64b7 Compare July 17, 2026 15:05
jwils added a commit that referenced this pull request Jul 20, 2026
## Why
- continue the pluggable ingestion serializer proposal after pulling
JSON Schema into its own gem
- revive the earlier protobuf prototype from #1056 on top of the new
serializer extension points

## What
- fill in the `elasticgraph-proto_ingestion` extension gem with core
generation: `schema_artifacts:dump` emits a `proto3` `schema.proto`
covering the schema's indexed types
- map built-in ElasticGraph scalars to proto types, with `t.protobuf
type:` for custom scalars (resolved via
`type_ref.with_reverted_override` so built-ins renamed with
`type_name_overrides` keep working)
- generate messages for object/interface/union types and enums (with a
zero-valued `*_UNSPECIFIED` entry), escaping proto reserved words and
wrapping lists of lists so the output stays valid
- keep `schema.proto` on public GraphQL field names; validate proto
package names
- hold extension state on a `ProtoIngestionState` container behind a
single `proto_ingestion_state` reader (matching #1281)

Field and enum value numbers are assigned sequentially in definition
order in this PR; the stacked follow-up adds the
`proto_field_numbers.yaml` sidecar that keeps them wire-stable across
schema evolution.

## Stacked follow-ups
1. this PR — core `schema.proto` generation
2. wire-stable field/enum value numbers via a `proto_field_numbers.yaml`
sidecar
3. `syntax: :proto2` support and custom file-level `headers:`
4. #1286 — enum value sourcing from existing proto enums + external
proto type references

## Verification
- `script/run_gem_specs elasticgraph-proto_ingestion` (100% line +
branch coverage at this commit)
- `script/type_check`, `script/lint`, `script/spellcheck`
- `script/quick_build` green at the stack head (whose tree is identical
to the previously reviewed single-PR revision)

## References
- #1059
- #1056
- #1079

## Update — 2026-07-10
- The current stack is #1080#1304#1306#1305#1286.
- Proto extension state now uses a mutable Struct, and keyword
package-name segments are validated without being rewritten.
- Lists of lists now raise an actionable schema error instead of
generating wrapper messages; this supersedes the earlier wrapping note
above.
---

## Update — 2026-07-15
- Interface and union messages now wrap concrete subtype messages in a
`oneof`, matching the JSON Schema `oneOf` representation.
- Concrete subtype messages omit the redundant `__typename`
discriminator.
- Proto type rendering is now stateless: the generator selects the
reachable type graph up front, then each extended type renders itself
without mutating shared traversal state.
- No-block extension coverage now completes the definitions so the
fixture remains valid under CI GraphQL-schema validation.

---

## Update — 2026-07-19
- Replaced keyword suffixing with fully qualified local message and enum
references, preserving source type and field names while disambiguating
contextual protobuf words and built-in scalar names.
- Removed the now-unnecessary keyword collision tracking and verified a
generated schema containing contextual names with `protoc` 35.1.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant