Skip to content

Support proto2 syntax and custom proto file headers#1305

Open
jwils wants to merge 7 commits into
joshuaw/proto-timestampfrom
joshuaw/proto-syntax-headers
Open

Support proto2 syntax and custom proto file headers#1305
jwils wants to merge 7 commits into
joshuaw/proto-timestampfrom
joshuaw/proto-syntax-headers

Conversation

@jwils

@jwils jwils commented Jul 8, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Why

Some consumers need the generated messages to reference proto2 types — protoc forbids a proto3 message from referencing a proto2 enum — and language-specific file options (like java_package) shouldn't require baking any convention into the gem.

What

proto_schema_artifacts gains two options:

  • syntax: :proto2 emits a proto2 file instead of the default proto3, labeling every field optional or repeated
  • headers: injects file-level lines (such as option declarations) verbatim as a contiguous section after the package declaration

Risk Assessment

Low — only affects the unreleased elasticgraph-proto_ingestion extension; both options are opt-in.

References

Update — 2026-07-10

This PR is stacked directly on #1306 (→ #1304 → #1080).

@jwils jwils mentioned this pull request Jul 8, 2026
Open
@jwils
jwils force-pushed the joshuaw/proto-field-numbers branch from 67b8198 to f62a859 Compare July 8, 2026 23:40
@jwils
jwils force-pushed the joshuaw/proto-syntax-headers branch 2 times, most recently from 9b72525 to 3396a37 Compare July 9, 2026 16:00
@jwils
jwils changed the base branch from joshuaw/proto-field-numbers to joshuaw/proto-timestamp July 9, 2026 16:00
@jwils
jwils force-pushed the joshuaw/proto-syntax-headers branch from 3396a37 to 98fac03 Compare July 9, 2026 17:29
@jwils
jwils force-pushed the joshuaw/proto-timestamp branch from 5a2e67d to 1537869 Compare July 9, 2026 17:29
@jwils
jwils force-pushed the joshuaw/proto-syntax-headers branch from 98fac03 to b3029f8 Compare July 9, 2026 18:13
@jwils
jwils force-pushed the joshuaw/proto-timestamp branch 2 times, most recently from 721fe97 to d516c31 Compare July 10, 2026 15:44
@jwils
jwils force-pushed the joshuaw/proto-syntax-headers branch from b3029f8 to d837b3a Compare July 10, 2026 15:44
@jwils jwils mentioned this pull request Jul 10, 2026
Merged
@jwils
jwils force-pushed the joshuaw/proto-syntax-headers branch from d837b3a to 6f1b1b1 Compare July 10, 2026 15:52
@jwils
jwils force-pushed the joshuaw/proto-timestamp branch 2 times, most recently from 9e7369b to 9e2b508 Compare July 11, 2026 13:36
@jwils
jwils force-pushed the joshuaw/proto-syntax-headers branch from 6f1b1b1 to d4471ca Compare July 11, 2026 13:38
@jwils
jwils force-pushed the joshuaw/proto-timestamp branch from 9e2b508 to 06e1531 Compare July 11, 2026 13:54
@jwils
jwils force-pushed the joshuaw/proto-syntax-headers branch from d4471ca to 5eb990f Compare July 11, 2026 13:54
@jwils
jwils force-pushed the joshuaw/proto-timestamp branch from 06e1531 to ee20f41 Compare July 11, 2026 14:00
@jwils
jwils force-pushed the joshuaw/proto-syntax-headers branch 2 times, most recently from d40ff2e to 40be848 Compare July 11, 2026 14:14
@jwils
jwils force-pushed the joshuaw/proto-timestamp branch from 508c016 to 4a6aa37 Compare July 15, 2026 23:52
@jwils
jwils force-pushed the joshuaw/proto-syntax-headers branch from 7b9a2b7 to c8b5e5e Compare July 15, 2026 23:52
@jwils
jwils force-pushed the joshuaw/proto-syntax-headers branch from c8b5e5e to 020eb17 Compare July 16, 2026 14:21
@jwils
jwils force-pushed the joshuaw/proto-timestamp branch from 4a6aa37 to 1aa384b Compare July 16, 2026 14:21
jwils added 7 commits July 17, 2026 09:55
@jwils
Generate a protobuf schema artifact from ElasticGraph schemas
8d05d78 
Fill in elasticgraph-proto_ingestion with proto schema generation for indexed types. Map built-in and custom scalars, render concrete and abstract types, preserve public field names, and generate enum defaults. Reject unrepresentable lists and unstable identifier collisions with actionable schema errors.
@jwils
Pin protobuf field and enum value numbers in a sidecar artifact
f2ef14e 
Field and enum value numbers were assigned sequentially in definition
order, so reordering or removing a field renumbered everything after it —
breaking wire compatibility with previously serialized data.

`schema_artifacts:dump` now reads and writes a `proto_field_numbers.yaml`
sidecar artifact:

- Existing numbers stay fixed even if field order changes; new fields get
  the next available numbers.
- `schema.proto` keeps the public GraphQL field names while the sidecar
  stores private `name_in_index` overrides.
- A field renamed with `field.renamed_from` reuses its existing number
  under the new public name.
- Enum value numbers are pinned in an `enums` section; removed values keep
  their numbers reserved so they are never reused (`0` remains the
  generated `*_UNSPECIFIED` value).
- Mappings are validated: collisions raise clear errors, and non-integer
  numbers are rejected instead of letting `Integer()` silently truncate.

`configure_proto_field_number_mappings` allows tests and advanced callers
to seed mappings without an artifact file.
@jwils
Cover protobuf numbering boundaries
8803f5e
@jwils
Map DateTime to google.protobuf.Timestamp and document temporal strin…
bf8fd07 
…g formats

`DateTime` fields now use the well-known `google.protobuf.Timestamp` type
(with `google/protobuf/timestamp.proto` imported automatically) instead of
`string`. A Timestamp cannot be malformed the way a string can, and proto
consumers get language-native timestamp types. Note that a Timestamp is a
UTC instant, so a publisher's original UTC offset is not preserved;
`t.protobuf type: "string"` remains available to override.

To support this, `t.protobuf` gains two options usable by any scalar:

- `import:` maps a scalar to an externally defined proto type, emitting the
  needed `import` statement in `schema.proto`.
- `comment:` documents the expected format on each generated field, used by
  the built-in `string`-typed temporal scalars (`Date`, `LocalTime`,
  `TimeZone`) whose proto type is wider than the ElasticGraph type (e.g.
  `// ISO 8601 date`). Values are still validated at ingestion time, just
  as with JSON ingestion.
@jwils
Test protobuf import stability
b94fd0e
@jwils
Support proto2 syntax and custom proto file headers
bac4cdc 
`proto_schema_artifacts` gains two options:

- `syntax: :proto2` emits a proto2 file instead of the default proto3
  (labeling every field `optional` or `repeated`). This is useful when the
  generated messages need to reference proto2 types — `protoc` forbids a
  proto3 message from referencing a proto2 enum.
- `headers:` injects file-level lines (such as `option` declarations)
  verbatim as a contiguous section after the `package` declaration, so
  language-specific options can be set without baking any particular
  convention into the gem.
@jwils
Test mixed protobuf header validation
9a7765f
@jwils
jwils force-pushed the joshuaw/proto-timestamp branch from 1aa384b to b94fd0e Compare July 17, 2026 15:05
@jwils
jwils force-pushed the joshuaw/proto-syntax-headers branch from 020eb17 to 9a7765f Compare July 17, 2026 15:05
jwils added a commit that referenced this pull request Jul 20, 2026
@jwils
Generate a protobuf schema artifact from ElasticGraph schemas (#1080)
48add3e 
## Why
- continue the pluggable ingestion serializer proposal after pulling
JSON Schema into its own gem
- revive the earlier protobuf prototype from #1056 on top of the new
serializer extension points

## What
- fill in the `elasticgraph-proto_ingestion` extension gem with core
generation: `schema_artifacts:dump` emits a `proto3` `schema.proto`
covering the schema's indexed types
- map built-in ElasticGraph scalars to proto types, with `t.protobuf
type:` for custom scalars (resolved via
`type_ref.with_reverted_override` so built-ins renamed with
`type_name_overrides` keep working)
- generate messages for object/interface/union types and enums (with a
zero-valued `*_UNSPECIFIED` entry), escaping proto reserved words and
wrapping lists of lists so the output stays valid
- keep `schema.proto` on public GraphQL field names; validate proto
package names
- hold extension state on a `ProtoIngestionState` container behind a
single `proto_ingestion_state` reader (matching #1281)

Field and enum value numbers are assigned sequentially in definition
order in this PR; the stacked follow-up adds the
`proto_field_numbers.yaml` sidecar that keeps them wire-stable across
schema evolution.

## Stacked follow-ups
1. this PR — core `schema.proto` generation
2. wire-stable field/enum value numbers via a `proto_field_numbers.yaml`
sidecar
3. `syntax: :proto2` support and custom file-level `headers:`
4. #1286 — enum value sourcing from existing proto enums + external
proto type references

## Verification
- `script/run_gem_specs elasticgraph-proto_ingestion` (100% line +
branch coverage at this commit)
- `script/type_check`, `script/lint`, `script/spellcheck`
- `script/quick_build` green at the stack head (whose tree is identical
to the previously reviewed single-PR revision)

## References
- #1059
- #1056
- #1079

## Update — 2026-07-10
- The current stack is #1080 → #1304 → #1306 → #1305 → #1286.
- Proto extension state now uses a mutable Struct, and keyword
package-name segments are validated without being rewritten.
- Lists of lists now raise an actionable schema error instead of
generating wrapper messages; this supersedes the earlier wrapping note
above.
---

## Update — 2026-07-15
- Interface and union messages now wrap concrete subtype messages in a
`oneof`, matching the JSON Schema `oneOf` representation.
- Concrete subtype messages omit the redundant `__typename`
discriminator.
- Proto type rendering is now stateless: the generator selects the
reachable type graph up front, then each extended type renders itself
without mutating shared traversal state.
- No-block extension coverage now completes the definitions so the
fixture remains valid under CI GraphQL-schema validation.

---

## Update — 2026-07-19
- Replaced keyword suffixing with fully qualified local message and enum
references, preserving source type and field names while disambiguating
contextual protobuf words and built-in scalar names.
- Removed the now-unnecessary keyword collision tracking and verified a
generated schema containing contextual names with `protoc` 35.1.
@jwils
jwils force-pushed the joshuaw/proto-timestamp branch from b94fd0e to d676254 Compare July 20, 2026 02:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@BrianSigafoos-SQ BrianSigafoos-SQ Awaiting requested review from BrianSigafoos-SQ BrianSigafoos-SQ is a code owner
@bsorbo bsorbo Awaiting requested review from bsorbo bsorbo is a code owner
@jwondrusch jwondrusch Awaiting requested review from jwondrusch jwondrusch is a code owner
@myronmarston myronmarston Awaiting requested review from myronmarston myronmarston is a code owner
@marcdaniels-toast marcdaniels-toast Awaiting requested review from marcdaniels-toast marcdaniels-toast is a code owner
@ellisandrews-toast ellisandrews-toast Awaiting requested review from ellisandrews-toast ellisandrews-toast is a code owner
@rossroberts-toast rossroberts-toast Awaiting requested review from rossroberts-toast rossroberts-toast is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@jwils