Support proto2 syntax and custom proto file headers

elasticgraph-proto_ingestion/README.md

@@ -1,6 +1,9 @@
 # ElasticGraph::ProtoIngestion
 
 An ElasticGraph extension that supports ingesting Protocol Buffer data into ElasticGraph.
+Currently it generates Protocol Buffers schema artifacts from ElasticGraph schemas: it emits
+`proto3` by default and can emit `proto2`, and supports arbitrary file-level headers (such
+as `option` declarations).
 
 ## Dependency Diagram
 
@@ -15,3 +18,222 @@ graph LR;
     elasticgraph-proto_ingestion --> elasticgraph-support;
     class elasticgraph-support otherEgGemStyle;
 ```
+
+## Usage
+
+First, add `elasticgraph-proto_ingestion` to your `Gemfile`, alongside the other ElasticGraph gems:
+
+```diff
+diff --git a/Gemfile b/Gemfile
+index 4a5ef1e..5c16c2b 100644
+--- a/Gemfile
++++ b/Gemfile
+@@ -8,6 +8,7 @@ gem "elasticgraph-query_registry", *elasticgraph_details
+
+ # Can be elasticgraph-elasticsearch or elasticgraph-opensearch based on the datastore you want to use.
+ gem "elasticgraph-opensearch", *elasticgraph_details
++gem "elasticgraph-proto_ingestion", *elasticgraph_details
+
+ gem "httpx", "~> 1.3"
+
+```
+
+Next, update your `Rakefile` so that `ElasticGraph::ProtoIngestion::SchemaDefinition::APIExtension` is
+included in the schema-definition extension modules:
+
+```diff
+diff --git a/Rakefile b/Rakefile
+index 2943335..26633c3 100644
+--- a/Rakefile
++++ b/Rakefile
+@@ -3,5 +3,6 @@
+ require "elastic_graph/json_ingestion/schema_definition/api_extension"
+ require "elastic_graph/local/rake_tasks"
++require "elastic_graph/proto_ingestion/schema_definition/api_extension"
+ require "elastic_graph/query_registry/rake_tasks"
+ require "rspec/core/rake_task"
+ require "standard/rake"
+@@ -16,6 +17,7 @@ ElasticGraph::Local::RakeTasks.new(
+   # Determines casing of field names. Can be either `:camelCase` or `:snake_case`.
+   tasks.schema_element_name_form = :camelCase
+   tasks.schema_definition_extension_modules << ElasticGraph::JSONIngestion::SchemaDefinition::APIExtension
++  tasks.schema_definition_extension_modules << ElasticGraph::ProtoIngestion::SchemaDefinition::APIExtension
+
+   # Customizes the names of fields generated by ElasticGraph.
+   tasks.schema_element_name_overrides = {
+```
+
+Adding the schema definition extension automatically enables `schema.proto` generation with the default
+`elasticgraph` package. Optionally, configure a custom package name from your schema definition:
+
+```ruby
+# in config/schema/protobuf.rb
+
+ElasticGraph.define_schema do |schema|
+  schema.proto_schema_artifacts package_name: "myapp.events.v1"
+end
+```
+
+After running `bundle exec rake schema_artifacts:dump`, ElasticGraph will generate:
+
+- `schema.proto`
+- `proto_field_numbers.yaml`
+
+## Schema Definition API
+
+### Protobuf Syntax (`proto2` / `proto3`)
+
+`proto_schema_artifacts` emits `proto3` by default. Pass `syntax: :proto2` to emit a proto2 file
+instead (every field is then labeled `optional` or `repeated`). This is useful when the generated
+messages need to reference proto2 types — for example, `protoc` forbids a `proto3` message from
+referencing a `proto2` enum:
+
+```ruby
+# in config/schema/protobuf.rb
+
+ElasticGraph.define_schema do |schema|
+  schema.proto_schema_artifacts package_name: "myapp.events.v1", syntax: :proto2
+end
+```
+
+### Custom Headers
+
+Pass `headers:` an array of strings to inject file-level lines (such as `option` declarations)
+verbatim, as a contiguous section immediately after the `package` declaration. This lets you set
+language-specific options without the gem baking in any particular convention:
+
+```ruby
+# in config/schema/protobuf.rb
+
+ElasticGraph.define_schema do |schema|
+  schema.proto_schema_artifacts(
+    package_name: "myapp.events.v1",
+    headers: [
+      %(option java_package = "com.myapp.events";),
+      "option java_multiple_files = true;"
+    ]
+  )
+end
+```
+
+produces:
+
+```text
+syntax = "proto3";
+
+package myapp.events.v1;
+
+option java_package = "com.myapp.events";
+option java_multiple_files = true;
+
+// ...messages...
+```
+
+### Custom Scalar Types
+
+Built-in ElasticGraph scalar types are automatically mapped to proto scalar types.
+For custom scalar types, use `protobuf` to define the proto scalar type:
+
+```ruby
+# in config/schema/email.rb
+
+ElasticGraph.define_schema do |schema|
+  schema.scalar_type "Email" do |t|
+    t.mapping type: "keyword"
+    t.json_schema type: "string", format: "email"
+    t.protobuf type: "string"
+  end
+end
+```
+
+A custom scalar can also map to an externally defined proto type by passing `import:` with the
+proto file that defines it, and `comment:` documents the expected format on each generated field
+(useful when the proto type is wider than the ElasticGraph type):
+
+```ruby
+# in config/schema/phone_number.rb
+
+ElasticGraph.define_schema do |schema|
+  schema.scalar_type "PhoneNumber" do |t|
+    t.mapping type: "keyword"
+    t.json_schema type: "string"
+    t.protobuf type: "string", comment: "E.164 phone number"
+  end
+end
+```
+
+### Stable Field Numbers
+
+`schema_artifacts:dump` automatically reads and writes `proto_field_numbers.yaml`
+in the schema artifacts directory. Existing numbers stay fixed even if field order
+changes, and new fields get the next available numbers. Field numbers follow protobuf's
+rules: they must be between 1 and 536,870,911, and the protobuf-reserved 19000-19999
+range is never allocated and is rejected in mappings.
+
+Alternatives inside generated interface and union `oneof` blocks use the same stable
+message-field mappings, so adding or removing a concrete subtype does not renumber the
+remaining alternatives.
+
+`schema.proto` always uses the public GraphQL field names. When a field uses a
+different `name_in_index`, the sidecar YAML stores that override privately:
+
+```yaml
+messages:
+  Widget:
+    fields:
+      id: 1
+      display_name:
+        field_number: 2
+        name_in_index: displayName
+```
+
+If a field is renamed with `field.renamed_from`, `elasticgraph-proto_ingestion` reuses the
+existing field number under the new public field name.
+
+### Stable Enum Value Numbers
+
+Enum value numbers are pinned the same way, in an `enums` section of the sidecar. Existing
+values keep their numbers when other values are added or removed, new values get the next
+available numbers, and removed values keep their numbers reserved so they are never reused
+(number `0` is always the generated `*_UNSPECIFIED` value):
+
+```yaml
+enums:
+  WidgetColor:
+    values:
+      RED: 1
+      BLUE: 2
+```
+
+## Type Mappings
+
+The generated `schema.proto` uses these built-in scalar mappings:
+
+| ElasticGraph Type | Protobuf Type               |
+|-------------------|-----------------------------|
+| `Boolean`         | `bool`                      |
+| `Cursor`          | `string`                    |
+| `Date`            | `string`                    |
+| `DateTime`        | `google.protobuf.Timestamp` |
+| `Float`           | `double`                    |
+| `ID`              | `string`                    |
+| `Int`             | `int32`                     |
+| `JsonSafeLong`    | `int64`                     |
+| `LocalTime`       | `string`                    |
+| `LongString`      | `int64`                     |
+| `String`          | `string`                    |
+| `TimeZone`        | `string`                    |
+| `Untyped`         | `string`                    |
+
+Additionally:
+- `DateTime` uses the [well-known `Timestamp` type](https://protobuf.dev/reference/protobuf/google.protobuf/#timestamp);
+  `schema.proto` imports `google/protobuf/timestamp.proto` automatically. Note that a `Timestamp`
+  is a UTC instant, so a publisher's original UTC offset is not preserved.
+- `string`-typed temporal scalars (`Date`, `LocalTime`, `TimeZone`) are wider than the
+  ElasticGraph types they carry, so generated fields of these types document the expected format
+  in a comment (e.g. `// ISO 8601 date, e.g. "2024-11-25"`). Values are validated when events
+  are ingested, just as with JSON ingestion.
+- List types become `repeated` fields.
+- Lists of lists (e.g. `[[Float!]!]!`) are not supported because Protocol Buffers cannot represent
+  them directly. Schema artifact generation raises an error identifying the unsupported field.
+- Enum types generate `enum` definitions whose values are prefixed with the enum type name in `UPPER_SNAKE_CASE`, including a zero-valued `*_UNSPECIFIED` entry.

elasticgraph-proto_ingestion/elasticgraph-proto_ingestion.gemspec

@@ -36,5 +36,8 @@ Gem::Specification.new do |spec|
 
   spec.add_dependency "elasticgraph-support", ElasticGraph::VERSION
 
+  # This gem's schema-definition extension code references `elasticgraph-schema_definition`, but
+  # applications load it through schema-definition tasks after `elasticgraph-schema_definition` is already
+  # available. Keeping this as a development dependency avoids a runtime dependency cycle.
   spec.add_development_dependency "elasticgraph-schema_definition", ElasticGraph::VERSION
 end

elasticgraph-proto_ingestion/lib/elastic_graph/proto_ingestion/schema_definition/api_extension.rb

@@ -6,7 +6,12 @@
 #
 # frozen_string_literal: true
 
+require "elastic_graph/errors"
 require "elastic_graph/proto_ingestion"
+require "elastic_graph/proto_ingestion/schema_definition/factory_extension"
+require "elastic_graph/proto_ingestion/schema_definition/identifier"
+require "elastic_graph/proto_ingestion/schema_definition/schema"
+require "elastic_graph/proto_ingestion/schema_definition/state_extension"
 
 module ElasticGraph
   module ProtoIngestion
@@ -16,10 +21,66 @@ module ProtoIngestion
     module SchemaDefinition
       # Module designed to be extended onto an {ElasticGraph::SchemaDefinition::API} instance
       # to enable protobuf schema artifact generation.
-      #
-      # @note The protobuf schema artifact generation logic has not been implemented yet, so
-      #   extending this module is currently a no-op.
       module APIExtension
+        # Wires up the protobuf extensions when this module is extended onto an API instance.
+        #
+        # @param api [ElasticGraph::SchemaDefinition::API] the API instance to extend
+        # @return [void]
+        # @api private
+        def self.extended(api)
+          api.state.extend(StateExtension)
+          api.factory.extend(FactoryExtension)
+        end
+
+        # Configures protobuf artifact generation behavior.
+        #
+        # @param package_name [String] proto package name to emit
+        # @param syntax [Symbol] `:proto3` (default) or `:proto2`
+        # @param headers [Array<String>] file-level lines rendered verbatim after the package declaration
+        # @return [void]
+        #
+        # @example Set the proto package name
+        #   ElasticGraph.define_schema do |schema|
+        #     schema.proto_schema_artifacts package_name: "myapp.events.v1"
+        #   end
+        def proto_schema_artifacts(package_name:, syntax: :proto3, headers: [])
+          if !package_name.is_a?(String) || package_name.empty?
+            raise Errors::SchemaError, "`package_name` must be a non-empty String"
+          end
+          unless Schema::SUPPORTED_SYNTAXES.include?(syntax.to_s)
+            raise Errors::SchemaError, "`syntax` must be one of #{Schema::SUPPORTED_SYNTAXES.inspect}, got: #{syntax.inspect}"
+          end
+          if !headers.is_a?(Array) || headers.any? { |header| !header.is_a?(String) }
+            raise Errors::SchemaError, "`headers` must be an Array of Strings"
+          end
+
+          ingestion_state = proto_ingestion_state
+          ingestion_state.package_name = Identifier.validate_package_name(package_name)
+          ingestion_state.syntax = syntax
+          ingestion_state.headers = headers
+          nil
+        end
+
+        # Configures proto field-number mappings directly from a hash.
+        # Useful for tests and advanced use cases where mappings are sourced outside artifacts.
+        # When artifacts are dumped, mappings from the existing `proto_field_numbers.yaml` artifact
+        # are loaded automatically; this method does not need to be called in that case.
+        #
+        # @param proto_field_number_mappings [Hash]
+        # @return [void]
+        def configure_proto_field_number_mappings(proto_field_number_mappings)
+          proto_ingestion_state.field_number_mappings = proto_field_number_mappings
+          nil
+        end
+
+        private
+
+        # Returns this gem's state container. Centralizes the Steep cast that's needed because
+        # Steep can't see the `extend(StateExtension)` applied at runtime in `extended`.
+        def proto_ingestion_state
+          extension_state = state # : ElasticGraph::SchemaDefinition::State & StateExtension
+          extension_state.proto_ingestion_state
+        end
       end
     end
   end