Map DateTime to google.protobuf.Timestamp and document temporal string formats

elasticgraph-proto_ingestion/README.md

@@ -72,7 +72,10 @@ ElasticGraph.define_schema do |schema|
 end
 ```
 
-After running `bundle exec rake schema_artifacts:dump`, ElasticGraph will generate `schema.proto`.
+After running `bundle exec rake schema_artifacts:dump`, ElasticGraph will generate:
+
+- `schema.proto`
+- `proto_field_numbers.yaml`
 
 ## Schema Definition API
 
@@ -93,27 +96,93 @@ ElasticGraph.define_schema do |schema|
 end
 ```
 
+A custom scalar can also map to an externally defined proto type by passing `import:` with the
+proto file that defines it, and `comment:` documents the expected format on each generated field
+(useful when the proto type is wider than the ElasticGraph type):
+
+```ruby
+# in config/schema/phone_number.rb
+
+ElasticGraph.define_schema do |schema|
+  schema.scalar_type "PhoneNumber" do |t|
+    t.mapping type: "keyword"
+    t.json_schema type: "string"
+    t.protobuf type: "string", comment: "E.164 phone number"
+  end
+end
+```
+
+### Stable Field Numbers
+
+`schema_artifacts:dump` automatically reads and writes `proto_field_numbers.yaml`
+in the schema artifacts directory. Existing numbers stay fixed even if field order
+changes, and new fields get the next available numbers. Field numbers follow protobuf's
+rules: they must be between 1 and 536,870,911, and the protobuf-reserved 19000-19999
+range is never allocated and is rejected in mappings.
+
+Alternatives inside generated interface and union `oneof` blocks use the same stable
+message-field mappings, so adding or removing a concrete subtype does not renumber the
+remaining alternatives.
+
+`schema.proto` always uses the public GraphQL field names. When a field uses a
+different `name_in_index`, the sidecar YAML stores that override privately:
+
+```yaml
+messages:
+  Widget:
+    fields:
+      id: 1
+      display_name:
+        field_number: 2
+        name_in_index: displayName
+```
+
+If a field is renamed with `field.renamed_from`, `elasticgraph-proto_ingestion` reuses the
+existing field number under the new public field name.
+
+### Stable Enum Value Numbers
+
+Enum value numbers are pinned the same way, in an `enums` section of the sidecar. Existing
+values keep their numbers when other values are added or removed, new values get the next
+available numbers, and removed values keep their numbers reserved so they are never reused
+(number `0` is always the generated `*_UNSPECIFIED` value):
+
+```yaml
+enums:
+  WidgetColor:
+    values:
+      RED: 1
+      BLUE: 2
+```
+
 ## Type Mappings
 
 The generated `schema.proto` uses these built-in scalar mappings:
 
-| ElasticGraph Type | Protobuf Type |
-|-------------------|------------|
-| `Boolean`         | `bool`     |
-| `Cursor`          | `string`   |
-| `Date`            | `string`   |
-| `DateTime`        | `string`   |
-| `Float`           | `double`   |
-| `ID`              | `string`   |
-| `Int`             | `int32`    |
-| `JsonSafeLong`    | `int64`    |
-| `LocalTime`       | `string`   |
-| `LongString`      | `int64`    |
-| `String`          | `string`   |
-| `TimeZone`        | `string`   |
-| `Untyped`         | `string`   |
+| ElasticGraph Type | Protobuf Type               |
+|-------------------|-----------------------------|
+| `Boolean`         | `bool`                      |
+| `Cursor`          | `string`                    |
+| `Date`            | `string`                    |
+| `DateTime`        | `google.protobuf.Timestamp` |
+| `Float`           | `double`                    |
+| `ID`              | `string`                    |
+| `Int`             | `int32`                     |
+| `JsonSafeLong`    | `int64`                     |
+| `LocalTime`       | `string`                    |
+| `LongString`      | `int64`                     |
+| `String`          | `string`                    |
+| `TimeZone`        | `string`                    |
+| `Untyped`         | `string`                    |
 
 Additionally:
+- `DateTime` uses the [well-known `Timestamp` type](https://protobuf.dev/reference/protobuf/google.protobuf/#timestamp);
+  `schema.proto` imports `google/protobuf/timestamp.proto` automatically. Note that a `Timestamp`
+  is a UTC instant, so a publisher's original UTC offset is not preserved.
+- `string`-typed temporal scalars (`Date`, `LocalTime`, `TimeZone`) are wider than the
+  ElasticGraph types they carry, so generated fields of these types document the expected format
+  in a comment (e.g. `// ISO 8601 date, e.g. "2024-11-25"`). Values are validated when events
+  are ingested, just as with JSON ingestion.
 - List types become `repeated` fields.
 - Lists of lists (e.g. `[[Float!]!]!`) are not supported because Protocol Buffers cannot represent
   them directly. Schema artifact generation raises an error identifying the unsupported field.

elasticgraph-proto_ingestion/lib/elastic_graph/proto_ingestion.rb

@@ -11,5 +11,8 @@ module ElasticGraph
   module ProtoIngestion
     # The name of the generated Protocol Buffers schema file.
     PROTO_SCHEMA_FILE = "schema.proto"
+
+    # The name of the generated proto field-number mapping file.
+    PROTO_FIELD_NUMBERS_FILE = "proto_field_numbers.yaml"
   end
 end

elasticgraph-proto_ingestion/lib/elastic_graph/proto_ingestion/schema_definition/api_extension.rb

@@ -45,6 +45,18 @@ def proto_schema_artifacts(package_name:)
           nil
         end
 
+        # Configures proto field-number mappings directly from a hash.
+        # Useful for tests and advanced use cases where mappings are sourced outside artifacts.
+        # When artifacts are dumped, mappings from the existing `proto_field_numbers.yaml` artifact
+        # are loaded automatically; this method does not need to be called in that case.
+        #
+        # @param proto_field_number_mappings [Hash]
+        # @return [void]
+        def configure_proto_field_number_mappings(proto_field_number_mappings)
+          proto_ingestion_state.field_number_mappings = proto_field_number_mappings
+          nil
+        end
+
         private
 
         # Returns this gem's state container. Centralizes the Steep cast that's needed because

elasticgraph-proto_ingestion/lib/elastic_graph/proto_ingestion/schema_definition/factory_extension.rb

@@ -18,21 +18,21 @@ module ProtoIngestion
     module SchemaDefinition
       # Extension module applied to Factory to add proto support.
       module FactoryExtension
-        # Default protobuf types applied to ElasticGraph's built-in scalar types as they are constructed.
-        BUILT_IN_SCALAR_PROTO_TYPES_BY_NAME = {
-          "Boolean" => "bool",
-          "Cursor" => "string",
-          "Date" => "string",
-          "DateTime" => "string",
-          "Float" => "double",
-          "ID" => "string",
-          "Int" => "int32",
-          "JsonSafeLong" => "int64",
-          "LocalTime" => "string",
-          "LongString" => "int64",
-          "String" => "string",
-          "TimeZone" => "string",
-          "Untyped" => "string"
+        # Default protobuf options applied to ElasticGraph's built-in scalar types as they are constructed.
+        BUILT_IN_SCALAR_PROTO_OPTIONS_BY_NAME = {
+          "Boolean" => {type: "bool"},
+          "Cursor" => {type: "string"},
+          "Date" => {type: "string", comment: %(ISO 8601 date, e.g. "2024-11-25")},
+          "DateTime" => {type: "google.protobuf.Timestamp", import: "google/protobuf/timestamp.proto"},
+          "Float" => {type: "double"},
+          "ID" => {type: "string"},
+          "Int" => {type: "int32"},
+          "JsonSafeLong" => {type: "int64"},
+          "LocalTime" => {type: "string", comment: %(ISO 8601 local time, e.g. "14:23:12")},
+          "LongString" => {type: "int64"},
+          "String" => {type: "string"},
+          "TimeZone" => {type: "string", comment: %(IANA time zone identifier, e.g. "America/Los_Angeles")},
+          "Untyped" => {type: "string"}
         }.freeze
 
         # Creates a new enum type with proto extensions.
@@ -93,8 +93,12 @@ def new_scalar_type(name)
           super(name) do |type|
             extended_type = type.extend(SchemaElements::ScalarTypeExtension) # : ::ElasticGraph::SchemaDefinition::SchemaElements::ScalarType & SchemaElements::ScalarTypeExtension
 
-            if (proto_type = BUILT_IN_SCALAR_PROTO_TYPES_BY_NAME[name.to_s])
-              extended_type.protobuf type: proto_type
+            if (proto_options = BUILT_IN_SCALAR_PROTO_OPTIONS_BY_NAME[name.to_s])
+              extended_type.protobuf(
+                type: proto_options.fetch(:type),
+                import: proto_options[:import],
+                comment: proto_options[:comment]
+              )
             end
 
             yield extended_type if block_given?

elasticgraph-proto_ingestion/lib/elastic_graph/proto_ingestion/schema_definition/proto_ingestion_state.rb

@@ -12,7 +12,7 @@ module SchemaDefinition
       # Holds the proto ingestion extension's schema definition state.
       #
       # @private
-      class ProtoIngestionState < ::Struct.new(:package_name)
+      class ProtoIngestionState < ::Struct.new(:package_name, :field_number_mappings)
       end
     end
   end

elasticgraph-proto_ingestion/lib/elastic_graph/proto_ingestion/schema_definition/results_extension.rb

@@ -20,19 +20,31 @@ def proto_schema
           @proto_schema ||= protobuf_schema_generator.to_proto
         end
 
+        # Returns proto field-number mappings suitable for artifact storage.
+        #
+        # @return [Hash]
+        def proto_field_number_mappings
+          # Ensure generation has occurred before reading mappings from the generator.
+          proto_schema
+          protobuf_schema_generator.field_number_mappings_for_artifact
+        end
+
         private
 
         def protobuf_schema_generator
-          # The cast is needed because Steep can't see the `extend(StateExtension)` applied at
-          # runtime in {APIExtension.extended}.
-          extension_state = state # : ElasticGraph::SchemaDefinition::State & StateExtension
-          ingestion_state = extension_state.proto_ingestion_state
+          @protobuf_schema_generator ||= begin
+            # The cast is needed because Steep can't see the `extend(StateExtension)` applied at
+            # runtime in {APIExtension.extended}.
+            extension_state = state # : ElasticGraph::SchemaDefinition::State & StateExtension
+            ingestion_state = extension_state.proto_ingestion_state
 
-          Schema.new(
-            state: extension_state,
-            all_types: all_types,
-            package_name: ingestion_state.package_name
-          )
+            Schema.new(
+              state: extension_state,
+              all_types: all_types,
+              package_name: ingestion_state.package_name,
+              proto_field_number_mappings: ingestion_state.field_number_mappings
+            )
+          end
         end
       end
     end