From fa85a9ec49f2fea6caf99bc274791bdce2d884cc Mon Sep 17 00:00:00 2001 From: Timo Stripf Date: Mon, 21 Sep 2026 20:36:41 +0000 Subject: [PATCH] Specify the binary encoding, so Protocol Buffers is not required Clause 12 stated that a model "is serialized as a Protocol Buffers message" and left the rest to an editorial note, because Protocol Buffers has no citable specification with a stable identifier. That was one of the blocking gaps: a standard cannot rest its wire format on a document it may not cite. The clause now states the octets itself. An implementer needs this document and nothing else. 12.2 the four primitive encodings, with their wire types: the variable-length integer, the 32-bit and 64-bit forms, and the length-delimited form; signed integers as the varint of their 64-bit two's complement, so every negative value is ten octets 12.3 the field key as the varint of (tag x 8) + wire type; order and repetition; the packed and unpacked forms of a repeated field 12.4 skipping a field the reader does not recognize, by wire type alone -- which is why wire types 3 and 4 are excluded 12.5 limits 12.6 the message definitions, which stay in Annex B 12.7 the relationship to Protocol Buffers Protocol Buffers moves from the normative references, where it could not be cited, to the bibliography. 12.7 says plainly that the encoding above is that of Protocol Buffers restricted to the constructs Annex B uses, that an implementation may therefore be built on a Protocol Buffers library, and that where the two differ this document governs. The restriction is what makes the clause short enough to state: no groups, no zig-zag or fixed-width integer types, seven scalar types in all. Annex B gains a wire type column, so the annex and the clause fit together: the tag and the wire type are what the field key is made of. None of this is asserted from a reading of the format. scripts/check-encoding.rb restates the rules of the clause as a small encoder and compares its output, byte for byte, against the Protocol Buffers runtime: - every scalar type the schema uses, including a negative int32, the largest uint64, a float negative zero and a string beyond ASCII - both field key lengths -- tag 1 and tag 20, since nine ONNX fields have tags above 15 and I would otherwise be stating the two-octet key untested - both forms of a repeated field, and both forms of one field in one message - the literal octets of the clause's own worked examples, so an edit that makes an example wrong fails the build - a nested message, by bootstrap: the script builds a message descriptor by encoding a FileDescriptorProto with its own encoder and hands it to the reference library, which rejects it or produces the wrong fields if the encoding is wrong. That is also where the high tags and the packed field come from, the library's own bundled types having neither. All 28 comparisons agree. `make check-encoding`, run by CI in both workflows. google-protobuf is declared in the Gemfile. It arrives transitively through the Metanorma toolchain, and an undeclared dependency has broken this build twice already. What remains of the serialization gap is the maximum message size, now a minor row rather than a major one: the reference implementation applies 2 GB to a single message and this document neither fixes that nor leaves it explicitly to the implementation. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01DeGJcqqgBdz7kUpH6EsYxD --- .github/workflows/metanorma.yml | 6 + .github/workflows/pages.yml | 6 + CONTRIBUTING.md | 30 ++ Gemfile | 6 + Gemfile.lock | 1 + Makefile | 8 +- README.md | 1 + scripts/check-encoding.rb | 326 ++++++++++++ scripts/generate-schema.rb | 94 +++- .../sections/02-normative-references.adoc | 11 +- sources/part1/sections/12-serialization.adoc | 256 ++++++++- .../sections/annex-b-protobuf-schema.adoc | 486 +++++++++--------- .../part1/sections/annex-c-known-gaps.adoc | 21 +- sources/part1/sections/bibliography.adoc | 3 + 14 files changed, 945 insertions(+), 310 deletions(-) create mode 100755 scripts/check-encoding.rb diff --git a/.github/workflows/metanorma.yml b/.github/workflows/metanorma.yml index 312a178..39dd268 100644 --- a/.github/workflows/metanorma.yml +++ b/.github/workflows/metanorma.yml @@ -47,6 +47,12 @@ jobs: - name: Check generated schema annex run: bundle exec make check-schema + # Clause 12 of Part 1 states the binary encoding itself. This checks it + # against a reference implementation, so the clause cannot drift from + # what implementations write. + - name: Check the binary encoding + run: bundle exec make check-encoding + - name: Check PDF stylesheet run: bundle exec ruby scripts/check-stylesheet.rb diff --git a/.github/workflows/pages.yml b/.github/workflows/pages.yml index f5fcb8a..181dbec 100644 --- a/.github/workflows/pages.yml +++ b/.github/workflows/pages.yml @@ -53,6 +53,12 @@ jobs: - name: Check generated schema annex run: bundle exec make check-schema + # Clause 12 of Part 1 states the binary encoding itself. This checks it + # against a reference implementation, so the clause cannot drift from + # what implementations write. + - name: Check the binary encoding + run: bundle exec make check-encoding + - name: Check PDF stylesheet run: bundle exec ruby scripts/check-stylesheet.rb diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 6d69791..6d9c833 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -164,6 +164,36 @@ on anything it does not recognize inside a message body, rather than skipping it — so a schema change upstream shows up as a failed run rather than as a missing row. +## The binary encoding + +Clause 12 of Part 1 specifies the octets of a model directly: the +variable-length integer, the field key, the four primitive encodings, the two +forms of a repeated field, and how to skip a field the reader does not know. +The point of stating it is that the format can be implemented from this +standard alone. Protocol Buffers is **not** a normative reference — it has no +citable specification with a stable identifier, which is what made this a +blocking gap — and an implementation may still be built on a Protocol Buffers +library, which Clause 12.7 says explicitly. + +That only means something if what the clause says is what implementations +actually write: + +```sh +make check-encoding +``` + +`scripts/check-encoding.rb` restates the rules of the clause as a small +encoder and compares its output, byte for byte, against the Protocol Buffers +runtime — every scalar type the schema uses, both field key lengths, both +forms of a repeated field, and the literal octets of the clause's own worked +examples. The nested-message case is a bootstrap: it builds a message +descriptor by encoding a `FileDescriptorProto` with its own encoder and hands +it to the reference library, which rejects it or produces the wrong fields if +the encoding is wrong. + +CI runs it. Edit Clause 12 and the check will tell you whether the edit is +true. + ## The vendored upstream copy `upstream/onnx/` is a verbatim copy of the ONNX documentation the draft diff --git a/Gemfile b/Gemfile index 93ec1af..6807de7 100644 --- a/Gemfile +++ b/Gemfile @@ -15,6 +15,12 @@ gem "metanorma-generic" # every HTML/DOC render fails with `cannot load such file -- sassc-embedded`. gem "sassc-embedded" +# Used by scripts/check-encoding.rb as the reference implementation the binary +# encoding of Clause 12 is checked against. It arrives transitively through the +# Metanorma toolchain; declared here because the check depends on it directly, +# and an undeclared dependency has already broken this build twice. +gem "google-protobuf" + # Used by scripts/generate-operators.rb to convert the upstream operator prose # from Markdown to AsciiDoc. Hand-rolling that conversion is not worth it: the # prose carries tables, lists, code fences and inline HTML. diff --git a/Gemfile.lock b/Gemfile.lock index 95af7f5..3c19caf 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -1176,6 +1176,7 @@ PLATFORMS x86_64-linux-musl DEPENDENCIES + google-protobuf kramdown-asciidoc metanorma-cli (~> 1.12) metanorma-generic diff --git a/Makefile b/Makefile index 52f6d2b..9416602 100644 --- a/Makefile +++ b/Makefile @@ -29,7 +29,7 @@ RUBY ?= ruby SEVERITY ?= 1 .PHONY: all html doc pdf site lint clean deps check-stylesheet \ - operators check-operators schema check-schema + operators check-operators schema check-schema check-encoding all: html @@ -79,6 +79,12 @@ schema: check-schema: @$(RUBY) scripts/generate-schema.rb --check +# Clause 12 of Part 1 specifies the binary encoding directly, so that the +# format can be implemented from the standard alone. This checks the rules it +# states, byte for byte, against a reference implementation. +check-encoding: + @$(RUBY) scripts/check-encoding.rb + # mn2pdf parses the stylesheet itself, and Metanorma exits 0 when that parse # fails — producing no PDF while the build looks successful. Check it first; # the check needs no fonts, so it also runs where a full PDF render cannot. diff --git a/README.md b/README.md index 3efe459..42dbb47 100644 --- a/README.md +++ b/README.md @@ -55,6 +55,7 @@ scripts/check-errors.rb Gates the build on Metanorma diagnostic severity scripts/vendor-onnx-docs.sh Refreshes the vendored upstream copy scripts/generate-operators.rb Generates the operator clauses of Part 2 scripts/generate-schema.rb Generates Annex B of Part 1 from the vendored .proto +scripts/check-encoding.rb Checks Clause 12's binary encoding against a reference upstream/onnx/ Verbatim upstream ONNX docs (complete) + .proto .github/workflows/ CI: build HTML + PDF, publish as artifacts ``` diff --git a/scripts/check-encoding.rb b/scripts/check-encoding.rb new file mode 100755 index 0000000..0ae08bb --- /dev/null +++ b/scripts/check-encoding.rb @@ -0,0 +1,326 @@ +#!/usr/bin/env ruby +# frozen_string_literal: true + +# Check the binary encoding of Clause 12 of Part 1 against a reference +# implementation. +# +# scripts/check-encoding.rb +# +# Clause 12 specifies the octets of an ONNX model directly, so that the format +# can be implemented from this standard alone. That is only worth anything if +# what the clause says is what implementations actually write, and the clause +# was written from a reading of the format rather than from a specification +# this standard is allowed to cite. So the rules are restated here as code and +# cross-checked, byte for byte, against the Protocol Buffers runtime. +# +# The encoder below is deliberately naive and self-contained: it is the clause, +# transcribed. If it and the reference disagree, one of the two is wrong and +# the clause is not to be trusted until that is settled. +# +# Coverage: every scalar type the schema of Annex B uses, both field key +# lengths, both repeated forms, and a nested message. The nested case is the +# bootstrap at the end -- a message descriptor is built by encoding a +# FileDescriptorProto with this encoder and handing it to the reference +# library, which will reject it or produce the wrong fields if the encoding is +# wrong. That also supplies the tags above 15 and the packed repeated field +# that the reference library's own bundled types do not have. + +require "google/protobuf" +require "google/protobuf/wrappers_pb" +require "google/protobuf/timestamp_pb" + +module Wire + VARINT = 0 + BIT64 = 1 + DELIM = 2 + BIT32 = 5 + + module_function + + # Clause 12.2.2: base-128, least significant group first, high bit set on + # every octet but the last. + def varint(value) + value.negative? and raise ArgumentError, "varint of a negative value" + + out = +"" + loop do + group = value & 0x7F + value >>= 7 + return (out << group.chr).b if value.zero? + + out << (group | 0x80).chr + end + end + + # Clause 12.2.3: a signed integer is the varint of its 64-bit two's + # complement, so every negative value occupies ten octets. + def signed(value) = varint(value.negative? ? value + (1 << 64) : value) + + # Clause 12.3.1: the key is the varint of (tag << 3) | wire type. + def key(tag, wire_type) = varint((tag << 3) | wire_type) + + # Clause 12.2.5. + def delimited(octets) = varint(octets.bytesize) + octets.b + + # Clause 12.2.4. + def bit32(value) = [value].pack("e") + def bit64(value) = [value].pack("E") + + def read_varint(octets, at) + value = 0 + shift = 0 + loop do + byte = octets.getbyte(at) or raise "truncated varint" + at += 1 + value |= (byte & 0x7F) << shift + return [value, at] if (byte & 0x80).zero? + + shift += 7 + shift >= 70 and raise "varint longer than ten octets" + end + end + + # Clause 12.4: read a message as a sequence of fields without knowing any of + # them, which is what skipping an unknown field requires. + def scan(octets) + octets = octets.b + at = 0 + fields = [] + while at < octets.bytesize + k, at = read_varint(octets, at) + tag = k >> 3 + wire_type = k & 7 + case wire_type + when VARINT + value, at = read_varint(octets, at) + fields << [tag, wire_type, value] + when BIT64 + fields << [tag, wire_type, octets.byteslice(at, 8)] + at += 8 + when BIT32 + fields << [tag, wire_type, octets.byteslice(at, 4)] + at += 4 + when DELIM + length, at = read_varint(octets, at) + fields << [tag, wire_type, octets.byteslice(at, length)] + at += length + else + raise "wire type #{wire_type} is not admitted" + end + end + fields + end +end + +# ---------------------------------------------------------------- comparison + +FAILURES = [] + +def compare(name, mine, reference) + if mine.b == reference.b + puts format(" ok %-34s %s", name, mine.unpack1("H*")) + else + FAILURES << name + puts format(" FAIL %-34s %s", name, mine.unpack1("H*")) + puts format(" %-34s %s (reference)", "", reference.unpack1("H*")) + end +end + +W = Google::Protobuf + +puts "Clause 12 against google-protobuf " \ + "#{Gem.loaded_specs['google-protobuf']&.version}" +puts +puts "Scalar types and field keys" + +[ + ["int32 zero (absent)", W::Int32Value.new(value: 0), -> { "" }], + ["int32 one", W::Int32Value.new(value: 1), + -> { Wire.key(1, Wire::VARINT) + Wire.signed(1) }], + ["int32 300 (two groups)", W::Int32Value.new(value: 300), + -> { Wire.key(1, Wire::VARINT) + Wire.signed(300) }], + ["int32 minus one", W::Int32Value.new(value: -1), + -> { Wire.key(1, Wire::VARINT) + Wire.signed(-1) }], + ["int64 minus two", W::Int64Value.new(value: -2), + -> { Wire.key(1, Wire::VARINT) + Wire.signed(-2) }], + ["int64 largest", W::Int64Value.new(value: (1 << 63) - 1), + -> { Wire.key(1, Wire::VARINT) + Wire.signed((1 << 63) - 1) }], + ["uint64 largest", W::UInt64Value.new(value: (1 << 64) - 1), + -> { Wire.key(1, Wire::VARINT) + Wire.varint((1 << 64) - 1) }], + ["bool true", W::BoolValue.new(value: true), + -> { Wire.key(1, Wire::VARINT) + Wire.varint(1) }], + ["float 1.5", W::FloatValue.new(value: 1.5), + -> { Wire.key(1, Wire::BIT32) + Wire.bit32(1.5) }], + ["float negative zero", W::FloatValue.new(value: -0.0), + -> { Wire.key(1, Wire::BIT32) + Wire.bit32(-0.0) }], + ["double 1.5", W::DoubleValue.new(value: 1.5), + -> { Wire.key(1, Wire::BIT64) + Wire.bit64(1.5) }], + ["string ASCII", W::StringValue.new(value: "abc"), + -> { Wire.key(1, Wire::DELIM) + Wire.delimited("abc") }], + ["string beyond ASCII", W::StringValue.new(value: "aä€"), + -> { Wire.key(1, Wire::DELIM) + Wire.delimited("aä€") }], + ["bytes with NUL and FF", W::BytesValue.new(value: "\x00\xFF".b), + -> { Wire.key(1, Wire::DELIM) + Wire.delimited("\x00\xFF".b) }], + ["two fields in order", W::Timestamp.new(seconds: -1, nanos: 7), + -> { + Wire.key(1, Wire::VARINT) + Wire.signed(-1) + + Wire.key(2, Wire::VARINT) + Wire.signed(7) + }], +].each { |name, message, mine| compare(name, mine.call, message.to_proto) } + +# ------------------------------------------------------- descriptor bootstrap + +# descriptor.proto, only the fields needed here. +FIELD_NAME = 1 +FIELD_NUMBER = 3 +FIELD_LABEL = 4 +FIELD_TYPE = 5 +FIELD_OPTIONS = 8 +FIELD_JSON_NAME = 10 +OPTION_PACKED = 2 +LABEL_OPTIONAL = 1 +LABEL_REPEATED = 3 +TYPE_INT64 = 3 +TYPE_INT32 = 5 +TYPE_STRING = 9 + +def field_descriptor(name, number, type, label: LABEL_OPTIONAL, packed: nil) + out = Wire.key(FIELD_NAME, Wire::DELIM) + Wire.delimited(name) + out += Wire.key(FIELD_NUMBER, Wire::VARINT) + Wire.signed(number) + out += Wire.key(FIELD_LABEL, Wire::VARINT) + Wire.varint(label) + out += Wire.key(FIELD_TYPE, Wire::VARINT) + Wire.varint(type) + out += Wire.key(FIELD_JSON_NAME, Wire::DELIM) + Wire.delimited(name) + unless packed.nil? + options = Wire.key(OPTION_PACKED, Wire::VARINT) + Wire.varint(packed ? 1 : 0) + out += Wire.key(FIELD_OPTIONS, Wire::DELIM) + Wire.delimited(options) + end + out +end + +FIELDS = [ + ["low", 1, TYPE_INT32, LABEL_OPTIONAL, nil], + ["high", 20, TYPE_INT32, LABEL_OPTIONAL, nil], + ["higher", 26, TYPE_STRING, LABEL_OPTIONAL, nil], + ["packed", 5, TYPE_INT64, LABEL_REPEATED, true], + ["plain", 6, TYPE_INT64, LABEL_REPEATED, false], +].freeze + +message = Wire.key(1, Wire::DELIM) + Wire.delimited("Probe") +FIELDS.each do |name, number, type, label, packed| + message += Wire.key(2, Wire::DELIM) + + Wire.delimited(field_descriptor(name, number, type, + label: label, packed: packed)) +end + +file = Wire.key(1, Wire::DELIM) + Wire.delimited("probe.proto") +file += Wire.key(2, Wire::DELIM) + Wire.delimited("probe") +file += Wire.key(4, Wire::DELIM) + Wire.delimited(message) +file += Wire.key(12, Wire::DELIM) + Wire.delimited("proto2") + +pool = Google::Protobuf::DescriptorPool.new +begin + pool.add_serialized_file(file) +rescue StandardError => e + warn "\nthe reference library rejected a descriptor this encoder produced:" + warn " #{e.message}" + exit 1 +end + +descriptor = pool.lookup("probe.Probe") +puts +puts "Nested messages, by building a descriptor with this encoder" +FIELDS.each do |name, number, _type, label, _packed| + field = descriptor.lookup(name) + wanted = [number, label == LABEL_REPEATED ? :repeated : :optional] + got = [field.number, field.label] + if got == wanted + puts format(" ok %-34s tag %d, %s", name, field.number, field.label) + else + FAILURES << "descriptor #{name}" + puts format(" FAIL %-34s got %p, wanted %p", name, got, wanted) + end +end + +probe = descriptor.msgclass +puts +puts "Field keys above tag 15, and repeated fields" +[ + ["tag 1, one-octet key", probe.new(low: 7), + -> { Wire.key(1, Wire::VARINT) + Wire.signed(7) }], + ["tag 20, two-octet key", probe.new(high: 7), + -> { Wire.key(20, Wire::VARINT) + Wire.signed(7) }], + ["tag 26, length-delimited", probe.new(higher: "x"), + -> { Wire.key(26, Wire::DELIM) + Wire.delimited("x") }], + ["repeated, packed", probe.new(packed: [1, 300, -1]), + -> { + Wire.key(5, Wire::DELIM) + + Wire.delimited(Wire.signed(1) + Wire.signed(300) + Wire.signed(-1)) + }], + ["repeated, not packed", probe.new(plain: [1, 300]), + -> { + Wire.key(6, Wire::VARINT) + Wire.signed(1) + + Wire.key(6, Wire::VARINT) + Wire.signed(300) + }], +].each { |name, message_value, mine| compare(name, mine.call, message_value.to_proto) } + +# ------------------------------------------------------- examples and mixing + +# The worked examples of Clause 12 give literal octets. They are checked here +# so that an edit to the clause that makes one of them wrong fails the build. +puts +puts "The worked examples of the clause" +[ + ["12.2.2 1", Wire.varint(1), "01"], + ["12.2.2 300", Wire.varint(300), "ac02"], + ["12.2.3 -1", Wire.signed(-1), "ffffffffffffffffff01"], + ["12.3.1 tag 1, type 0", Wire.key(1, Wire::VARINT), "08"], + ["12.3.1 tag 20, type 0", Wire.key(20, Wire::VARINT), "a001"], + ["12.3.3 packed", + Wire.key(5, Wire::DELIM) + Wire.delimited(Wire.signed(1) + Wire.signed(300)), + "2a0301ac02"], + ["12.3.3 unpacked", + Wire.key(5, Wire::VARINT) + Wire.signed(1) + + Wire.key(5, Wire::VARINT) + Wire.signed(300), + "2801 28ac02".delete(" ")], +].each do |name, mine, wanted| + compare(name, mine, [wanted].pack("H*")) +end + +# Clause 12.3.3 requires a consumer to accept both forms of one repeated field +# in one message, concatenating in order of appearance. Checked against the +# reference rather than asserted. +puts +puts "Both forms of one repeated field in one message" +mixed = Wire.key(5, Wire::DELIM) + Wire.delimited(Wire.signed(1) + Wire.signed(2)) + + Wire.key(5, Wire::VARINT) + Wire.signed(3) +got = probe.decode(mixed).packed.to_a +if got == [1, 2, 3] + puts format(" ok %-34s %p", "packed then unpacked", got) +else + FAILURES << "mixed repeated forms" + puts format(" FAIL %-34s got %p, wanted [1, 2, 3]", "packed then unpacked", got) +end + +# ------------------------------------------------------------------- scanning + +puts +puts "Reading a message without knowing its fields" +sample = probe.new(low: 1, high: 2, higher: "x", packed: [7], plain: [8]).to_proto +seen = Wire.scan(sample).map { |tag, wire_type, _| [tag, wire_type] } +wanted = [[1, Wire::VARINT], [5, Wire::DELIM], [6, Wire::VARINT], + [20, Wire::VARINT], [26, Wire::DELIM]] +if seen.sort == wanted.sort + puts format(" ok %-34s %p", "every field located", seen) +else + FAILURES << "scan" + puts format(" FAIL %-34s got %p, wanted %p", "scan", seen, wanted) +end + +puts +if FAILURES.empty? + puts "encoding: the clause and the reference implementation agree" +else + warn "encoding: #{FAILURES.length} disagreement(s): #{FAILURES.join(', ')}" + warn " Clause 12 of Part 1 is wrong, or this transcription of it is." + exit 1 +end diff --git a/scripts/generate-schema.rb b/scripts/generate-schema.rb index 3a2c72e..1886be4 100755 --- a/scripts/generate-schema.rb +++ b/scripts/generate-schema.rb @@ -23,6 +23,7 @@ # informative aid Annex B points at. require "optparse" +require "set" ROOT = File.expand_path("..", __dir__) @@ -42,7 +43,8 @@ MANDATORY = /MUST be present (?:in|for) this version of the IR/i.freeze -Field = Struct.new(:label, :type, :name, :tag, :mandatory, :oneof, :deprecated) +Field = Struct.new(:label, :type, :name, :tag, :mandatory, :oneof, :deprecated, + :packed) Enum = Struct.new(:name, :values) Msg = Struct.new(:name, :fields, :enums, :reserved) @@ -127,12 +129,14 @@ def parse(path) type = Regexp.last_match(2).strip name = Regexp.last_match(3) tag = Regexp.last_match(4).to_i + options = Regexp.last_match(5).to_s text = comment.join(" ") owner = current.call or raise "#{path}:#{lineno}: field outside a message" owner.fields << Field.new( label, type, name, tag, text.match?(MANDATORY), oneof, - text.match?(/\bdeprecated\b/i) + text.match?(/\bdeprecated\b/i), + options.include?("packed") ) comment.clear else @@ -146,9 +150,26 @@ def parse(path) # --------------------------------------------------------------------- emit +# Clause 12.2.1. A field's wire type follows from its type, and from whether a +# repeated field of a numeric type is written packed. +SCALAR_WIRE_TYPES = { + "int32" => 0, "int64" => 0, "uint32" => 0, "uint64" => 0, "bool" => 0, + "double" => 1, + "string" => 2, "bytes" => 2, + "float" => 5, +}.freeze + +def wire_type(field, enums) + return 2 if field.packed # a packed repeated field is length-delimited + + SCALAR_WIRE_TYPES[field.type] || + (enums.include?(field.type) ? 0 : 2) # an enumeration is a varint; anything + # else names a message +end + def obligation(field) return "deprecated" if field.deprecated - return "repeated" if field.label == "repeated" + return field.packed ? "repeated, packed" : "repeated" if field.label == "repeated" return "one of the group" if field.label == "oneof" field.mandatory ? "mandatory" : "optional" @@ -162,7 +183,7 @@ def anchor(name) "schema-#{name.downcase.tr('.', '-')}" end -def render_message(msg) +def render_message(msg, enums) out = [] out << "[[#{anchor(msg.name)}]]" out << "==== #{msg.name}" @@ -171,12 +192,13 @@ def render_message(msg) unless msg.fields.empty? out << "[[tbl-#{anchor(msg.name)}]]" out << ".Fields of `#{msg.name}`" - out << '[cols="3,1,3,2"]' + out << '[cols="3,1,1,3,2"]' out << "|===" - out << "| Field | Tag | Type | Obligation" + out << "| Field | Tag | Wire type | Type | Obligation" out << "" msg.fields.each do |f| - out << "| `#{f.name}` | #{f.tag} | #{adoc_type(f.type)} | #{obligation(f)}" + out << "| `#{f.name}` | #{f.tag} | #{wire_type(f, enums)} | " \ + "#{adoc_type(f.type)} | #{obligation(f)}" end out << "|===" out << "" @@ -212,6 +234,12 @@ def render_message(msg) end def render(groups) + # Every enumeration defined anywhere, so that a field naming one is known to + # be a varint rather than an embedded message. + enum_names = groups.flat_map do |g| + g[:messages].flat_map { |m| m.enums.map { |e| [e.name, e.name.split(".").last] } } + end.flatten.to_set + counts = groups.sum { |g| g[:messages].length } fields = groups.sum { |g| g[:messages].sum { |m| m.fields.length } } mandatory = groups.sum do |g| @@ -226,28 +254,37 @@ def render(groups) out << "" out << "[[annex-schema]]" out << "[appendix,obligation=normative]" - out << "== Protocol Buffers schema" + out << "== Message definitions" out << "" out << "=== General" out << "" out << "This annex states the message definitions of the serialized form: " \ - "#{counts} messages and #{fields} fields." + "#{counts} messages and #{fields} fields. <> states " \ + "how the octets of a message are laid out; this annex says which " \ + "fields there are." + out << "" + out << "The tag of a field identifies it within its message and SHALL NOT " \ + "be reused. The wire type is the encoding of its value, per " \ + "<>; together the two make the field key " \ + "(<>)." out << "" - out << "The tag of a field is normative; it identifies the field on the " \ - "wire and SHALL NOT be reused. A field marked mandatory SHALL be " \ - "present. A field marked optional MAY be absent, and a " \ - "<> SHALL accept a message in which it is. A field marked " \ - "repeated holds zero or more values. A field marked deprecated " \ - "SHALL NOT be written by a <>, and a consumer that reads " \ - "one SHALL ignore it." + out << "A field marked mandatory SHALL be present. A field marked optional " \ + "MAY be absent, and a <> SHALL accept a message " \ + "in which it is. A field marked repeated holds zero or more values, " \ + "and a <> SHOULD write it in the form recorded " \ + "here, packed or not; a consumer SHALL accept either " \ + "(<>). A field marked deprecated SHALL NOT be " \ + "written by a producer, and a consumer that reads one SHALL ignore " \ + "it." out << "" out << "The obligations are those in force at the IR version stated in " \ "<>." out << "" - out << "NOTE: #{mandatory} of the #{fields} fields are mandatory. Protocol " \ - "Buffers cannot express the distinction: in the syntax this schema " \ - "uses every field is optional, and the obligation is carried in the " \ - "comments. Stating it in a table is the reason this annex exists." + out << "NOTE: #{mandatory} of the #{fields} fields are mandatory. A schema " \ + "cannot express the distinction: in the syntax this standard was " \ + "derived from every field is optional, and the obligation is " \ + "carried in the comments. Stating it in a table is the reason this " \ + "annex exists." out << "" out << "[[tbl-schema-sources]]" out << ".Subclauses of this annex and the schema file each restates" @@ -264,16 +301,19 @@ def render(groups) groups.each do |g| out << "=== #{g[:title]}" out << "" - g[:messages].each { |m| out << render_message(m) } + g[:messages].each { |m| out << render_message(m, enum_names) } end - out << "=== Schema source" + out << "=== Derivation" out << "" - out << "The schema source is vendored in the repository at " \ - "`upstream/onnx/proto/`, at the release recorded in " \ - "`upstream/onnx/SOURCE.txt`. It is an informative aid: where a " \ - "comment in it carries a normative statement, that statement is in " \ - "the clause it belongs to and not in this annex." + out << "The tables above are derived from the Protocol Buffers schema of " \ + "the ONNX reference implementation, vendored in the repository at " \ + "`upstream/onnx/proto/` at the release recorded in " \ + "`upstream/onnx/SOURCE.txt`. That schema is an informative aid, not " \ + "a normative reference: the tables above are normative, and where a " \ + "comment in the schema carries a normative statement, that " \ + "statement is in the clause it belongs to. See " \ + "<>." out << "" out.join("\n").gsub(/\n{3,}/, "\n\n") diff --git a/sources/part1/sections/02-normative-references.adoc b/sources/part1/sections/02-normative-references.adoc index 0682224..5486457 100644 --- a/sources/part1/sections/02-normative-references.adoc +++ b/sources/part1/sections/02-normative-references.adoc @@ -26,9 +26,8 @@ edition of the referenced document (including any amendments) applies. * [[[rfc2119,IETF RFC 2119]]], _Key words for use in RFCs to Indicate Requirement Levels_ -NOTE: Editorial. Protocol Buffers has no citable normative reference with a -stable identifier. Either (a) normatively include the required subset of the -wire format in an annex, or (b) cite a dated release. This is to be resolved -before the committee draft; see also the editorial note in <>. -A bibliography clause admits no editorial note block, so this appears as a -note. +NOTE: Protocol Buffers is deliberately not a normative reference. It has no +citable specification with a stable identifier, and this document therefore +states the binary encoding itself, in <>, rather than citing +one. An implementation may still be built on a Protocol Buffers library; see +<>. diff --git a/sources/part1/sections/12-serialization.adoc b/sources/part1/sections/12-serialization.adoc index 1194abe..e50d6fd 100644 --- a/sources/part1/sections/12-serialization.adoc +++ b/sources/part1/sections/12-serialization.adoc @@ -3,40 +3,248 @@ === General -An ONNX <> is serialized as a Protocol Buffers message. +An ONNX <> is serialized as a sequence of octets. This clause +specifies that sequence: the encoding of each kind of value, the framing of the +fields of a message, and the treatment of a field a <> does +not recognize. <> states the messages and their fields. -[IMPORTANT] -.Editorial note +The encoding is specified here in full. A <> or consumer can +be implemented from this document alone, and this document does not require any +other specification in order to be implemented. + +NOTE: The encoding defined here is the one a Protocol Buffers implementation +produces for the schema from which <> is derived. Using such an +implementation is therefore a convenient way to satisfy this clause, and the +way most existing tools do satisfy it, but it is an implementation choice and +not a requirement. See <>. + +[[primitive-encodings]] +=== Primitive encodings + +==== General + +Four encodings carry every value. <> names them and gives the +wire type of each, which is the number that identifies the encoding in a field +key (<>). + +[[tbl-wire-types]] +.Primitive encodings and their wire types +[cols="1,2,4"] +|=== +| Wire type | Encoding | Used for + +| 0 | variable-length integer | `int32`, `int64`, `uint64`, `bool`, enumerations +| 1 | 64-bit | `double` +| 2 | length-delimited | `string`, `bytes`, embedded messages, packed repeated fields +| 5 | 32-bit | `float` +|=== + +The wire types 3 and 4 SHALL NOT appear. A consumer that encounters one SHALL +reject the model. + +NOTE: Wire types 3 and 4 delimit a construct the format no longer uses. They +are excluded so that a consumer can skip an unrecognized field by its wire type +alone, which <> requires. + +[[varint]] +==== Variable-length integer + +A variable-length integer encodes a non-negative integer as a sequence of +octets. The integer is divided into groups of seven bits, least significant +group first. Each group occupies the seven low-order bits of one octet. The +high-order bit of an octet is 1 where a further octet follows and 0 in the last +octet. + +The value encoded is at most stem:[2^64 - 1], so the sequence is at most ten +octets. A consumer SHALL reject a sequence of more than ten octets. + +.Variable-length integers +[example] +==== +1 is `01`. 300 is `ac 02`: the groups are stem:[300 = 44 + 2 * 128], the first +octet carries 44 with the continuation bit set, the second carries 2. +==== + +[[signed-integers]] +==== Signed integers + +A value of a signed integer type is encoded as the variable-length integer of +its representation as a 64-bit two's complement integer. + +A negative value therefore occupies ten octets, whatever its magnitude and +whether its type is `int32` or `int64`. + +.A negative integer +[example] +==== +-1 is `ff ff ff ff ff ff ff ff ff 01`, being the encoding of +stem:[2^64 - 1]. +==== + +A consumer reading a field of type `int32` SHALL reject a value outside the +range of a 32-bit two's complement integer. + +==== Fixed-width values + +A 32-bit value is four octets, least significant octet first. A value of type +`float` is the binary32 representation of <> in that order. + +A 64-bit value is eight octets, least significant octet first. A value of type +`double` is the binary64 representation of <> in that order. + +The octets are those of the representation, so a signalling NaN, a quiet NaN +and a negative zero are each preserved exactly. + +[[length-delimited]] +==== Length-delimited values + +A length-delimited value is a variable-length integer giving a length in +octets, followed by that many octets. + +The octets are: + +* for `string`, the UTF-8 encoding of the character sequence, as specified in + <>, without a byte order mark; +* for `bytes`, the octets themselves; +* for an embedded message, the encoding of that message under this clause; +* for a packed repeated field, the encodings of its elements, in order, each + without a field key (<>). + +A consumer SHALL reject a length that extends beyond the end of the enclosing +message. + +[[field-framing]] +=== Fields + +[[field-key]] +==== Field key + +Each field is preceded by a field key, which is the variable-length integer of + +[stem] +++++ +(t xx 8) + w +++++ + +where stem:[t] is the tag of the field, as given in <>, and +stem:[w] is its wire type, from <>. + +A tag is at least 1. A field whose tag is at most 15 therefore has a key of one +octet, and one whose tag is at least 16 has a key of two or more. + +.Field keys +[example] ==== -See the note in Clause 2. The normative status of the Protocol Buffers encoding -must be settled: this document either cites a dated specification or reproduces -the required subset of the wire format. +The field of tag 1 and wire type 0 has the key `08`. The field of tag 20 and +wire type 0 has the key `a0 01`. ==== -=== Encoding +==== Order and repetition of fields + +The fields of a message MAY appear in any order. A producer SHOULD write them +in order of increasing tag. + +Where a field that is not repeated appears more than once, the value is that of +the last occurrence, save that occurrences of an embedded message field are +merged, field by field, under this same rule. + +[[repeated-fields]] +==== Repeated fields + +A repeated field is encoded in one of two forms. + +In the packed form, the field appears once, with wire type 2, and its +length-delimited value holds the encodings of the elements in order, each +without a field key. Only a repeated field whose elements are of a type with +wire type 0, 1 or 5 may take this form. + +In the unpacked form, the field key and value appear once per element, in +order. + +A producer SHOULD write a repeated field in the form <> records +for it. + +A consumer SHALL accept both forms for any repeated field, and SHALL accept a +field that appears in both forms in one message, concatenating the elements in +order of appearance. + +.The two forms +[example] +==== +A repeated `int64` field of tag 5 holding 1 and 300 is `2a 03 01 ac 02` packed, +and `28 01 28 ac 02` unpacked. +==== + +[[unknown-fields]] +=== Unrecognized fields + +A consumer that encounters a field whose tag it does not recognize SHALL skip +it and SHALL continue reading the message. The length to skip is determined by +the wire type in the field key alone: + +* wire type 0: read the variable-length integer and discard it; +* wire type 1: skip eight octets; +* wire type 5: skip four octets; +* wire type 2: read the length and skip that many octets. + +A consumer SHALL NOT reject a model for carrying a field it does not +recognize. + +NOTE: This is what allows a model written against a later IR version to be read +by a consumer written against an earlier one, so far as the fields they have in +common allow. <> states when that is enough for the model to be +evaluated. + +A consumer MAY retain the octets of a field it does not recognize. A consumer +that rewrites a model and does not retain them SHALL NOT represent the result +as the same model. + +=== Limits + +A consumer MAY impose a maximum size on a message and on the model as a whole. +A consumer that does so SHALL document the limit and SHALL report a model +exceeding it under <>; see <>. [IMPORTANT] .Editorial note ==== -State which subset of the Protocol Buffers wire format a conforming encoder may -emit and a conforming decoder must accept, including the treatment of unknown -fields, repeated field ordering and the maximum message size. Protocol Buffers -permits encodings that differ octet for octet while carrying the same content, -which is why <> is a separate question. +No maximum is stated here. The reference implementation applies one of 2 +gigabytes to a single message, which is also the point beyond which a length +does not fit the field it is read into in some implementations, and the +external data mechanism of <> exists so that a model can exceed +it. State whether this document fixes that maximum or leaves it +implementation-defined; if it is fixed, <> has to say which +messages it applies to. ==== === Message definitions -The message definitions are given normatively in <>, as a table -of fields per message giving wire tag, type and obligation. +The messages, their fields, and for each field its tag, wire type and +obligation are given normatively in <>. -A field's tag identifies it on the wire. A tag SHALL NOT be reused for a +A tag identifies a field within its message. A tag SHALL NOT be reused for a different field, and a tag recorded as reserved SHALL NOT be used at all. -NOTE: The obligation is what the schema source cannot express, and is the -reason the annex is a table rather than the source. In the Protocol Buffers -syntax this schema uses, every field is syntactically optional; which ones a -producer must supply is carried in the comments. +NOTE: The obligation is what a schema cannot express, and is the reason +<> is a table rather than a schema. In the schema this standard +was derived from, every field is syntactically optional. + +[[protobuf-relationship]] +=== Relationship to Protocol Buffers + +The encoding of <> to <> is that of +Protocol Buffers, restricted to the constructs <> uses. An +implementation MAY therefore be built on a Protocol Buffers library, by +compiling a schema whose messages, fields, tags and types are those of +<>. + +This document does not require that, and does not depend on any specification +of Protocol Buffers. Where the two differ, this document governs a model +claimed to conform to it. + +NOTE: The restriction is substantial and is what makes the encoding short +enough to state here. Groups are excluded; no field uses a zig-zag encoded or +fixed-width integer type; and the schema uses seven scalar types in all. [[canonical-form]] === Canonical form @@ -44,8 +252,10 @@ producer must supply is carried in the comments. [IMPORTANT] .Editorial note ==== -Decide whether this document defines a canonical serialization. A canonical -form is a prerequisite for signing and for reproducible model hashes, both of -which have been requested by adopters, but Protocol Buffers does not guarantee -a canonical encoding on its own. +Decide whether this document defines a canonical serialization. This clause +admits encodings that differ octet for octet while carrying the same content: +fields may appear in any order, a repeated field may be packed or not, and a +signed value may be written in fewer octets than ten only where it is +non-negative. A canonical form is a prerequisite for signing and for +reproducible model hashes, both of which have been requested by adopters. ==== diff --git a/sources/part1/sections/annex-b-protobuf-schema.adoc b/sources/part1/sections/annex-b-protobuf-schema.adoc index f9b1db6..6fce6be 100644 --- a/sources/part1/sections/annex-b-protobuf-schema.adoc +++ b/sources/part1/sections/annex-b-protobuf-schema.adoc @@ -5,17 +5,19 @@ [[annex-schema]] [appendix,obligation=normative] -== Protocol Buffers schema +== Message definitions === General -This annex states the message definitions of the serialized form: 35 messages and 166 fields. +This annex states the message definitions of the serialized form: 35 messages and 166 fields. <> states how the octets of a message are laid out; this annex says which fields there are. -The tag of a field is normative; it identifies the field on the wire and SHALL NOT be reused. A field marked mandatory SHALL be present. A field marked optional MAY be absent, and a <> SHALL accept a message in which it is. A field marked repeated holds zero or more values. A field marked deprecated SHALL NOT be written by a <>, and a consumer that reads one SHALL ignore it. +The tag of a field identifies it within its message and SHALL NOT be reused. The wire type is the encoding of its value, per <>; together the two make the field key (<>). + +A field marked mandatory SHALL be present. A field marked optional MAY be absent, and a <> SHALL accept a message in which it is. A field marked repeated holds zero or more values, and a <> SHOULD write it in the form recorded here, packed or not; a consumer SHALL accept either (<>). A field marked deprecated SHALL NOT be written by a producer, and a consumer that reads one SHALL ignore it. The obligations are those in force at the IR version stated in <>. -NOTE: 23 of the 166 fields are mandatory. Protocol Buffers cannot express the distinction: in the syntax this schema uses every field is optional, and the obligation is carried in the comments. Stating it in a table is the reason this annex exists. +NOTE: 23 of the 166 fields are mandatory. A schema cannot express the distinction: in the syntax this standard was derived from every field is optional, and the obligation is carried in the comments. Stating it in a table is the reason this annex exists. [[tbl-schema-sources]] .Subclauses of this annex and the schema file each restates @@ -61,28 +63,28 @@ NOTE: 23 of the 166 fields are mandatory. Protocol Buffers cannot express the di [[tbl-schema-attributeproto]] .Fields of `AttributeProto` -[cols="3,1,3,2"] -|=== -| Field | Tag | Type | Obligation - -| `name` | 1 | `string` | mandatory -| `ref_attr_name` | 21 | `string` | optional -| `doc_string` | 13 | `string` | optional -| `type` | 20 | `AttributeType` | mandatory -| `f` | 2 | `float` | mandatory -| `i` | 3 | `int64` | optional -| `s` | 4 | `bytes` | optional -| `t` | 5 | `TensorProto` | optional -| `g` | 6 | `GraphProto` | optional -| `sparse_tensor` | 22 | `SparseTensorProto` | optional -| `tp` | 14 | `TypeProto` | deprecated -| `floats` | 7 | `float` | repeated -| `ints` | 8 | `int64` | repeated -| `strings` | 9 | `bytes` | repeated -| `tensors` | 10 | `TensorProto` | repeated -| `graphs` | 11 | `GraphProto` | repeated -| `sparse_tensors` | 23 | `SparseTensorProto` | repeated -| `type_protos` | 15 | `TypeProto` | repeated +[cols="3,1,1,3,2"] +|=== +| Field | Tag | Wire type | Type | Obligation + +| `name` | 1 | 2 | `string` | mandatory +| `ref_attr_name` | 21 | 2 | `string` | optional +| `doc_string` | 13 | 2 | `string` | optional +| `type` | 20 | 0 | `AttributeType` | mandatory +| `f` | 2 | 5 | `float` | mandatory +| `i` | 3 | 0 | `int64` | optional +| `s` | 4 | 2 | `bytes` | optional +| `t` | 5 | 2 | `TensorProto` | optional +| `g` | 6 | 2 | `GraphProto` | optional +| `sparse_tensor` | 22 | 2 | `SparseTensorProto` | optional +| `tp` | 14 | 2 | `TypeProto` | deprecated +| `floats` | 7 | 5 | `float` | repeated +| `ints` | 8 | 0 | `int64` | repeated +| `strings` | 9 | 2 | `bytes` | repeated +| `tensors` | 10 | 2 | `TensorProto` | repeated +| `graphs` | 11 | 2 | `GraphProto` | repeated +| `sparse_tensors` | 23 | 2 | `SparseTensorProto` | repeated +| `type_protos` | 15 | 2 | `TypeProto` | repeated |=== Tags and names reserved in `AttributeProto`, which SHALL NOT be used: `12, 16 to 19`; `"v"`. @@ -115,14 +117,14 @@ Tags and names reserved in `AttributeProto`, which SHALL NOT be used: `12, 16 to [[tbl-schema-valueinfoproto]] .Fields of `ValueInfoProto` -[cols="3,1,3,2"] +[cols="3,1,1,3,2"] |=== -| Field | Tag | Type | Obligation +| Field | Tag | Wire type | Type | Obligation -| `name` | 1 | `string` | mandatory -| `type` | 2 | `TypeProto` | mandatory -| `doc_string` | 3 | `string` | optional -| `metadata_props` | 4 | `StringStringEntryProto` | repeated +| `name` | 1 | 2 | `string` | mandatory +| `type` | 2 | 2 | `TypeProto` | mandatory +| `doc_string` | 3 | 2 | `string` | optional +| `metadata_props` | 4 | 2 | `StringStringEntryProto` | repeated |=== [[schema-nodeproto]] @@ -130,20 +132,20 @@ Tags and names reserved in `AttributeProto`, which SHALL NOT be used: `12, 16 to [[tbl-schema-nodeproto]] .Fields of `NodeProto` -[cols="3,1,3,2"] +[cols="3,1,1,3,2"] |=== -| Field | Tag | Type | Obligation +| Field | Tag | Wire type | Type | Obligation -| `input` | 1 | `string` | repeated -| `output` | 2 | `string` | repeated -| `name` | 3 | `string` | optional -| `op_type` | 4 | `string` | optional -| `domain` | 7 | `string` | optional -| `overload` | 8 | `string` | optional -| `attribute` | 5 | `AttributeProto` | repeated -| `doc_string` | 6 | `string` | optional -| `metadata_props` | 9 | `StringStringEntryProto` | repeated -| `device_configurations` | 10 | `NodeDeviceConfigurationProto` | repeated +| `input` | 1 | 2 | `string` | repeated +| `output` | 2 | 2 | `string` | repeated +| `name` | 3 | 2 | `string` | optional +| `op_type` | 4 | 2 | `string` | optional +| `domain` | 7 | 2 | `string` | optional +| `overload` | 8 | 2 | `string` | optional +| `attribute` | 5 | 2 | `AttributeProto` | repeated +| `doc_string` | 6 | 2 | `string` | optional +| `metadata_props` | 9 | 2 | `StringStringEntryProto` | repeated +| `device_configurations` | 10 | 2 | `NodeDeviceConfigurationProto` | repeated |=== [[schema-intintlistentryproto]] @@ -151,12 +153,12 @@ Tags and names reserved in `AttributeProto`, which SHALL NOT be used: `12, 16 to [[tbl-schema-intintlistentryproto]] .Fields of `IntIntListEntryProto` -[cols="3,1,3,2"] +[cols="3,1,1,3,2"] |=== -| Field | Tag | Type | Obligation +| Field | Tag | Wire type | Type | Obligation -| `key` | 1 | `int64` | optional -| `value` | 2 | `int64` | repeated +| `key` | 1 | 0 | `int64` | optional +| `value` | 2 | 0 | `int64` | repeated |=== [[schema-nodedeviceconfigurationproto]] @@ -164,13 +166,13 @@ Tags and names reserved in `AttributeProto`, which SHALL NOT be used: `12, 16 to [[tbl-schema-nodedeviceconfigurationproto]] .Fields of `NodeDeviceConfigurationProto` -[cols="3,1,3,2"] +[cols="3,1,1,3,2"] |=== -| Field | Tag | Type | Obligation +| Field | Tag | Wire type | Type | Obligation -| `configuration_id` | 1 | `string` | mandatory -| `sharding_spec` | 2 | `ShardingSpecProto` | repeated -| `pipeline_stage` | 3 | `int32` | optional +| `configuration_id` | 1 | 2 | `string` | mandatory +| `sharding_spec` | 2 | 2 | `ShardingSpecProto` | repeated +| `pipeline_stage` | 3 | 0 | `int32` | optional |=== [[schema-shardingspecproto]] @@ -178,14 +180,14 @@ Tags and names reserved in `AttributeProto`, which SHALL NOT be used: `12, 16 to [[tbl-schema-shardingspecproto]] .Fields of `ShardingSpecProto` -[cols="3,1,3,2"] +[cols="3,1,1,3,2"] |=== -| Field | Tag | Type | Obligation +| Field | Tag | Wire type | Type | Obligation -| `tensor_name` | 1 | `string` | mandatory -| `device` | 2 | `int64` | repeated -| `index_to_device_group_map` | 3 | `IntIntListEntryProto` | repeated -| `sharded_dim` | 4 | `ShardedDimProto` | repeated +| `tensor_name` | 1 | 2 | `string` | mandatory +| `device` | 2 | 0 | `int64` | repeated +| `index_to_device_group_map` | 3 | 2 | `IntIntListEntryProto` | repeated +| `sharded_dim` | 4 | 2 | `ShardedDimProto` | repeated |=== [[schema-shardeddimproto]] @@ -193,12 +195,12 @@ Tags and names reserved in `AttributeProto`, which SHALL NOT be used: `12, 16 to [[tbl-schema-shardeddimproto]] .Fields of `ShardedDimProto` -[cols="3,1,3,2"] +[cols="3,1,1,3,2"] |=== -| Field | Tag | Type | Obligation +| Field | Tag | Wire type | Type | Obligation -| `axis` | 1 | `int64` | mandatory -| `simple_sharding` | 2 | `SimpleShardedDimProto` | repeated +| `axis` | 1 | 0 | `int64` | mandatory +| `simple_sharding` | 2 | 2 | `SimpleShardedDimProto` | repeated |=== [[schema-simpleshardeddimproto]] @@ -206,13 +208,13 @@ Tags and names reserved in `AttributeProto`, which SHALL NOT be used: `12, 16 to [[tbl-schema-simpleshardeddimproto]] .Fields of `SimpleShardedDimProto` -[cols="3,1,3,2"] +[cols="3,1,1,3,2"] |=== -| Field | Tag | Type | Obligation +| Field | Tag | Wire type | Type | Obligation -| `dim_value` | 1 | `int64` | one of the group -| `dim_param` | 2 | `string` | one of the group -| `num_shards` | 3 | `int64` | mandatory +| `dim_value` | 1 | 0 | `int64` | one of the group +| `dim_param` | 2 | 2 | `string` | one of the group +| `num_shards` | 3 | 0 | `int64` | mandatory |=== Exactly one of `dim_value`, `dim_param` SHALL be present, being the group `dim`. @@ -222,14 +224,14 @@ Exactly one of `dim_value`, `dim_param` SHALL be present, being the group `dim`. [[tbl-schema-traininginfoproto]] .Fields of `TrainingInfoProto` -[cols="3,1,3,2"] +[cols="3,1,1,3,2"] |=== -| Field | Tag | Type | Obligation +| Field | Tag | Wire type | Type | Obligation -| `initialization` | 1 | `GraphProto` | optional -| `algorithm` | 2 | `GraphProto` | optional -| `initialization_binding` | 3 | `StringStringEntryProto` | repeated -| `update_binding` | 4 | `StringStringEntryProto` | repeated +| `initialization` | 1 | 2 | `GraphProto` | optional +| `algorithm` | 2 | 2 | `GraphProto` | optional +| `initialization_binding` | 3 | 2 | `StringStringEntryProto` | repeated +| `update_binding` | 4 | 2 | `StringStringEntryProto` | repeated |=== [[schema-modelproto]] @@ -237,22 +239,22 @@ Exactly one of `dim_value`, `dim_param` SHALL be present, being the group `dim`. [[tbl-schema-modelproto]] .Fields of `ModelProto` -[cols="3,1,3,2"] +[cols="3,1,1,3,2"] |=== -| Field | Tag | Type | Obligation +| Field | Tag | Wire type | Type | Obligation -| `ir_version` | 1 | `int64` | optional -| `opset_import` | 8 | `OperatorSetIdProto` | repeated -| `producer_name` | 2 | `string` | optional -| `producer_version` | 3 | `string` | optional -| `domain` | 4 | `string` | optional -| `model_version` | 5 | `int64` | optional -| `doc_string` | 6 | `string` | optional -| `graph` | 7 | `GraphProto` | optional -| `metadata_props` | 14 | `StringStringEntryProto` | repeated -| `training_info` | 20 | `TrainingInfoProto` | repeated -| `functions` | 25 | `FunctionProto` | repeated -| `configuration` | 26 | `DeviceConfigurationProto` | repeated +| `ir_version` | 1 | 0 | `int64` | optional +| `opset_import` | 8 | 2 | `OperatorSetIdProto` | repeated +| `producer_name` | 2 | 2 | `string` | optional +| `producer_version` | 3 | 2 | `string` | optional +| `domain` | 4 | 2 | `string` | optional +| `model_version` | 5 | 0 | `int64` | optional +| `doc_string` | 6 | 2 | `string` | optional +| `graph` | 7 | 2 | `GraphProto` | optional +| `metadata_props` | 14 | 2 | `StringStringEntryProto` | repeated +| `training_info` | 20 | 2 | `TrainingInfoProto` | repeated +| `functions` | 25 | 2 | `FunctionProto` | repeated +| `configuration` | 26 | 2 | `DeviceConfigurationProto` | repeated |=== [[schema-deviceconfigurationproto]] @@ -260,13 +262,13 @@ Exactly one of `dim_value`, `dim_param` SHALL be present, being the group `dim`. [[tbl-schema-deviceconfigurationproto]] .Fields of `DeviceConfigurationProto` -[cols="3,1,3,2"] +[cols="3,1,1,3,2"] |=== -| Field | Tag | Type | Obligation +| Field | Tag | Wire type | Type | Obligation -| `name` | 1 | `string` | mandatory -| `num_devices` | 2 | `int32` | mandatory -| `device` | 3 | `string` | repeated +| `name` | 1 | 2 | `string` | mandatory +| `num_devices` | 2 | 0 | `int32` | mandatory +| `device` | 3 | 2 | `string` | repeated |=== [[schema-stringstringentryproto]] @@ -274,12 +276,12 @@ Exactly one of `dim_value`, `dim_param` SHALL be present, being the group `dim`. [[tbl-schema-stringstringentryproto]] .Fields of `StringStringEntryProto` -[cols="3,1,3,2"] +[cols="3,1,1,3,2"] |=== -| Field | Tag | Type | Obligation +| Field | Tag | Wire type | Type | Obligation -| `key` | 1 | `string` | optional -| `value` | 2 | `string` | optional +| `key` | 1 | 2 | `string` | optional +| `value` | 2 | 2 | `string` | optional |=== [[schema-tensorannotation]] @@ -287,12 +289,12 @@ Exactly one of `dim_value`, `dim_param` SHALL be present, being the group `dim`. [[tbl-schema-tensorannotation]] .Fields of `TensorAnnotation` -[cols="3,1,3,2"] +[cols="3,1,1,3,2"] |=== -| Field | Tag | Type | Obligation +| Field | Tag | Wire type | Type | Obligation -| `tensor_name` | 1 | `string` | optional -| `quant_parameter_tensor_names` | 2 | `StringStringEntryProto` | repeated +| `tensor_name` | 1 | 2 | `string` | optional +| `quant_parameter_tensor_names` | 2 | 2 | `StringStringEntryProto` | repeated |=== [[schema-graphproto]] @@ -300,20 +302,20 @@ Exactly one of `dim_value`, `dim_param` SHALL be present, being the group `dim`. [[tbl-schema-graphproto]] .Fields of `GraphProto` -[cols="3,1,3,2"] +[cols="3,1,1,3,2"] |=== -| Field | Tag | Type | Obligation +| Field | Tag | Wire type | Type | Obligation -| `node` | 1 | `NodeProto` | repeated -| `name` | 2 | `string` | optional -| `initializer` | 5 | `TensorProto` | repeated -| `sparse_initializer` | 15 | `SparseTensorProto` | repeated -| `doc_string` | 10 | `string` | optional -| `input` | 11 | `ValueInfoProto` | repeated -| `output` | 12 | `ValueInfoProto` | repeated -| `value_info` | 13 | `ValueInfoProto` | repeated -| `quantization_annotation` | 14 | `TensorAnnotation` | repeated -| `metadata_props` | 16 | `StringStringEntryProto` | repeated +| `node` | 1 | 2 | `NodeProto` | repeated +| `name` | 2 | 2 | `string` | optional +| `initializer` | 5 | 2 | `TensorProto` | repeated +| `sparse_initializer` | 15 | 2 | `SparseTensorProto` | repeated +| `doc_string` | 10 | 2 | `string` | optional +| `input` | 11 | 2 | `ValueInfoProto` | repeated +| `output` | 12 | 2 | `ValueInfoProto` | repeated +| `value_info` | 13 | 2 | `ValueInfoProto` | repeated +| `quantization_annotation` | 14 | 2 | `TensorAnnotation` | repeated +| `metadata_props` | 16 | 2 | `StringStringEntryProto` | repeated |=== Tags and names reserved in `GraphProto`, which SHALL NOT be used: `3, 4, 6 to 9`; `"ir_version", "producer_version", "producer_tag", "domain"`. @@ -323,25 +325,25 @@ Tags and names reserved in `GraphProto`, which SHALL NOT be used: `3, 4, 6 to 9` [[tbl-schema-tensorproto]] .Fields of `TensorProto` -[cols="3,1,3,2"] -|=== -| Field | Tag | Type | Obligation - -| `dims` | 1 | `int64` | repeated -| `data_type` | 2 | `int32` | optional -| `segment` | 3 | `Segment` | optional -| `float_data` | 4 | `float` | repeated -| `int32_data` | 5 | `int32` | repeated -| `string_data` | 6 | `bytes` | repeated -| `int64_data` | 7 | `int64` | repeated -| `name` | 8 | `string` | optional -| `doc_string` | 12 | `string` | optional -| `raw_data` | 9 | `bytes` | optional -| `external_data` | 13 | `StringStringEntryProto` | repeated -| `data_location` | 14 | `DataLocation` | optional -| `double_data` | 10 | `double` | repeated -| `uint64_data` | 11 | `uint64` | repeated -| `metadata_props` | 16 | `StringStringEntryProto` | repeated +[cols="3,1,1,3,2"] +|=== +| Field | Tag | Wire type | Type | Obligation + +| `dims` | 1 | 0 | `int64` | repeated +| `data_type` | 2 | 0 | `int32` | optional +| `segment` | 3 | 2 | `Segment` | optional +| `float_data` | 4 | 2 | `float` | repeated, packed +| `int32_data` | 5 | 2 | `int32` | repeated, packed +| `string_data` | 6 | 2 | `bytes` | repeated +| `int64_data` | 7 | 2 | `int64` | repeated, packed +| `name` | 8 | 2 | `string` | optional +| `doc_string` | 12 | 2 | `string` | optional +| `raw_data` | 9 | 2 | `bytes` | optional +| `external_data` | 13 | 2 | `StringStringEntryProto` | repeated +| `data_location` | 14 | 0 | `DataLocation` | optional +| `double_data` | 10 | 2 | `double` | repeated, packed +| `uint64_data` | 11 | 2 | `uint64` | repeated, packed +| `metadata_props` | 16 | 2 | `StringStringEntryProto` | repeated |=== [[tbl-schema-tensorproto-datatype]] @@ -396,12 +398,12 @@ Tags and names reserved in `GraphProto`, which SHALL NOT be used: `3, 4, 6 to 9` [[tbl-schema-tensorproto-segment]] .Fields of `TensorProto.Segment` -[cols="3,1,3,2"] +[cols="3,1,1,3,2"] |=== -| Field | Tag | Type | Obligation +| Field | Tag | Wire type | Type | Obligation -| `begin` | 1 | `int64` | optional -| `end` | 2 | `int64` | optional +| `begin` | 1 | 0 | `int64` | optional +| `end` | 2 | 0 | `int64` | optional |=== [[schema-sparsetensorproto]] @@ -409,13 +411,13 @@ Tags and names reserved in `GraphProto`, which SHALL NOT be used: `3, 4, 6 to 9` [[tbl-schema-sparsetensorproto]] .Fields of `SparseTensorProto` -[cols="3,1,3,2"] +[cols="3,1,1,3,2"] |=== -| Field | Tag | Type | Obligation +| Field | Tag | Wire type | Type | Obligation -| `values` | 1 | `TensorProto` | optional -| `indices` | 2 | `TensorProto` | optional -| `dims` | 3 | `int64` | repeated +| `values` | 1 | 2 | `TensorProto` | optional +| `indices` | 2 | 2 | `TensorProto` | optional +| `dims` | 3 | 0 | `int64` | repeated |=== [[schema-tensorshapeproto]] @@ -423,11 +425,11 @@ Tags and names reserved in `GraphProto`, which SHALL NOT be used: `3, 4, 6 to 9` [[tbl-schema-tensorshapeproto]] .Fields of `TensorShapeProto` -[cols="3,1,3,2"] +[cols="3,1,1,3,2"] |=== -| Field | Tag | Type | Obligation +| Field | Tag | Wire type | Type | Obligation -| `dim` | 1 | `Dimension` | repeated +| `dim` | 1 | 2 | `Dimension` | repeated |=== [[schema-tensorshapeproto-dimension]] @@ -435,13 +437,13 @@ Tags and names reserved in `GraphProto`, which SHALL NOT be used: `3, 4, 6 to 9` [[tbl-schema-tensorshapeproto-dimension]] .Fields of `TensorShapeProto.Dimension` -[cols="3,1,3,2"] +[cols="3,1,1,3,2"] |=== -| Field | Tag | Type | Obligation +| Field | Tag | Wire type | Type | Obligation -| `dim_value` | 1 | `int64` | one of the group -| `dim_param` | 2 | `string` | one of the group -| `denotation` | 3 | `string` | optional +| `dim_value` | 1 | 0 | `int64` | one of the group +| `dim_param` | 2 | 2 | `string` | one of the group +| `denotation` | 3 | 2 | `string` | optional |=== Exactly one of `dim_value`, `dim_param` SHALL be present, being the group `value`. @@ -451,17 +453,17 @@ Exactly one of `dim_value`, `dim_param` SHALL be present, being the group `value [[tbl-schema-typeproto]] .Fields of `TypeProto` -[cols="3,1,3,2"] +[cols="3,1,1,3,2"] |=== -| Field | Tag | Type | Obligation +| Field | Tag | Wire type | Type | Obligation -| `tensor_type` | 1 | `Tensor` | one of the group -| `sequence_type` | 4 | `Sequence` | one of the group -| `map_type` | 5 | `Map` | one of the group -| `optional_type` | 9 | `Optional` | one of the group -| `sparse_tensor_type` | 8 | `SparseTensor` | one of the group -| `opaque_type` | 7 | `Opaque` | one of the group -| `denotation` | 6 | `string` | optional +| `tensor_type` | 1 | 2 | `Tensor` | one of the group +| `sequence_type` | 4 | 2 | `Sequence` | one of the group +| `map_type` | 5 | 2 | `Map` | one of the group +| `optional_type` | 9 | 2 | `Optional` | one of the group +| `sparse_tensor_type` | 8 | 2 | `SparseTensor` | one of the group +| `opaque_type` | 7 | 2 | `Opaque` | one of the group +| `denotation` | 6 | 2 | `string` | optional |=== Exactly one of `tensor_type`, `sequence_type`, `map_type`, `optional_type`, `sparse_tensor_type`, `opaque_type` SHALL be present, being the group `value`. @@ -471,12 +473,12 @@ Exactly one of `tensor_type`, `sequence_type`, `map_type`, `optional_type`, `spa [[tbl-schema-typeproto-tensor]] .Fields of `TypeProto.Tensor` -[cols="3,1,3,2"] +[cols="3,1,1,3,2"] |=== -| Field | Tag | Type | Obligation +| Field | Tag | Wire type | Type | Obligation -| `elem_type` | 1 | `int32` | mandatory -| `shape` | 2 | `TensorShapeProto` | optional +| `elem_type` | 1 | 0 | `int32` | mandatory +| `shape` | 2 | 2 | `TensorShapeProto` | optional |=== [[schema-typeproto-sequence]] @@ -484,11 +486,11 @@ Exactly one of `tensor_type`, `sequence_type`, `map_type`, `optional_type`, `spa [[tbl-schema-typeproto-sequence]] .Fields of `TypeProto.Sequence` -[cols="3,1,3,2"] +[cols="3,1,1,3,2"] |=== -| Field | Tag | Type | Obligation +| Field | Tag | Wire type | Type | Obligation -| `elem_type` | 1 | `TypeProto` | mandatory +| `elem_type` | 1 | 2 | `TypeProto` | mandatory |=== [[schema-typeproto-map]] @@ -496,12 +498,12 @@ Exactly one of `tensor_type`, `sequence_type`, `map_type`, `optional_type`, `spa [[tbl-schema-typeproto-map]] .Fields of `TypeProto.Map` -[cols="3,1,3,2"] +[cols="3,1,1,3,2"] |=== -| Field | Tag | Type | Obligation +| Field | Tag | Wire type | Type | Obligation -| `key_type` | 1 | `int32` | mandatory -| `value_type` | 2 | `TypeProto` | mandatory +| `key_type` | 1 | 0 | `int32` | mandatory +| `value_type` | 2 | 2 | `TypeProto` | mandatory |=== [[schema-typeproto-optional]] @@ -509,11 +511,11 @@ Exactly one of `tensor_type`, `sequence_type`, `map_type`, `optional_type`, `spa [[tbl-schema-typeproto-optional]] .Fields of `TypeProto.Optional` -[cols="3,1,3,2"] +[cols="3,1,1,3,2"] |=== -| Field | Tag | Type | Obligation +| Field | Tag | Wire type | Type | Obligation -| `elem_type` | 1 | `TypeProto` | mandatory +| `elem_type` | 1 | 2 | `TypeProto` | mandatory |=== [[schema-typeproto-sparsetensor]] @@ -521,12 +523,12 @@ Exactly one of `tensor_type`, `sequence_type`, `map_type`, `optional_type`, `spa [[tbl-schema-typeproto-sparsetensor]] .Fields of `TypeProto.SparseTensor` -[cols="3,1,3,2"] +[cols="3,1,1,3,2"] |=== -| Field | Tag | Type | Obligation +| Field | Tag | Wire type | Type | Obligation -| `elem_type` | 1 | `int32` | mandatory -| `shape` | 2 | `TensorShapeProto` | optional +| `elem_type` | 1 | 0 | `int32` | mandatory +| `shape` | 2 | 2 | `TensorShapeProto` | optional |=== [[schema-typeproto-opaque]] @@ -534,12 +536,12 @@ Exactly one of `tensor_type`, `sequence_type`, `map_type`, `optional_type`, `spa [[tbl-schema-typeproto-opaque]] .Fields of `TypeProto.Opaque` -[cols="3,1,3,2"] +[cols="3,1,1,3,2"] |=== -| Field | Tag | Type | Obligation +| Field | Tag | Wire type | Type | Obligation -| `domain` | 1 | `string` | optional -| `name` | 2 | `string` | optional +| `domain` | 1 | 2 | `string` | optional +| `name` | 2 | 2 | `string` | optional |=== Tags and names reserved in `TypeProto.Opaque`, which SHALL NOT be used: `3`; `"parameters"`. @@ -549,12 +551,12 @@ Tags and names reserved in `TypeProto.Opaque`, which SHALL NOT be used: `3`; `"p [[tbl-schema-operatorsetidproto]] .Fields of `OperatorSetIdProto` -[cols="3,1,3,2"] +[cols="3,1,1,3,2"] |=== -| Field | Tag | Type | Obligation +| Field | Tag | Wire type | Type | Obligation -| `domain` | 1 | `string` | mandatory -| `version` | 2 | `int64` | mandatory +| `domain` | 1 | 2 | `string` | mandatory +| `version` | 2 | 0 | `int64` | mandatory |=== [[schema-operatorstatus]] @@ -575,22 +577,22 @@ Tags and names reserved in `TypeProto.Opaque`, which SHALL NOT be used: `3`; `"p [[tbl-schema-functionproto]] .Fields of `FunctionProto` -[cols="3,1,3,2"] +[cols="3,1,1,3,2"] |=== -| Field | Tag | Type | Obligation +| Field | Tag | Wire type | Type | Obligation -| `name` | 1 | `string` | optional -| `input` | 4 | `string` | repeated -| `output` | 5 | `string` | repeated -| `attribute` | 6 | `string` | repeated -| `attribute_proto` | 11 | `AttributeProto` | repeated -| `node` | 7 | `NodeProto` | repeated -| `doc_string` | 8 | `string` | optional -| `opset_import` | 9 | `OperatorSetIdProto` | repeated -| `domain` | 10 | `string` | optional -| `overload` | 13 | `string` | optional -| `value_info` | 12 | `ValueInfoProto` | repeated -| `metadata_props` | 14 | `StringStringEntryProto` | repeated +| `name` | 1 | 2 | `string` | optional +| `input` | 4 | 2 | `string` | repeated +| `output` | 5 | 2 | `string` | repeated +| `attribute` | 6 | 2 | `string` | repeated +| `attribute_proto` | 11 | 2 | `AttributeProto` | repeated +| `node` | 7 | 2 | `NodeProto` | repeated +| `doc_string` | 8 | 2 | `string` | optional +| `opset_import` | 9 | 2 | `OperatorSetIdProto` | repeated +| `domain` | 10 | 2 | `string` | optional +| `overload` | 13 | 2 | `string` | optional +| `value_info` | 12 | 2 | `ValueInfoProto` | repeated +| `metadata_props` | 14 | 2 | `StringStringEntryProto` | repeated |=== Tags and names reserved in `FunctionProto`, which SHALL NOT be used: `2`; `"since_version"`; `3`; `"status"`. @@ -602,14 +604,14 @@ Tags and names reserved in `FunctionProto`, which SHALL NOT be used: `2`; `"sinc [[tbl-schema-operatorproto]] .Fields of `OperatorProto` -[cols="3,1,3,2"] +[cols="3,1,1,3,2"] |=== -| Field | Tag | Type | Obligation +| Field | Tag | Wire type | Type | Obligation -| `op_type` | 1 | `string` | mandatory -| `since_version` | 2 | `int64` | mandatory -| `status` | 3 | `OperatorStatus` | optional -| `doc_string` | 10 | `string` | optional +| `op_type` | 1 | 2 | `string` | mandatory +| `since_version` | 2 | 0 | `int64` | mandatory +| `status` | 3 | 0 | `OperatorStatus` | optional +| `doc_string` | 10 | 2 | `string` | optional |=== [[schema-operatorsetproto]] @@ -617,19 +619,19 @@ Tags and names reserved in `FunctionProto`, which SHALL NOT be used: `2`; `"sinc [[tbl-schema-operatorsetproto]] .Fields of `OperatorSetProto` -[cols="3,1,3,2"] +[cols="3,1,1,3,2"] |=== -| Field | Tag | Type | Obligation +| Field | Tag | Wire type | Type | Obligation -| `magic` | 1 | `string` | mandatory -| `ir_version` | 2 | `int64` | mandatory -| `ir_version_prerelease` | 3 | `string` | optional -| `ir_build_metadata` | 7 | `string` | optional -| `domain` | 4 | `string` | optional -| `opset_version` | 5 | `int64` | optional -| `doc_string` | 6 | `string` | optional -| `operator` | 8 | `OperatorProto` | repeated -| `functions` | 9 | `FunctionProto` | repeated +| `magic` | 1 | 2 | `string` | mandatory +| `ir_version` | 2 | 0 | `int64` | mandatory +| `ir_version_prerelease` | 3 | 2 | `string` | optional +| `ir_build_metadata` | 7 | 2 | `string` | optional +| `domain` | 4 | 2 | `string` | optional +| `opset_version` | 5 | 0 | `int64` | optional +| `doc_string` | 6 | 2 | `string` | optional +| `operator` | 8 | 2 | `OperatorProto` | repeated +| `functions` | 9 | 2 | `FunctionProto` | repeated |=== === Map and sequence messages @@ -639,17 +641,17 @@ Tags and names reserved in `FunctionProto`, which SHALL NOT be used: `2`; `"sinc [[tbl-schema-sequenceproto]] .Fields of `SequenceProto` -[cols="3,1,3,2"] +[cols="3,1,1,3,2"] |=== -| Field | Tag | Type | Obligation +| Field | Tag | Wire type | Type | Obligation -| `name` | 1 | `string` | optional -| `elem_type` | 2 | `int32` | optional -| `tensor_values` | 3 | `TensorProto` | repeated -| `sparse_tensor_values` | 4 | `SparseTensorProto` | repeated -| `sequence_values` | 5 | `SequenceProto` | repeated -| `map_values` | 6 | `MapProto` | repeated -| `optional_values` | 7 | `OptionalProto` | repeated +| `name` | 1 | 2 | `string` | optional +| `elem_type` | 2 | 0 | `int32` | optional +| `tensor_values` | 3 | 2 | `TensorProto` | repeated +| `sparse_tensor_values` | 4 | 2 | `SparseTensorProto` | repeated +| `sequence_values` | 5 | 2 | `SequenceProto` | repeated +| `map_values` | 6 | 2 | `MapProto` | repeated +| `optional_values` | 7 | 2 | `OptionalProto` | repeated |=== [[tbl-schema-sequenceproto-datatype]] @@ -671,15 +673,15 @@ Tags and names reserved in `FunctionProto`, which SHALL NOT be used: `2`; `"sinc [[tbl-schema-mapproto]] .Fields of `MapProto` -[cols="3,1,3,2"] +[cols="3,1,1,3,2"] |=== -| Field | Tag | Type | Obligation +| Field | Tag | Wire type | Type | Obligation -| `name` | 1 | `string` | optional -| `key_type` | 2 | `int32` | optional -| `keys` | 3 | `int64` | repeated -| `string_keys` | 4 | `bytes` | repeated -| `values` | 5 | `SequenceProto` | optional +| `name` | 1 | 2 | `string` | optional +| `key_type` | 2 | 0 | `int32` | optional +| `keys` | 3 | 0 | `int64` | repeated +| `string_keys` | 4 | 2 | `bytes` | repeated +| `values` | 5 | 2 | `SequenceProto` | optional |=== [[schema-optionalproto]] @@ -687,17 +689,17 @@ Tags and names reserved in `FunctionProto`, which SHALL NOT be used: `2`; `"sinc [[tbl-schema-optionalproto]] .Fields of `OptionalProto` -[cols="3,1,3,2"] +[cols="3,1,1,3,2"] |=== -| Field | Tag | Type | Obligation +| Field | Tag | Wire type | Type | Obligation -| `name` | 1 | `string` | optional -| `elem_type` | 2 | `int32` | optional -| `tensor_value` | 3 | `TensorProto` | optional -| `sparse_tensor_value` | 4 | `SparseTensorProto` | optional -| `sequence_value` | 5 | `SequenceProto` | optional -| `map_value` | 6 | `MapProto` | optional -| `optional_value` | 7 | `OptionalProto` | optional +| `name` | 1 | 2 | `string` | optional +| `elem_type` | 2 | 0 | `int32` | optional +| `tensor_value` | 3 | 2 | `TensorProto` | optional +| `sparse_tensor_value` | 4 | 2 | `SparseTensorProto` | optional +| `sequence_value` | 5 | 2 | `SequenceProto` | optional +| `map_value` | 6 | 2 | `MapProto` | optional +| `optional_value` | 7 | 2 | `OptionalProto` | optional |=== [[tbl-schema-optionalproto-datatype]] @@ -714,6 +716,6 @@ Tags and names reserved in `FunctionProto`, which SHALL NOT be used: `2`; `"sinc | `OPTIONAL` | 5 |=== -=== Schema source +=== Derivation -The schema source is vendored in the repository at `upstream/onnx/proto/`, at the release recorded in `upstream/onnx/SOURCE.txt`. It is an informative aid: where a comment in it carries a normative statement, that statement is in the clause it belongs to and not in this annex. +The tables above are derived from the Protocol Buffers schema of the ONNX reference implementation, vendored in the repository at `upstream/onnx/proto/` at the release recorded in `upstream/onnx/SOURCE.txt`. That schema is an informative aid, not a normative reference: the tables above are normative, and where a comment in the schema carries a normative statement, that statement is in the clause it belongs to. See <>. diff --git a/sources/part1/sections/annex-c-known-gaps.adoc b/sources/part1/sections/annex-c-known-gaps.adoc index 6130a35..70f64e7 100644 --- a/sources/part1/sections/annex-c-known-gaps.adoc +++ b/sources/part1/sections/annex-c-known-gaps.adoc @@ -29,10 +29,6 @@ form a standard cannot adopt. <> cannot pass or fail a vector without one. | Blocking -| Clause 2, <> -| Protocol Buffers has no citable specification with a stable identifier. -| Blocking - | <> | The checks a validating consumer owes are drafted but not yet stated as testable conditions with severities. @@ -94,13 +90,12 @@ form a standard cannot adopt. be implemented from the documentation at all. | Major -| Clause 12 (encoding) -| The admissible subset of the Protocol Buffers wire format --- unknown fields, - repeated field ordering, maximum message size --- is unstated. It cannot be - derived from the schema, which says nothing about any of the three; the - reference implementation enforces a maximum message size in its checker, - whose source is not part of the vendored baseline. -| Major +| <> (limits) +| No maximum message size is stated. The rest of the encoding is now specified + in <>, but the reference implementation applies a limit of 2 + gigabytes to a single message and this document neither fixes that nor + leaves it explicitly to the implementation. +| Minor | <> | The annex states which fields are mandatory at one IR version, being the one @@ -180,6 +175,10 @@ transcription rather than take it on trust, and so that a refresh of | <> (the whole annex) | `upstream/onnx/proto/`, generated by `scripts/generate-schema.rb` +| <> (the binary encoding) +| `upstream/onnx/proto/`, for the types and tags; the encoding itself is + checked against a reference implementation by `scripts/check-encoding.rb` + | <> (the IR version history) | `docs/Versioning.md`, "Released Versions"; `onnx-ml.proto`, `Version` diff --git a/sources/part1/sections/bibliography.adoc b/sources/part1/sections/bibliography.adoc index 0ae893f..b9c5e67 100644 --- a/sources/part1/sections/bibliography.adoc +++ b/sources/part1/sections/bibliography.adoc @@ -6,6 +6,9 @@ * [[[isopas,ISO/IEC JTC 1 PAS]]], _Publicly Available Specification (PAS) transposition process_ +* [[[protobuf,Protocol Buffers]]], _Protocol Buffers_, + https://protobuf.dev + * [[[onnxrepo,ONNX]]], _Open Neural Network Exchange_, https://onnx.ai * [[[jdf,JDF]]], _Joint Development Foundation_,