Expand model package with authoring tools and schema versioning

[jambayk]

Description

Builds out the standalone model_package/ C library so a single library
covers the full lifecycle of an ONNX Runtime model package: inspection,
authoring, content-addressed shared assets, commit, prune, and validation.
The library remains free of any dependency on ONNX Runtime itself.

Public C API (model_package/include/model_package.h)

• Lifecycle: ModelPackage_Open / ModelPackage_New / ModelPackage_Close,
  with ModelPackageOpenOptions controlling external-path access, symlink
  following, and strict unknown-field handling.
• Inspection: a POD ModelPackageInfo tree (ModelPackage_Info) plus
  by-name lookups for components, variants, and per-namespace
  executor_info entries, and round-trip JSON getters that preserve
  fields unknown to the current build.
• Path resolution: ModelPackage_ResolveStringRef implements the package's
  resolution rules — relative paths anchored at a base directory,
  sha256:<hex>[/sub/path] for shared-asset content, and portable vs
  installed confinement (absolute paths and .. segments only allowed
  under layout: "installed").
• Shared assets: SHA-256 directory hashing, ModelPackage_AddSharedAsset
  (with reproducible-build URI check and an optional copy_in staging
  mode), ModelPackage_RemoveSharedAsset, and
  ModelPackage_ResolveAssetUri. Assets under
  <package_root>/shared_assets/ are auto-discovered at Open.
• Authoring: inline/external component setters, variant upsert/remove,
  per-namespace executor_info setters (inline and external), and
  package-level metadata / layout / additional_metadata setters.
  Mutations invalidate cached pointers in the mutated scope and its
  descendants.
• Commit / prune / validate: ModelPackage_Commit writes the in-memory
  model to disk either in place or to a fresh dest_root ("save as"),
  with PRESERVE or DENSE write modes; ModelPackage_Prune reclaims
  unreferenced files under shared_assets/ and tracked orphan
  variant/component directories left behind by removals; and
  ModelPackage_Validate runs schema, path-reachability, asset-rehash,
  and unknown-field checks and returns a JSON report.
• Errors: opaque ModelPackageStatus* with a stable additive
  ModelPackageErrorCode enum (IO, schema, version, path confinement,
  asset missing, asset hash mismatch, not found, invalid arg, state).

Internal layout

The implementation is split into focused translation units:
manifest_parser, model_package_impl, authoring,
commit_prune_validate, path_resolver, asset_hasher, and an
in-tree sha256. Shared error plumbing lives in status_impl.h.

ONNX Runtime integration (onnxruntime/core/session/model_package/)

The ORT-side glue is wired onto the library through the C inspection
and path-resolution entry points. model_package_context now translates
the library's info tree into ORT-internal structs and parses the
executor_info["ort"] payload (model_file, external_data,
session_options, provider_options). When a variant declares
external_data, CreateSessionForModelPackage loads the model from a
memory mapping and passes the resolved folder to the session via
kOrtSessionOptionsModelExternalInitializersFileFolderPath, so external
initializers — including those backed by a shared asset — are picked up
at Initialize time.

The experimental OrtModelPackageApi_*_SinceV28 C entries introduced in
#28990 are unchanged.

Documentation and tests

• model_package/README.md documents the on-disk layout, manifest and
  component schemas, shared-asset rules, path resolution, the authoring
  flow, and commit / prune / validate semantics.
• onnxruntime/core/session/model_package/README.md documents the ORT
  consumer-side glue: the executor_info["ort"] schema, the variant
  selection algorithm, the session-creation contract, and the registered
  experimental C entries.
• New library tests cover inspection, authoring, asset hashing, and
  commit (model_package/tests/). The ORT integration tests in
  onnxruntime/test/autoep/test_model_package.cc are reworked against
  the current C API surface.

Motivation and Context

ORT needs a single library that owns the model package format end to
end — not just reading it, but producing it, validating it, and
maintaining it on disk with content-addressed shared assets.
Consolidating this behind one C API lets ORT, publisher tooling, and
third-party consumers share the same parser, path-resolution rules, and
on-disk invariants without each reimplementing them, and keeps the
library independent of the ORT session runtime.

[Copilot]

Pull request overview

This PR expands the standalone model_package/ library from read-only inspection into a full authoring + validation + commit workflow (with content-addressed shared assets and schema versioning), and updates ONNX Runtime’s model-package integration to consume the library via its public C API (including wiring external_data into session creation).

Changes:

• Replaces the old internal-only model package parser with a new public model_package.h C API and in-library implementation split across focused translation units (parsing, path resolution, hashing, authoring, commit/prune/validate).
• Updates ORT’s model package integration to open/inspect packages via the library C API and adds support for external_data by switching to buffer-load + setting the external-initializers folder hint.
• Adds standalone library tests (inspection/authoring/asset hashing/commit) and new integration documentation.

Reviewed changes

Copilot reviewed 32 out of 32 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
onnxruntime/core/session/utils.cc	Adjusts model-package session creation to support external initializers via buffer-load + config entry.
onnxruntime/core/session/model_package/README.md	New documentation for ORT-side consumption, selection, and experimental API usage.
onnxruntime/core/session/model_package/model_package_context.h	Extends ORT’s internal variant model info with resolved external initializer folder path.
onnxruntime/core/session/model_package/model_package_context.cc	Switches ORT integration to the model_package public C API; parses executor_info["ort"]; resolves model/external_data refs.
onnxruntime/core/session/model_package_api.cc	Tightens error/out-param behavior for experimental model package API getters.
model_package/tests/test_inspection.cc	New standalone inspection API tests.
model_package/tests/test_commit.cc	New commit/prune/validate tests (in-place + dest_root flows).
model_package/tests/test_authoring.cc	New authoring/mutation API tests.
model_package/tests/test_asset_hashing.cc	New SHA-256 + directory Merkle-hash tests for shared assets.
model_package/src/status_impl.h	New internal status representation shared across the library.
model_package/src/sha256.h	New minimal SHA-256 interface for content-addressed assets.
model_package/src/sha256.cc	New SHA-256 implementation (used for directory hashing and asset validation).
model_package/src/path_resolver.h	New path resolution + confinement helpers and sha256 URI helpers.
model_package/src/path_resolver.cc	Implements portable/installed path rules and sha256 URI parsing.
model_package/src/model_package_impl.h	Defines the in-memory package representation and cached Info view materialization.
model_package/src/model_package_impl.cc	Implements lifecycle, info tree, lookups, string ref resolution, and hashing API.
model_package/src/manifest_parser.h	Declares manifest/component/variant parsing + cache refresh helpers.
model_package/src/manifest_parser.cc	Implements manifest/component/variant parsing into the in-memory model.
model_package/src/authoring.cc	Implements mutation APIs (set/remove components/variants/executor_info, shared assets, metadata).
model_package/src/asset_hasher.h	Declares directory Merkle-hash algorithm for canonical asset URIs.
model_package/src/asset_hasher.cc	Implements directory hashing rules (sorted manifest text, reject symlinks, posix paths).
model_package/src/commit_prune_validate.cc	Implements commit (“preserve”/“dense”), prune, and validate workflows.
model_package/src/parser.h	Deletes old internal parser header (superseded by new library).
model_package/src/parser.cc	Deletes old internal parser implementation (superseded by new library).
model_package/src/model_package_internal.h	Deletes old internal type definitions (replaced by new in-memory model).
model_package/src/api.cc	Deletes old standalone API implementation (replaced by new C API surface).
model_package/README.md	Major documentation expansion for layout/schema/assets/path resolution/commit-prune-validate.
model_package/include/model_package.h	New public API header defining lifecycle/inspection/authoring/commit/prune/validate.
model_package/include/model_package_api.h	Refactors shared types (export macro, status handle, stable error enum).
model_package/CMakeLists.txt	Updates sources/installs new headers and adds standalone tests wiring.
cmake/onnxruntime_session.cmake	Updates ORT build integration to include only the model_package public headers and use the public C API.

[chilo-ms]

abi_version  is stored in every struct but never validated. If a caller compiled against a future v2 header passes abi_version = 2 (where field semantics or layout may have changed), this code silently proceeds with v1 interpretation, this might cause issue.

Should we gate on abi_version before using the struct?

[jambayk]

The structs no longer carry an abi_version (or struct_size) field. The model package library is consumed as source — each consumer compiles it into its own binary — so there is no separately-built binary boundary to version, and nothing to gate at runtime. On-disk compatibility is governed solely by schema_version ("major.minor"): the library accepts any package whose major is within its supported range, and consumers branch on schema_version_major / schema_version_minor to know which optional fields a package may carry. See the README "Schema versioning and source distribution" section.

[chilo-ms]

abi_version  is documented as 1  in every struct's doc comment, but there's no named constant for it. Callers have to hardcode the literal 1 , and the library has no single source of truth to validate against.

Consider adding a define:

#define MODEL_PACKAGE_ABI_VERSION 1

[jambayk]

abi_version has been removed entirely rather than turned into a named constant. Since the library ships as source with no binary boundary, an ABI-version macro would be vestigial (the same reasoning that removed struct_size). The only compatibility contract that remains is schema_version (major.minor), which is what consumers key off of.

[chilo-ms]

Same concern here.

It should check abi_version of the ModelComponentInfo struct first, otherwise, consider the scenario where library/api is newer and consumer/caller is older, as per implementation of this function, it might return corrupted data.

[jambayk]

This accessor no longer exists — collections are plain array members again (components / num_components, etc.), and there is no abi_version or struct_size. With source distribution there is a single struct definition per build, so the newer-library / older-caller layout skew that motivated a check cannot arise. A breaking change to the on-disk data contract is instead carried by a schema_version major bump, which the parser gates on.

[Copilot]

Pull request overview

Copilot reviewed 33 out of 33 changed files in this pull request and generated 5 comments.