Validate and inline external data in node tensor attributes during session initialization

[yuslepukhin]

This pull request introduces improved support and validation for external data in tensor attributes, particularly ensuring that external data in node attributes is properly validated and inlined, and that in-memory references are correctly rejected. Additionally, it introduces new tests to cover these scenarios, and refactors some utility functions in the test code for clarity and consistency.

External Data Handling and Validation:

• Updated Graph::ConvertInitializersIntoOrtValues() in graph.cc to:
  • Use InlinedHashSet for tracking validated external data paths for efficiency.
  • Add a new step that validates and inlines external data in node tensor attributes, ensuring that only file-based external data is accepted and in-memory references are rejected. This guarantees all execution providers have uniform access to attribute data. [1] [2]

Testing:

• Added comprehensive tests in label_encoder_test.cc to verify:
  • Valid external data in tensor attributes is loaded and inlined.
  • In-memory external data references in node attributes are rejected.
  • Duplicate key handling and singleton default tensor behavior.
• Added a test in tree_ensembler_test.cc to ensure that in-memory external data references in node attributes are rejected, preventing invalid attribute usage.

Test Utility Refactoring:

• Refactored utility functions in tree_ensembler_test.cc and treeregressor_test.cc:
  • Renamed and standardized helper functions for array and string manipulations to improve code clarity (e.g., MultiplyUpdateArray, MultiplyArraysValues, MultiplyUpdateArrayString). [1] [2] [3] [4] [5] [6] [7] [8] [9]

Test File Includes:

• Added necessary includes for file utilities and path handling in test files to support new test scenarios involving external data files. [1] [2] [3]

[Copilot]

Pull request overview

This PR hardens ONNX Runtime’s CPU ML operator handling by rejecting tensor attributes that reference ONNX external data, preventing kernels from attempting to load attribute data from external locations during kernel construction.

Changes:

• Add kernel-side validation to reject TensorProto attributes that use external data for LabelEncoder and TreeEnsemble*.
• Add CPU unit tests asserting construction-time failure when external-data tensor attributes are provided for LabelEncoder, TreeEnsemble, and TreeEnsembleRegressor.
• Standardize failure messaging to include “external data is not supported” for test matching.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
onnxruntime/core/providers/cpu/ml/label_encoder.h	Enforces rejection of external-data tensor attributes for keys_tensor/values_tensor and default_tensor.
onnxruntime/core/providers/cpu/ml/tree_ensemble_helper.cc	Adds a guard in tensor attribute unpacking to reject external data.
onnxruntime/test/providers/cpu/ml/label_encoder_test.cc	Adds negative tests for external-data tensor attributes on LabelEncoder (opset 4), excluding CUDA EP.
onnxruntime/test/providers/cpu/ml/tree_ensembler_test.cc	Adds a negative test ensuring TreeEnsemble rejects external-data tensor attributes.
onnxruntime/test/providers/cpu/ml/treeregressor_test.cc	Adds a negative test ensuring TreeEnsembleRegressor rejects external-data tensor attributes.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

onnxruntime/core/providers/cpu/ml/label_encoder.h:146

• The external-data rejection for default_tensor is only applied when utils::HasDataType(attr_tensor_proto) is true. A malformed default_tensor TensorProto could set data_location=EXTERNAL (and external_data entries) but omit data_type, and this would bypass the intended rejection and fall back to the non-tensor default attributes. If the goal is to reject any tensor attribute that uses external data, the check should run whenever GetAttr succeeds, regardless of data_type presence.

  auto result = info.GetAttr("default_tensor", &attr_tensor_proto);
  if (result.IsOK() && utils::HasDataType(attr_tensor_proto)) {
    ORT_ENFORCE(!utils::HasExternalData(attr_tensor_proto),
                "Tensor attribute default_tensor with external data is not supported.");
    T default_value;
    result = utils::UnpackTensor<T>(attr_tensor_proto, std::filesystem::path(), &default_value, 1);
    ORT_ENFORCE(result.IsOK(), "LabelEncoder could not unpack default tensor ", attr_name);
    return default_value;
  } else if constexpr (std::is_same_v<T, std::string> || std::is_same_v<T, float> || std::is_same_v<T, int64_t>) {

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 6 comments.

[xadupre]

I'm curiuous. I thought after loading the model, there was no external tensor. I did not exepect to validate that for every kernel (I'll see many PRs about that then). So do we allow external tensor in kernels in case they want to load the data themselves?

[yuslepukhin]

I'm curiuous. I thought after loading the model, there was no external tensor. I did not expect to validate that for every kernel (I'll see many PRs about that then). So do we allow external tensor in kernels in case they want to load the data themselves?

I re-worked this. We now validate external references and inline the tensor attributes.
This is because kernels currently do not have model paths to properly load the files or in-memory references.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 11 comments.