kernels: verify the signature of a kernel while loading by danieldk · Pull Request #828 · huggingface/kernels

danieldk · 2026-09-15T10:04:30Z

The basics of kernel signature verification were already in kernels, but verification would only be performed when explicitly running the kernels verify-signature command.

This change enables kernel signature verification during kernel loading. However, as part of the step-wise roll-out, signature verification will now only warn if the metadata is not signed, not correctly signed, or the data's digest is incorrect.

Signature verification during kernel loading is implemented as follows. First, we add the KernelValidator protocol, this is the counterpart to MetadataValidator that can only run when the kernel is fully downloaded. get_kernel_with_resolver now requires an additional kernel_validator argument and runs the given validators on the kernel dependency tree after the kernels are downloaded ('installed').

We add SignatureValidator as the first validator to implement the KernelValidator protocol and enable it in the relevant get_.* functions. This validator calls the existing verification structure. Validation is only performed on remote kernels, since local kernels are used in development and are typically not signed.

There was one more wrinkle in that validating the hash of all kernel files on each load is fairly expensive. So upon the first successful verification, we write a receipt that attests that an earlier verification is done. The receipt is content-addressed by the repo id, the resolved Git commit SHA and build variant, so it does not attest for newly-downloaded revisions or build variants.

The receipts allow us to skip the most expensive parts of the verification. However, the user might use a different policy after the verification that produced the receipt, so we will always test the kernel's signing certificate against the policy. This ensures that the kernel signature will be rejected if the kernel violates the policy used during a kernel load.

HuggingFaceDocBuilderDev · 2026-09-15T10:09:22Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

The basics of kernel signature verification were already in `kernels`, but verification would only be performed when explicitly running the `kernels verify-signature` command. This change enables kernel signature verification during kernel loading. However, as part of the step-wise roll-out, signature verification will now only warn if the metadata is not signed, not correctly signed, or the data's digest is incorrect. Signature verification during kernel loading is implemented as follows. First, we add the `KernelValidator` protocol, this is the counterpart to `MetadataValidator` that can only run when the kernel is fully downloaded. `get_kernel_with_resolver` now requires an additional `kernel_validator` argument and runs the given validators on the kernel dependency tree after the kernels are downloaded ('installed'). We add `SignatureValidator` as the first validator to implement the `KernelValidator` protocol and enable it in the relevant `get_.*` functions. This validator calls the existing verification structure. Validation is only performed on remote kernels, since local kernels are used in development and are typically not signed. There was one more wrinkle in that validating the hash of all kernel files on each load is fairly expensive. So upon the first successful verification, we write a receipt that attests that an earlier verification is done. The receipt is content-addressed by the repo id and the resolved Git commit SHA, so it does not attest for newly-downloaded versions. The receipts allow us to skip the most expensive parts of the verification. However, the user might use a different policy after the verification that produced the receipt, so we will always test the kernel's signing certificate against the policy. This ensures that the kernel signature will be rejected if the kernel violates the policy used during a kernel load.

We resolve kernels refs to full commit SHAs as early as possible (since we need immutable identifiers for e.g. receipt caching). However, that also means that we are doing snapshot downloads by Git SHA. As a result, refs such as `v1` do not get cached, breaking non-locked offline use.

github-actions · 2026-09-15T14:04:08Z

Coverage report — `kernels/`

Measured on: Python 3.10 / Torch 2.13.0.
Other CI configurations are not included in this number.
Hardware-gated code paths (ROCm/XPU/NPU/Darwin/Windows) are excluded or unreachable on the Linux+CUDA runner.

Total coverage: 87.2% — threshold: 80% — ✅

Per-file breakdown

Name	Stmts	Miss	Cover	Missing
src/kernels/__init__.py	14	0	100%
src/kernels/_system.py	6	1	83%	10
src/kernels/_versions.py	130	14	89%	53, 59-60, 63-64, 102, 165-170, 199, 219
src/kernels/archs.py	56	1	98%	94
src/kernels/backends.py	213	62	71%	42, 46, 50-53, 70, 92, 110, 119, 123, 127-129, 150, 159, 163, 167-169, 190, 201, 203, 210-213, 226, 230, 234-254, 262, 285-305
src/kernels/compat.py	9	1	89%	5
src/kernels/deps.py	70	1	99%	56
src/kernels/hf_hub.py	63	2	97%	21, 23
src/kernels/importer.py	44	5	89%	80, 84, 87, 101-102
src/kernels/install.py	21	7	67%	76-100
src/kernels/layer/__init__.py	6	0	100%
src/kernels/layer/_interval_tree.py	103	4	96%	23, 52, 147, 150
src/kernels/layer/device.py	48	14	71%	42, 47-49, 91, 96-98, 101, 149, 152, 155-157
src/kernels/layer/func.py	85	6	93%	90, 115, 191, 311, 338, 368
src/kernels/layer/globals.py	5	0	100%
src/kernels/layer/kernelize.py	80	8	90%	258, 293, 301-302, 308, 312, 328-330
src/kernels/layer/layer.py	215	14	93%	182, 229, 256, 390, 470-471, 492, 500, 511, 540, 544, 557, 610, 640
src/kernels/layer/mode.py	14	0	100%
src/kernels/layer/repos.py	144	42	71%	27, 33, 36-43, 63-64, 70, 73-76, 90, 94, 103-104, 110, 113-116, 123-124, 130, 133-136, 143-144, 150, 153-156, 163-164, 170, 173-176, 257
src/kernels/load.py	71	2	97%	338, 378
src/kernels/locking.py	89	64	28%	35-83, 91-98, 102-125, 137, 152-159, 165-175, 179-186
src/kernels/python_deps.py	58	6	90%	59-60, 64-65, 101, 104
src/kernels/resolver.py	156	2	99%	220, 226
src/kernels/status.py	50	2	96%	25, 79
src/kernels/validate.py	88	5	94%	9, 100, 167, 190-191
src/kernels/variants.py	278	19	93%	65, 96, 117, 147, 256-257, 299-302, 304, 388-394, 400-406, 437-443, 455-461
src/kernels/verify.py	127	6	95%	46, 202-204, 318-319
TOTAL	2243	288	87%

Updated by the Test kernels workflow on commit 4bb1309648d3dd26b6106dca4b6643ea2c3d6a8e.

sayakpaul

Most of my comments are just appreciation.

Some high-level comments:

How much overhead are we talking about? Depending on that, I wonder if users should have knob to control it (true, by default).
I think we should document the validation parts a little bit even if we consider the feature to be experimental (doesn't need to be this PR).
Is there any merit to parallelize the expensive verification bit? Or the side-effects would be too crazy?

There was one more wrinkle in that validating the hash of all kernel files on each load is fairly expensive. So upon the first successful verification, we write a receipt that attests that an earlier verification is done. The receipt is content-addressed by the repo id, the resolved Git commit SHA and build variant, so it does not attest for newly-downloaded revisions or build variants.

Should we write that receipt in a "kernels" cache? However, what happens if the version we're loading was updated in between two loads? Since Git commit SHA is part of the content address, the earlier receipt should already be invalidated in that case? Do we do the verification from scratch in that case or do we utilize part of the earlier receipt to determine the new stuff we need to verify and only focus on those?

sayakpaul · 2026-09-18T06:57:00Z

+use thiserror::Error;

 use crate::git::Oid;
+use crate::hf::{UnknownCacheDir, kernels_cache};


I think we should document kernels cache a bit more, showing how users can configure it, etc. Okay if that is done in a separate PR.

sayakpaul · 2026-09-18T07:16:08Z

+    def validate_kernel(self: "DepTreeNode[LocalKernel]", validator: KernelValidator):
+        validator.validate_kernel(kernel=self.location)
+
+        for node in self.deps.values():
+            node.validate_kernel(validator)


Nice. I like the segregation.

sayakpaul · 2026-09-18T07:18:04Z

    )
    tree.validate_metadata(metadata_validator)
    tree_only_local = tree.install(api=api)
+    tree_only_local.validate_kernel(kernel_validator)


sayakpaul · 2026-09-18T07:27:13Z

+        match result:
+            case VerificationResult.Success():
+                logger.debug(f"{kernel_str}: {result}")
+            case VerificationResult.Failure():
+                logger.warning(f"{kernel_str}: {result}", stacklevel=3)
+            case _ as unreachable:
+                assert_never(unreachable)


sayakpaul · 2026-09-18T08:26:44Z

+    return sorted(p.name for p in refs.iterdir()) if refs.is_dir() else []
+
+
+@pytest.mark.parametrize("version", [KernelVersion.Revision("v1"), KernelVersion.Version(1)])


sayakpaul · 2026-09-18T08:27:56Z

+
+    install_kernel("kernels-community/relu", revision="v1", backend="cpu")
+
+    assert install_kernel("kernels-community/relu", revision="v1", backend="cpu", local_files_only=True)


Should we assert against the path returned by install_kernel() rather than a bare assert?

sayakpaul · 2026-09-18T08:28:04Z

+
+    install_kernel("kernels-community/relu", version=1, backend="cpu")
+
+    assert install_kernel("kernels-community/relu", version=1, backend="cpu", local_files_only=True)


Same as above.

danieldk force-pushed the kernel-validator-signature branch from b0de845 to 7fbf0fd Compare September 15, 2026 10:15

danieldk added 2 commits September 15, 2026 13:01

nix-builder: update get-kernel-check-hook

4bb1309

danieldk marked this pull request as ready for review September 15, 2026 15:00

sayakpaul reviewed Sep 18, 2026

View reviewed changes

Navigation Menu

kernels: verify the signature of a kernel while loading - #828

kernels: verify the signature of a kernel while loading#828
danieldk wants to merge 3 commits into
mainfrom
kernel-validator-signature

danieldk commented Sep 15, 2026 •

edited

Loading

Uh oh!

HuggingFaceDocBuilderDev commented Sep 15, 2026

Uh oh!

github-actions Bot commented Sep 15, 2026

Uh oh!

sayakpaul left a comment

Uh oh!

sayakpaul Sep 18, 2026

Uh oh!

sayakpaul Sep 18, 2026

Uh oh!

sayakpaul Sep 18, 2026

Uh oh!

sayakpaul Sep 18, 2026

Uh oh!

sayakpaul Sep 18, 2026

Uh oh!

sayakpaul Sep 18, 2026

Uh oh!

sayakpaul Sep 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		return sorted(p.name for p in refs.iterdir()) if refs.is_dir() else []


		@pytest.mark.parametrize("version", [KernelVersion.Revision("v1"), KernelVersion.Version(1)])


		install_kernel("kernels-community/relu", revision="v1", backend="cpu")

		assert install_kernel("kernels-community/relu", revision="v1", backend="cpu", local_files_only=True)

Conversation

danieldk commented Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HuggingFaceDocBuilderDev commented Sep 15, 2026

Uh oh!

github-actions Bot commented Sep 15, 2026

Coverage report — kernels/

Uh oh!

sayakpaul left a comment

Choose a reason for hiding this comment

Uh oh!

sayakpaul Sep 18, 2026

Choose a reason for hiding this comment

Uh oh!

sayakpaul Sep 18, 2026

Choose a reason for hiding this comment

Uh oh!

sayakpaul Sep 18, 2026

Choose a reason for hiding this comment

Uh oh!

sayakpaul Sep 18, 2026

Choose a reason for hiding this comment

Uh oh!

sayakpaul Sep 18, 2026

Choose a reason for hiding this comment

Uh oh!

sayakpaul Sep 18, 2026

Choose a reason for hiding this comment

Uh oh!

sayakpaul Sep 18, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

danieldk commented Sep 15, 2026 •

edited

Loading

Coverage report — `kernels/`