Hash modulo meta

[roberth]

Motivation

Make SBOM and similar analyses easier and complete.

As for choosing this implementation over out of band solutions like DetNix's, this keeps the data model simpler; see commit message.

Solve the metadata tracking problem entirely within the slack that already exists in input addressing.
(CA could handle it with the same transformation before trace lookup, or lean on build reproducibility + early cutoff)

A change like this is also a prerequisite for fixing meta.timeout handling properly.

Example

$ nix run github:roberth/nix/hash-modulo-meta -- \
  build --store ~/stores/tmp \
    --extra-experimental-features derivation-meta \
    --impure --expr \
    'with import (builtins.fetchTarball { url = "https://github.com/roberth/nixpkgs/archive/derivation-meta.tar.gz"; }) { config.derivationMeta = true; }; python3'
...
<progress bar, substituting from cache.nixos.org>
...

$ nix run github:roberth/nix/hash-modulo-meta -- derivation show --store ~/stores/tmp --extra-experimental-features derivation-meta --impure --expr 'with import (builtins.fetchTarball { url = "https://github.com/roberth/nixpkgs/archive/derivation-meta.tar.gz"; }) { config.derivationMeta = true; }; python3'
{
  "vcllzyc6d87wxlncjcwxlgy82lna4h4v-python3-3.13.9.drv": {
    "args": ...
    ...
    "meta": {
      "available": true,
      "broken": false,
      "changelog": "https://docs.python.org/release/3.13.9/whatsnew/changelog.html",
...
  }
}

Context

• Nixpkgs counterpart: pkgs: Add config.derivationMeta nixpkgs#466932
• Rebase of Hash modulo meta #14686 (new PR because the intervening refactors make it more like a rewrite)
  • Unblocked by libstore: compute the hash modulo via an intermediate derivation #16191
• Step towards correctly implementing meta.timeout
• Implements, closes Have our cake and eat it too derivation metadata #10780
• Incremental change on topic of New derivation-making builtin #9774
• Closes perhaps Provide SBOM information in an "queryable" way #13564
• Possible solution to More context in context string #4677

Docs suggestions
Co-authored-by: Valentin Gagarin <valentin@gagarin.work>

Assisted-by: Claude Code (unknown model version)
Assisted-by: Claude Code (Claude Opus 4.7)
Assisted-by: Claude Code (Claude Opus 5)

---

Add 👍 to pull requests you find important.

The Nix maintainer team uses a GitHub project board to schedule and track reviews.

[arianvp]

Does this replace the derivationWithMeta construct in the DS fork?

[roberth]

DS could "rebase" their primop to use the same mechanism, and/or make their existing provenance feature listen to the __meta field and the required system feature. I'd recommend them to follow this PR's representation as the preferred source of truth, because it is more accurate.

I like the idea of a cleaner primop, but I'd rather make a mess out of derivation so that we don't have to litter builtins with a new primop whenever anyone wants to extend its meaning.
We're tracking a potential future primop cleanup at

• New derivation-making builtin #9774

[xokdvium]

I do have to ask:

What the plan with this feature? Ok, suppose we do preserve meta in the drv but exclude it from affecting the input address calculations.

The next steps would be to trace the output path back to the deriver if you want to operate on outpaths. I imagine one would want to be able to know the deriver during the build and such, but I'm not sure we have a way to query the deriver in any way now that allows to get it into the input paths of the build sandbox so that you could "reflect" on it?

And well, the deriver is quite often not available when things come from the binary cache too.

[xokdvium]

Another concern is probably that if you really do care about SBOMs then there's an argument to not remove meta from outPath hashing at all, because stuff with a different metadata (i.e. licences - imagine a case where nixpkgs carried a wrong license and that bug got fixed for a package), but the by avoiding it in the input address same packages with different licenses resolve to the same address (arguably that's already the case with the current state of things where meta is completely excluded from the derivation attrs).

But in general I think it's important that we are not punting on the deriver link to walk back from our paths. Otherwise introspection on store paths that you got from a cache and whatnot isn't really solved and has no avenue to be solved. Also because deriving relation isn't injective we'll always have an ambiguity when it comes to tracing from paths back to their origins.

In that regard DS approach isn't all that bad because somewhere you really do need to store this information to break the ambiguity.

[xokdvium]

In part this relation is recorded for "resolved" drvs in build trace for CA, but it also has the issue that walking back to the unresolved drv is impossible without also having the original drvs locally.

@Ericson2314, could you go a bit into what you had in mind for better input addressing that solves the provenance issues?

[roberth]

What the plan with this feature?

Provide a robust way to identify a derivation and its metadata by means of the derivation hash, which is already non-unique wrt outputs.

In terms of practical execution, I'd say experimental until end of year or so. That should be enough time to find unknown unknowns and be able to back out or make breaking changes.
There's plenty of interest around SBOMs, so I'd expect it to be exercised well enough.
Also @yyovil wants to use this as part of his GSoC project, which is still ongoing.

Ok, suppose we do preserve meta in the drv but exclude it from affecting the input address calculations.

That is exactly what this does.

The next steps would be to trace the output path back to the deriver if you want to operate on outpaths.

Not really. I wouldn't recommend throwing about the real deriver in favor of a questionable supposedly-first deriver field.
Outpaths are not unique, and any system that relies exclusively on those, even implicitly, is flawed.
Even more flawed if you were to use floating outputs (CA).

But in general I think it's important that we are not punting on the deriver link to walk back from our paths.

I think the factorization is important.
The deriving link is its own problem in need of a better provenance solution than that one questionable field.

I suppose the alternative would look like upstreaming the DS provenance work, and then also hooking provenance into the scheduler for meta.timeout?
I guess you could then also remove FODs from inputDrvs and let the acquisition of them in inputSrcs be scheduled based on the provenance system too.
You could take it in that direction, but I like my drvs complete.

Otherwise introspection on store paths that you got from a cache and whatnot isn't really solved and has no avenue to be solved.

I'm not saying we shouldn't improve provenance at all. Both build traces and an expression-level thing will be good additions that address this problem.

In that regard DS approach isn't all that bad because somewhere you really do need to store this information to break the ambiguity.

They can't accurate correlate drv path and metadata; there they inherit the deriver problem unconditionally, whereas with this PR you only get the deriver problem if you've lost your drvPath.
If they link it directly to a flakeref, then yes that sidesteps the problem, but it's denormalized and slightly wasteful, and still doesn't answer the question based on an output path.

[xokdvium]

Provide a robust way to identify a derivation and its metadata by means of the derivation hash, which is already non-unique wrt outputs.

Ok then I see, this is much smaller scope and doesn't help us with outPath based provenance (and not intended to).

I think the factorization is important.
The deriving link is its own problem in need of a better provenance solution than that one questionable field.

True.

whereas with this PR you only get the deriver problem if you've lost your drvPath.

Do I get it right that the idea is to manually propagate/remember (in the derivations themselves) the drvPath as well at the original derivation, so that the deriver problem isn't necessary to solve at all? Do we have an easy way for a derivation to reference its own drvPath?

[roberth]

Do I get it right that the idea is to manually propagate/remember (in the derivations themselves) the drvPath as well at the original derivation, so that the deriver problem isn't necessary to solve at all?

I think you meant s/drvPath/outPath, so the combination of outPath for the thing itself and drvPath for its metadata.

Yes, I think that is a decent solution.
It gives users the possibility to have two scripts or workflows or whatever.

1. nix build/nix run etc for deployments
2. nix eval foo#bar.drvPath/nix-instantiate/... or an SBOM tool that invokes it for them, for getting the derivation, meta, output analysis, etc.

So yes, this avoids the deriver field and its problems completely.

Working your way back from an output path is useful, but it is a separable problem, not a prerequisite.

Do we have an easy way for a derivation to reference its own drvPath?

I don't think that's desirable, because it defeats early cutoff, but it is possible to, say, generate a file or derivation that lists out a bunch of derivation paths. I don't think it works too well before we put some time into fixing the way unsafeDiscardOutputDependency is handled downstream from the evaluator.
Without solving that, I don't think references to .drvs in outputs can be created properly, but users could work around that by e.g. storing the .drvs in a bucket they don't GC.
It's been a couple of years since I checked unsafeDiscardOutputDependency handling in derivations, so if anyone has worked on that, apologies :)

[Ericson2314]

@Ericson2314, could you go a bit into what you had in mind for better input addressing that solves the provenance issues?

Two things:

• I think post the derivation refactors separating the ATerm and in-memory formats, it will be easier to add new features in general to Derivation in a less hacky way than was previously possible. WIth the ATerm format there is no choice but to "sneak it in there", but we shouldn't let that sneakiness slip in the rest of the situation. That this argument has little to do with the specifics of this feature.

• hash modulo meta is also changing how input-addressing works. So as a matter of stabilization I think it is good if we batch better input addressing and this together. But the PRs can land in either order.

[domenkozar]

ssh:// uses LegacySSHStore, which inherits this true default even though the legacy serve protocol never negotiates featureDerivationMeta. A meta derivation can therefore bypass the compatibility check and later fail with an input-addressed output-path mismatch. Could legacy SSH return false here unless the capability was explicitly negotiated?