Fix fetchToStore caching by edolstra · Pull Request #14050 · NixOS/nix

edolstra · 2025-09-22T21:29:05Z

Motivation

fetchToStore() caching is broken because it uses the fingerprint of the accessor, but now that the accessor (typically storeFS) is a composite (like MountedSourceAccessor or AllowListSourceAccessor), there is no fingerprint anymore.

So fetchToStore() now uses the new getFingerprint() method to get the specific fingerprint for the subpath. We now also mount inputs on top of storeFS to make their fingerprints visible to fetchToStore().

To prevent future regressions like this, this PR also adds tests to make sure that most paths are cacheable.

Context

Mostly cherry-picked from DetSys Nix and #13225.

Add 👍 to pull requests you find important.

The Nix maintainer team uses a GitHub project board to schedule and track reviews.

xokdvium · 2025-09-22T22:24:41Z

+struct MountedSourceAccessor : SourceAccessor
+{
+    virtual void mount(CanonPath mountPoint, ref<SourceAccessor> accessor) = 0;
+
+    /**
+     * Return the accessor mounted on `mountPoint`, or `nullptr` if
+     * there is no such mount point.
+     */
+    virtual std::shared_ptr<SourceAccessor> getMount(CanonPath mountPoint) = 0;


This is great. Like this exactly what's needed for in-memory dummy store.

We'll need to use this with care. E.g. the path ${m}/foo != getMount m foo, so if both are observable in the language, that would be a problem. (And they are substantially different in terms of root and path components)

This shouldn't be a problem here since we don't have lazy trees yet. I.e. identical inputs are deterministically mounted on the same store paths.

xokdvium · 2025-09-22T22:26:20Z

+
+    if (originalInput.getNarHash() && narHash != *originalInput.getNarHash())
+        throw Error(
+            (unsigned int) 102,


Isn't there an enum for this?

Don't think so... There is Worker::failingExitStatus() which should probably be factored out.

I was meaning to clean that up and unify with BuildResult statuses at some point, fwiw.

Ericson2314 · 2025-09-25T03:15:49Z

+{
+    auto storePath = fetchToStore(fetchSettings, *store, accessor, FetchMode::Copy, input.getName());
+
+    allowPath(storePath); // FIXME: should just whitelist the entire virtual store


I would like this part done sooner rather than later

Ericson2314 · 2025-09-25T04:17:29Z

I took the "easy parts" of this and did some other cleanups and put in #14080.

Ericson2314 · 2025-09-25T04:44:26Z

-        auto realStoreDir = dirOf(store->toRealPath(StorePath::dummy));
-        if (settings.pureEval || store->storeDir != realStoreDir) {
-            accessor = settings.pureEval ? storeFS : makeUnionSourceAccessor({accessor, storeFS});
-        }
+        accessor = settings.pureEval ? storeFS.cast<SourceAccessor>() : makeUnionSourceAccessor({accessor, storeFS});


I don't think this is quite right --- evidently we need better unit tests to catch this.

#14080 removes redundant branches while keeping the store->storeDir != realStoreDir fix.

No, this change is necessary. Otherwise, if pureEval isn't enabled, rootFS ends up not containing storeFS, so the mounts on storeFS won't be visible in rootFS.

Mic92 · 2025-09-25T07:21:11Z

@Ericson2314 now this one needs a rebase?

With FilteringSourceAccessor, lstat() needs to throw a different exception if the path is inaccessible than if it doesn't exist.

This returns the fingerprint for a specific subpath. This is intended for "composite" accessors like MountedSourceAccessor, where different subdirectories can have different fingerprints.

Ericson2314 · 2025-09-25T15:25:21Z

I'll rebase it

fetchToStore() caching was broken because it uses the fingerprint of the accessor, but now that the accessor (typically storeFS) is a composite (like MountedSourceAccessor or AllowListSourceAccessor), there was no fingerprint anymore. So fetchToStore now uses the new getFingerprint() method to get the specific fingerprint for the subpath.

This is to test the non-functional property that most paths should be cacheable. We've had frequent cases where caching broken but we didn't notice.

Ericson2314 · 2025-09-25T18:02:01Z

+
+        // Backward compatibility hack: throw an exception if access
+        // to this path is not allowed.
+        if (auto accessor = res.accessor.dynamic_pointer_cast<FilteringSourceAccessor>())
+            accessor->checkAccess(res.path);


This stuff has been on lazy trees for a while I know, but I guess a this point I am skeptical it is still needed? Especially with #14081

No, they're still needed.

Ericson2314 · 2025-09-30T15:19:54Z

So my current plan (as of yesterday) is that after @xokdvium fixes some bugs, I think #14081 will pass tests, and then I hope we can do that, and then I think we will be better prepared to review this. I want avoid a mixture of mounting and filtering in the pure eval case because I think that is hard to reason about.

edolstra · 2025-10-06T09:24:47Z

and then I think we will be better prepared to review this.

Let's please not block pretty important bug fixes on nice-to-have refactorings. Currently the fetchToStore cache is broken on master, which is a pretty major regression.

xokdvium

Looks good. Other nice-to-haves we can improve on in follow-ups

xokdvium · 2025-10-24T00:43:06Z

This seems to have regressed: nix eval --expr "builtins.readDir /nix/store" --impure

nix eval --expr "builtins.readDir /nix/store" --impure
error:
       … while calling the 'readDir' builtin
         at «string»:1:1:
            1| builtins.readDir /nix/store
             | ^

       error: path '/nix/store/' is not in the Nix store

nixos-discourse · 2025-10-24T23:34:11Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nix-2-32-0-released/70528/9

bryango · 2025-12-22T17:02:26Z

This seems to cause nix edit nixpkgs#hello to fail with:

$ nix edit nixpkgs#hello
error: cannot open '«github:NixOS/nixpkgs/af84f9d270d404c17699522fab95bbf928a2d92f»/pkgs/by-name/he/hello/package.nix' in an editor because it has no physical path

Note that nix edit --impure works fine, similar to what was encountered in Regression: Edit command broken without --impure #10369 and nix edit is broken on master #9652. Possibly related: nix eval nixpkgs#path with and without --impure produces very different paths:
```
 $ nix eval nixpkgs#path
 «github:NixOS/nixpkgs/af84f9d270d404c17699522fab95bbf928a2d92f»/
 
 $ nix eval --impure nixpkgs#path
 /nix/store/krhf9igjyi7h8c5mlddxd8fhwv1ljmzm-source
```
This PR is found by bisecting with the hydra cache, which actually leads to the range 5e65584...1e70955 in which this PR seems most suspicious.
The error seems to persist up to the current master; I have reproduced the issue on top of f457245 (again using hydra cache).

Additional information: I have a user registry pin (this is probably unrelated to the issue?)

$ nix registry list | grep nixpkgs
user   flake:nixpkgs github:NixOS/nixpkgs/af84f9d270d404c17699522fab95bbf928a2d92f
global flake:nixpkgs github:NixOS/nixpkgs/nixpkgs-unstable

xokdvium · 2025-12-22T19:58:44Z

This seems to cause nix edit nixpkgs#hello to fail with:

Yeah that in the current form won't work well with anything "lazy tree" shaped. The file really doesn't have a location on disk and the file lives the git packfiles (although it is copied to the store still). I guess what we can do is when the file doesn't have a physical location we copy the contents into a temporary and open that.

roberth · 2025-12-23T12:18:02Z

For some input schemes like github:, you could xdg-open/open in the browser, but many users will prefer to stay in the terminal, so that choice should be directed by a setting.

xokdvium · 2026-01-09T17:24:18Z

Another regression from this: #14961 (though it was already broken with a relocated store when realStoreDir != storeDir)

Since #14050 we now mount over the store paths in the rootFS accessor in the evaluator. This broke :e and nix edit on something like `:e "${outPath}/src/nix-manual/utils.nix"` when having the `github:nixos/nix` flake loaded in a `--pure-eval` repl. `nix edit` was also broken similarly with: `because it has no physical path`. For regular files, we can copy the outputs into a temporary read-only file and open that instead. For directories, we could open a directory listing in the editor, but I haven't implemented that here. Copying the whole directory tree seems a bit wasteful to me, so we can't truthfully mimic the old behavior of `:e` on directories (that editors usually allow the user to traverse when opened). This also implements some infrastructure for getting a writable file descriptor for the file that would get edited - I intend to use it for #15633.

Since NixOS#14050 we now mount over the store paths in the rootFS accessor in the evaluator. This broke :e and nix edit on something like `:e "${outPath}/src/nix-manual/utils.nix"` when having the `github:nixos/nix` flake loaded in a `--pure-eval` repl. `nix edit` was also broken similarly with: `because it has no physical path`. For regular files, we can copy the outputs into a temporary read-only file and open that instead. For directories, we could open a directory listing in the editor, but I haven't implemented that here. Copying the whole directory tree seems a bit wasteful to me, so we can't truthfully mimic the old behavior of `:e` on directories (that editors usually allow the user to traverse when opened). This also implements some infrastructure for getting a writable file descriptor for the file that would get edited - I intend to use it for NixOS#15633.

This builds on top of #14050 to actually make flakes get lazily copied to the store. This repurposes a slightly less lazy (but also more deterministic) approach that determinate nix has taken. We do still pay to cost of hashing an input once to compute the store path and narHash daemon-client-side. This could be improved in follow-ups in case we don't actually need to check the narHash (like during local development). We need certain backwards compatibility hacks for getFlake with a discarded string context, those are similar to what detnix does. See: DeterminateSystems#422 See: DeterminateSystems#402 Co-authored-by: Eelco Dolstra <edolstra@gmail.com>

This builds on top of #14050 to actually make flakes get lazily copied to the store. This repurposes a slightly less lazy (but also more deterministic) approach than determinate nix has taken. We do still pay to cost of hashing an input once to compute the store path and narHash daemon-client-side. This could be improved in follow-ups in case we don't actually need to check the narHash (like during local development). We need certain backwards compatibility hacks for getFlake with a discarded string context, those are similar to what detnix does. See: DeterminateSystems#422 See: DeterminateSystems#402 Co-authored-by: Eelco Dolstra <edolstra@gmail.com>

This builds on top of NixOS#14050 to actually make flakes get lazily copied to the store. This repurposes a slightly less lazy (but also more deterministic) approach than determinate nix has taken. We do still pay to cost of hashing an input once to compute the store path and narHash daemon-client-side. This could be improved in follow-ups in case we don't actually need to check the narHash (like during local development). We need certain backwards compatibility hacks for getFlake with a discarded string context, those are similar to what detnix does. See: DeterminateSystems#422 See: DeterminateSystems#402 Co-authored-by: Eelco Dolstra <edolstra@gmail.com>

github-actions Bot added new-cli Relating to the "nix" command with-tests Issues related to testing. PRs with tests have some priority fetching Networking with the outside (non-Nix) world, input locking labels Sep 22, 2025

edolstra mentioned this pull request Sep 22, 2025

Create a second Store::getFSAccessor for a single store object #14049

Merged

xokdvium reviewed Sep 22, 2025

View reviewed changes

edolstra added this to Nix team Sep 24, 2025

github-project-automation Bot moved this to Triage in Nix team Sep 24, 2025

Ericson2314 reviewed Sep 25, 2025

View reviewed changes

Ericson2314 mentioned this pull request Sep 25, 2025

Some storeFS and similar cleanup #14080

Merged

Ericson2314 reviewed Sep 25, 2025

View reviewed changes

xokdvium mentioned this pull request Sep 25, 2025

libfetchers: avoid re-copying substituted inputs #14041

Merged

getchoo mentioned this pull request Sep 25, 2025

nix copies inputs already in the nix store #11228

Closed

edolstra added 4 commits September 25, 2025 11:20

SourceAccessor: Make lstat() virtual

55c7ef9

With FilteringSourceAccessor, lstat() needs to throw a different exception if the path is inaccessible than if it doesn't exist.

Add SourceAccessor::getFingerprint()

28d11c5

This returns the fingerprint for a specific subpath. This is intended for "composite" accessors like MountedSourceAccessor, where different subdirectories can have different fingerprints.

Git fetcher: Make dirty repos with no commits cacheable

3450a72

Path fetcher: Simplify fingerprint computation

ec6d5c7

edolstra added 2 commits September 25, 2025 11:30

Test against uncacheable paths

4b9735b

This is to test the non-functional property that most paths should be cacheable. We've had frequent cases where caching broken but we didn't notice.

Ericson2314 force-pushed the fix-fetch-to-store-caching branch from 7618094 to 4b9735b Compare September 25, 2025 15:30

Ericson2314 reviewed Sep 25, 2025

View reviewed changes

Merge branch 'master' into fix-fetch-to-store-caching

6721ef5

Mic92 requested a review from Ericson2314 September 26, 2025 18:45

xokdvium approved these changes Oct 6, 2025

View reviewed changes

edolstra merged commit 1e70955 into master Oct 6, 2025
32 checks passed

edolstra deleted the fix-fetch-to-store-caching branch October 6, 2025 17:39

github-project-automation Bot moved this from Triage to Done in Nix team Oct 6, 2025

xokdvium mentioned this pull request Oct 24, 2025

Source path error improvements #12915

Closed

xokdvium mentioned this pull request Dec 9, 2025

Refering to channel: in search path got significantly slower in nix 2.21+ #10946

Closed

xokdvium mentioned this pull request Mar 8, 2026

nix build --store ssh-ng:// fails since 2.29 with /nix/store is not in the Nix store #15418

Closed

2 tasks done

xokdvium mentioned this pull request Apr 14, 2026

edit: Allow opening paths that don't exist on disk in EDITOR #15686

Merged

xokdvium mentioned this pull request Apr 19, 2026

Don't copy flakes to the store unnecessarily (maybe 3rd time's the charm) #15711

Merged

Uh oh!

Conversation

edolstra commented Sep 22, 2025

Motivation

Context

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Ericson2314 commented Sep 25, 2025

Uh oh!

Ericson2314 Sep 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Mic92 commented Sep 25, 2025

Uh oh!

Ericson2314 commented Sep 25, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Ericson2314 commented Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

edolstra commented Oct 6, 2025

Uh oh!

xokdvium left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

xokdvium commented Oct 24, 2025

Uh oh!

nixos-discourse commented Oct 24, 2025

Uh oh!

bryango commented Dec 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

xokdvium commented Dec 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

roberth commented Dec 23, 2025

Uh oh!

xokdvium commented Jan 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Ericson2314 Sep 25, 2025 •

edited

Loading

Ericson2314 commented Sep 30, 2025 •

edited

Loading

xokdvium left a comment •

edited

Loading

bryango commented Dec 22, 2025 •

edited

Loading

xokdvium commented Dec 22, 2025 •

edited

Loading