ADD with an outdated checksum succeeds using outdated content, where a build error should occur

[k0pernikus]

Note: Created with the help of Claude AI

ADD --checksum with an outdated checksum can produce a build that appears to succeed while using
outdated file content. A build error should have occurred, because the content at the URL does not match
the declared checksum.

This happens when the URL is updated and the checksum is not updated with it. The URL is not part of the
cache key, so two URLs sharing a final path segment share a cache entry and the previously cached content
is reused. No request is made and nothing is compared.

Reproducer

https://github.com/k0pernikus/add-cached-file-even-after-content-change

• repro fails on the step Build 2 should invalidate the cache and get file content BRAVO:
  https://github.com/k0pernikus/add-cached-file-even-after-content-change/actions/runs/33623957687
• updated-checksum passes, because it updates the URL and the checksum together:
  https://github.com/k0pernikus/add-cached-file-even-after-content-change/actions/runs/33623957548

Two files with the same basename in different directories, served over python -m http.server:

path	content	sha256
folder_1/same_file	ALPHA	1921b918b15842c7fdb115078e610263fac85f159c1d8e0ecec3d89a0faa4005
folder_2/same_file	BRAVO	8a51c1b8853b568b5ca25570ef461d6f5a8896f64a952364ed61be1a1e99eb00

# syntax=docker/dockerfile:1
FROM scratch
ARG SRC_URL
ARG SRC_SHA256
ADD --checksum=sha256:${SRC_SHA256} ${SRC_URL} /payload

Observed

build	URL	declared digest	/payload	expected
1	folder_1/same_file	ALPHA's	ALPHA	ALPHA
2	folder_2/same_file	ALPHA's (stale)	ALPHA	a failed build
3	folder_2/same_file	BRAVO's	BRAVO	BRAVO

Build 2 logs the reused layer:

#6 [1/1] ADD --checksum=sha256:1921b918... http://127.0.0.1:8099/folder_2/same_file /payload
#6 CACHED

Build 3 shows that URL serves different bytes. Build 2's content therefore came from the cache, not the
server. The HTTP server's access log records no request during build 2.

Mechanism

In source/http/source.go, resolveMetadata calls resolveMetadataStatic, which returns early when a
checksum is declared, without making a request:

if hs.src.Checksum != "" {
    return &Metadata{
        Digest:   hs.src.Checksum,
        Filename: getFileName(hs.src.URL, hs.src.Filename, nil),
    }, nil
}

CacheKey then returns formatCacheKey(md.Filename, md.Digest, md.LastModified). The URL is not among
those inputs. getFileName reduces it to path.Base(u.Path), so folder_1/same_file and
folder_2/same_file both reduce to same_file. LastModified is nil because nothing was fetched.

Environment

Reproduced on two BuildKit versions, both with the docker driver embedded in dockerd:

• GitHub Actions ubuntu-latest: BuildKit v0.20.2, Docker 28.0.4, buildx v0.19.0
• Local: BuildKit v0.32.2, Docker 29.7.2, buildx v0.36.1

linux/amd64, frontend docker/dockerfile:1.

[tonistiigi]

This is expected with the current optimizations. If you define sha256 checksum for your remote payload then this becomes the cache key for the artifact. Then on repeated runs because you have defined that you want to add artifact with a specific checksum and the there already exists such artifact in cache, it is used directly. Otherwise, we would need to access the remote again, but this way the build works completely offline when you have pulled the artifact once. An additional benefit is that if you use mirrors or different remote sources, they don't invalidate your cache, and the storage remains deduplicated.

[thaJeztah]

Perhaps the docs could use some extra information about this to prevent confusion, or is it documented elsewhere? https://docs.docker.com/reference/dockerfile#add---checksum

[k0pernikus]

An additional benefit is that if you use mirrors or different remote sources, they don't invalidate your cache, and the storage remains deduplicated.

Problem is, that the cache is also active on URLs that encode a version number if it is not part of the filename, as that's how this bit me. i.e.: v29/a_file will be cached just the same as as v30/a_file. You cannot really distinguish mirrors from upgrade paths.

[tonistiigi]

Problem is, that the cache is also active on URLs that encode a version number if it is not part of the filename, as that's how this bit me.

Yeah, the downside is that you may not notice such typo mistake immediately, but you will as soon as you run your build once without the previous cache.

The upside is that you get a deduplicated cache, no matter where the blob is pulled from.