Skip to content

Commit 6597f8e

Browse files
committed
improvement(ci): measure where the layer-cache disk actually goes
The layer cache cap is enforced against `buildctl du`, and on the busiest disk that number no longer describes the disk. In the same job, the app image's amd64 sticky disk reports 224.9 GB while du reads 65.87 GB — so the prune correctly does nothing, being already under its 100 GB cap by its own accounting, and the disk keeps growing. The same Dockerfile's arm64 disk, which builds far less often, reads 88.04 GB against an 86.4 GB disk: no gap at all. The divergence tracks build frequency, not image content. The mount log ("Filesystem free space after mount") shows the filesystem really is holding those bytes, so this is not a thin-provisioned volume failing to release freed blocks — there are real files that du's total does not cover. The `RUN --mount=type=cache` dirs (apt, bun, npm, next, turbo) are the candidates, being a different record class from the layer cache `--keep-storage` trims. Those two causes need opposite fixes, and neither is proven from the logs available today, so this adds the three readings that separate them rather than a speculative fix: df for the filesystem's own view, a du summary by record type for whether cache mounts are accounted at all, and a bounded directory walk to locate the bytes if they are not. All best-effort and guarded. Composite steps run under `bash -e -o pipefail`, where a failing command in an assignment or pipeline aborts the step and fails the build — the bug that broke an app image build in #7252. Verified by running this block under those exact flags with every command failing: it reaches the end and exits 0. Shellcheck clean.
1 parent a5badf0 commit 6597f8e

1 file changed

Lines changed: 30 additions & 0 deletions

File tree

‎.github/actions/docker-build/action.yml‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -163,6 +163,36 @@ runs:
163163
echo "::warning::Layer cache prune failed; this sticky disk is unbounded for this run"
164164
fi
165165
166+
# Why this block exists: the cap above is enforced against `buildctl du`,
167+
# and on the busiest disk that number no longer describes the disk. The
168+
# app image's amd64 sticky disk reports 224.9 GB while du inside the same
169+
# job reads 65.87 GB, so the prune correctly does nothing — it is already
170+
# under a 100 GB cap by its own accounting — and the disk keeps growing.
171+
# The same Dockerfile's arm64 disk, which builds far less often, reads
172+
# 88.04 GB against an 86.4 GB disk: no gap at all. So the divergence
173+
# tracks build frequency, not image content.
174+
#
175+
# The mount log ("Filesystem free space after mount") says the filesystem
176+
# really is holding those bytes, so this is not a thin-provisioned volume
177+
# failing to release freed blocks — there are real files here that du's
178+
# total does not cover. The candidates are the `RUN --mount=type=cache`
179+
# dirs (apt, bun, npm, next, turbo), which are a different record class
180+
# from the layer cache `--keep-storage` trims.
181+
#
182+
# These three readings separate those cases: df is the filesystem's own
183+
# view, the record-type summary shows whether du even accounts for cache
184+
# mounts, and the directory walk locates the bytes if it does not. All are
185+
# best-effort and bounded — this step must never be able to fail a deploy,
186+
# which is why every command is guarded and the walk has a timeout.
187+
echo "--- cache accounting (diagnostic) ---"
188+
df -h /var/lib/buildkit 2>/dev/null || true
189+
echo "du by record type:"
190+
sudo buildctl --addr "$addr" du --verbose 2>/dev/null \
191+
| awk '/^Type:/ { t=$2 } /^Size:/ { s[t] += $2 } END { for (k in s) printf " %-24s %10.2f GB\n", k, s[k]/1e9 }' \
192+
| sort -k2 -rn || true
193+
echo "largest directories under /var/lib/buildkit:"
194+
sudo timeout 120 du -xh -d1 /var/lib/buildkit 2>/dev/null | sort -rh | head -8 || true
195+
166196
- name: Set up Docker Buildx
167197
if: inputs.provider != '' && inputs.provider != 'blacksmith'
168198
uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4

0 commit comments

Comments
 (0)