Pin conda sysroot_linux-64 to a version that exists, and stop deleting committed lockfiles on a failed solve - #2374
Open
cwssemi wants to merge 3 commits into
Open
Conversation
build-setup.sh extracts the pinned sysroot version with
grep -i "sysroot_linux-64=" conda-reqs/chipyard-base.yaml | awk -F= '{print $2}'
which returns everything after the first '=', so on the committed file
DEFAULT_GLIBC is
2.34 # need to be close to system glibc for VCS compatibility
while SYS_GLIBC is a bare version such as 2.35. The two can never compare
equal -- not even on a host whose glibc is exactly the pinned 2.34 -- so the
mismatch branch is taken on every fresh clone: the pin is rewritten and the
lockfiles are regenerated unconditionally. The committed lockfiles are
therefore never what a fresh clone installs, which defeats the purpose of
committing them.
Fixed from both sides, because either alone leaves a trap:
- strip a trailing comment before the compare, so the parser no longer
depends on the line having none;
- move the comment onto its own line, so the intent is visible to the next
person editing the file, and note there that the value is read by grep.
This is the only line in chipyard-base.yaml that is both an inline trailing
comment and read by a shell parser rather than by conda, which is why no
other comment in the file causes trouble.
generate-conda-lockfiles.sh removes each committed lockfile before the solve
that is meant to replace it:
rm -rf $LOCKFILE
conda-lock ... --lockfile $LOCKFILE
Under `set -e` a failed solve therefore leaves the tree missing a file that
is committed to the repository, and the user cannot simply retry: the
fallback the script would have had -- install the committed lockfile -- was
destroyed by the attempt to improve on it. Recovering needs a `git checkout`
the error message gives no hint of.
Both lockfiles are regenerated in one loop, full before lean, so this is not
avoided by --use-lean-conda: the full solve runs first, and a lean setup that
fails there loses the non-lean lockfile it was never going to install.
Generate beside the target instead and move into place only once conda-lock
has succeeded, with a trap to remove the temporary file on any exit. The
temporary name keeps the .conda-lock.yml suffix that conda-lock requires.
Verified with a stubbed conda-lock: on a forced solve failure the committed
lockfiles are untouched and no temporary file is left behind, where the
current script deletes the full lockfile; on success both are replaced.
build-setup.sh writes the host glibc into the sysroot pin verbatim:
sed -i.bak "s/^\([[:space:]]*-\s*sysroot_linux-64=\).*/\1$SYS_GLIBC/" ...
but conda-forge publishes sysroot_linux-64 for only six glibc versions --
2.12, 2.12.2, 2.17, 2.28, 2.34 and 2.39 -- and nothing checks that the value
written is one of them. On a host whose glibc is not published the solve
cannot be satisfied and step 1 fails outright:
The following packages are not available from current channels:
- sysroot_linux-64=2.35
That is Ubuntu 22.04, and also Debian 12 (2.36). It is not a rare corner:
ucb-bar#2241, the issue the adaptation was written for, was reported from Ubuntu
20.04, whose glibc 2.31 is likewise unpublished -- so that reporter is moved
from a build failure to a conda solve failure rather than being fixed.
The sysroot is a build-target floor, not a mirror of the host: it needs to be
<= the host glibc so the result runs, and >= 2.33 for the VCS shared
libraries that motivated the 2.34 pin in ucb-bar#2174. Equality was never required,
and the comment on the pinned line says as much.
Select the greatest published sysroot that is <= the host glibc, and warn
rather than silently proceeding when that falls below the VCS floor. Hosts
that cannot be served at all (glibc older than 2.12, or ldd absent) keep the
committed pin and get a warning instead of having an unusable value written
into the file.
host glibc distro before after
2.31 Ubuntu 20.04 2.31, fails 2.28 + VCS warning
2.34 RHEL/Rocky 9 relock 2.34, no-op
2.35 Ubuntu 22.04 2.35, fails 2.34, no-op
2.36 Debian 12 2.36, fails 2.34
2.39 Ubuntu 24.04 relock 2.39
This builds on the parse fix in the preceding commit and wants to land with
it: without that, DEFAULT_GLIBC still carries the trailing comment, so the
comparison never matches and the lockfiles are regenerated on every clone
even when the clamp selects the value already pinned.
version_le compares dotted versions numerically, so 2.9 sorts below 2.10 and
2.12 below 2.12.2. Every row above was exercised against the real code path
with ldd stubbed, along with glibc 2.11 and an absent ldd.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
changelog:fixed
Please Backport
Related PRs / Issues:
fixes
ucb-bar/chipyard#2373.ucb-bar/chipyard#2271introduced the adaptation,ucb-bar/chipyard#2241is the issue it was written for, anducb-bar/chipyard#2174is the source of both the 2.34 pin and the VCS constraint behind it.Type of change:
Impact:
Contributor Checklist:
mainas the base branch?changelog:<topic>label?changelog:label?.conda-lock.ymlfile if you updated the conda requirements file?Please Backport?CI Help:
Add the following labels to modify the CI for a set of features.
Generally, a label added only affect subsequent changes to the PR (i.e. new commits, force pushing, closing/reopening).
See
ci:*for full list of labels:ci:fpga-deploy- Run FPGA-based E2E testingci:local-fpga-buildbitstream-deploy- Build local FPGA bitstreams for platforms that are releasedci:disable- Disable CISummary
scripts/build-setup.shadapts the condasysroot_linux-64pin to the host, and the adaptation hasthree separable problems. On Ubuntu 22.04 the first one makes a fresh
./build-setup.shfailoutright, and the second deletes a committed lockfile on the way out.
What is wrong
1. The pin is set to a version that may not exist. The script writes the host glibc into the pin
verbatim, and conda-forge publishes
sysroot_linux-64for only six glibc versions — 2.12, 2.12.2,2.17, 2.28, 2.34, 2.39. On a host with glibc 2.35 the solve cannot be satisfied:
That is Ubuntu 22.04, and Debian 12 (2.36) fails the same way. Note this also means
ucb-bar/chipyard#2271does not fix the case it was written for:ucb-bar/chipyard#2241came fromUbuntu 20.04, whose glibc 2.31 is likewise unpublished, so that reporter moves from a build failure
to a solve failure.
2. The guard never guards.
DEFAULT_GLIBCis read withawk -F= '{print $2}', which returnseverything after the first
=— including the trailing comment on the pinned line:SYS_GLIBCis a bare version, so the two never compare equal — not even on a host at exactly thepinned 2.34. The rewrite-and-relock branch is taken on every fresh clone, which means the
committed lockfiles are never what a fresh clone installs.
3. A failed solve deletes a committed file.
generate-conda-lockfiles.shdoesrm -rf $LOCKFILEbefore the
conda-lockthat replaces it, so problem 1 leaves the tree missing a committed lockfileand the user cannot simply retry — the fallback was destroyed by the attempt to improve on it. Both
lockfiles are regenerated in one loop, full first, so
--use-lean-condadoes not avoid it: the leansetup loses the non-lean lockfile it was never going to install.
What this changes
1/3 — read the pin without its trailing comment. Strip
#…before the compare, and move thecomment onto its own line with a note that the value is read by grep. Either alone fixes today; the
parser fix is what stops a re-added comment silently re-arming it.
2/3 — generate lockfiles to a temp path, move on success. With a
trapto remove the temporaryfile on any exit. A failed solve now leaves the committed lockfiles exactly as they were.
3/3 — clamp instead of assign. The sysroot is a build-target floor: it must be ≤ the host
glibc, and ≥ 2.33 for the VCS shared libraries that motivated the 2.34 pin
(
ucb-bar/chipyard#2174) — it never had to equal the host glibc, and the comment on the pinnedline says as much. Select the greatest published sysroot ≤ host glibc, and warn instead of proceeding
silently when that lands below the VCS floor:
A host that cannot be served at all — glibc older than 2.12, or no
ldd— now keeps the committedpin and warns, rather than writing an unusable value into the requirements file.
Version comparison is a small
version_leinscripts/utils.sh, numeric per component so 2.9 sortsbelow 2.10 and 2.12 below 2.12.2.
How it was tested
2.34on the committed file, on the fixed file, and on a filewhere a trailing comment has been re-added.
conda-lock: on a forced solve failure both committedlockfiles are untouched and no temporary file is left behind. The current script, run as a control
in the same harness, deletes the full lockfile.
lddstubbed: every row of the table above, plusglibc 2.11 and an absent
ldd. Unstubbed on a glibc 2.35 host the whole block is now a no-op —no rewrite, no
.bak, no relock — so the committed lockfile installs as authored.content_hash_for_platform("linux-64")computed with conda-lock 2.5.7over the four requirement files is identical before and after this PR, so no lockfile regeneration
is needed. (The same computation reproduces the hash committed in 1.14.0's lockfile from 1.14.0's
requirement files, as a check on the method.)
Not tested: a real conda solve on a 2.31 or 2.39 host — I have neither. The "before" column for
those rows follows from the published sysroot list rather than from a run.
Reproducing problem 1 — one trap worth knowing: it does not reproduce by re-running in the same
tree, because the first run's
sedrewrote the pinned line without its comment, after which thecomparison behaves as intended. Start from
git checkout -- conda-reqs/.