feat(core): Rich Platform support#6178
Conversation
|
This is implemented completely differently from the design in #4458, what made you go for The first thing I tried was: [workspace]
channels = ["conda-forge"]
platforms = [{ name = "osx-arm64", subdir = "osx-arm64", virtual-packages = ["__osx=14"] }]
# [system-requirements]
# macos = "13.3"
[dependencies]
mlx = "*"Which only works with the system requirements enabled. So I believe this the virtual packages are not hooked up. There is very little for me to test right now as not sure what could work. It might be good to start out with making the |
|
I am aware that this still differs significantly from #4458. You repeatedly asked to see this ASAP, this is the minimum working set I could push to enable you to take a look. Apparently I did fail abysmally at expectation management. Sorry for that. I went with what we decided to put into the lock file: That is the bare-bones implementation we have to deal with -- at least while we stick with lockfile-v7 and Pixi has to interface with that. What is here is the plumbing: You can define, edit and delete rich platforms -- which will internally stay with the internals they have I expect. They we wired up to get passed around and target selection works for them. Satisfiability checks fail when you change platforms. The cli is just for me to debug this plumbing. It needs to change, I am fully aware of that. The |
|
The pixi.toml file seems to trigger a bit too eagerly. I'll take a look next week. |
hunger
force-pushed
the
push-swkuqkqmorpx
branch
2 times, most recently
from
3d42139 to
eb930a7
Compare
|
Forgot to update the docs ;-) |
[workspace]
channels = ["conda-forge"]
platforms = [{ name = "osx-arm64", platform = "osx-arm64", macos="14" }]
[system-requirements]
macos = "13.3"
[dependencies]
mlx = "*"This creates one platform with a different name: Which I think is incorrect, the platforms configuration should "win" over the The platform logic can't deal with the "rich" platforms yet. And I think we shouldn't rename them when we don't rename them. [workspace]
channels = ["conda-forge"]
platforms = ["osx-arm64", "linux-64"]
[system-requirements]
macos = "13.3"
libc = "2.27"
cuda = "11"
[dependencies]
mlx = "*"I believe the platforms should still be named [workspace]
channels = ["conda-forge"]
platforms = [
{ name = "osx-64", platform = "osx-64", macos="10.15" },
{ name = "latest-osx-arm64", platform = "osx-arm64", macos="15" },
{ name = "osx-arm64", platform = "osx-arm64", macos="14" },
"linux-64",
]
[dependencies]
mlx = "*"I'm not sure what is going on here, but somehow I got into a state where this error was shown. I can't reproduce it so maybe check the error and if there could be a code path that ends up in that error state. Because that makes no sense to me. |
|
@ruben-arts: You are hitting a limitation I do not see any good way around: We use TargetSelector a lot, which are serialized as simple strings. The rich platforms have to fit into that model (or at least I thought that would be best), so I must make sure they serialize to a string that does not cause contradictions in the TargetSelector. For this reason family names like Case 1This should have error-ed out on the platform definition (and indeed does so now;-). Case 2This is what happens: The current machine has no CUDA, so it does not match the specification of the platform attached to the workspace and errors out. That is technically correct, but the error message is abysmal and needs improvement. No question about that! And macos machines should not get the cuda virtual package defined in the first place. Update: This is what it will look in after the next push: Update II: Mac is now ignoring __cuda when generating a platfrom based on system-requirements. It still accepts the flag when you add it manually. Case 3I can not test this one: It effects macos only. Update: It should be fixed in the next push. |
hunger
force-pushed
the
push-swkuqkqmorpx
branch
2 times, most recently
from
1fbf69f to
568a026
Compare
Can you please make this default to |
Could you clearify in this info view which platform would be selected as the current platform? |
hunger
force-pushed
the
push-swkuqkqmorpx
branch
2 times, most recently
from
8543f28 to
f2d5455
Compare
|
I am not happy yet with the Then dim out anything not matching your computer? |
|
If I rename a platform it doesn't invalidate the lockfile. |
|
As discussed, could you rewrite the |
Workspace-platform UX reworkFive commits, all building on the rich-platform work that's already on the Commits
1.
|
| Command | Status before today | Status after today |
|---|---|---|
add, remove, task add/remove/alias |
mixed (strict on tasks) | unified via shared resolver |
update, list, tree, search, exec, import |
Platform (subdir) |
PixiPlatformName |
install, reinstall |
no flag | --platform <PixiPlatformName> with cross-target warning |
workspace platform list / show |
two commands | one list with rich output |
Still pending (flagged but not in this batch):
--platformparity onlock,upgrade.--platformonshell-hook(informational).- Migrate
build/publishfrom--target-platform <Platform>to a
PixiPlatformName-aware resolver.
Tests
- 1973 workspace tests pass (
cargo nextest run --workspace). - Clippy clean on all targets.
- All integration-test additions:
platform_renameunit tests (4): identity match, no-rename when target
name is already taken, no-rename for unmatched entries, default-VP
tolerance.- Task integration tests (2):
task add --platform <subdir>auto-declares;
task remove --platform <subdir>does not. pixi workspace platform listPython integration tests covering the new
output: detection header, supported marker, dim/highlight behaviour,
env/feature continuation lines,CONDA_OVERRIDE_CUDA/
PIXI_OVERRIDE_PLATFORMhonoured.resolve_install_platformunit tests (4): unset returnsNone, declared
platform resolves, cross-platform subdir resolves with warning, invalid
name errors.
CLI documentation regenerated via pixi run generate-cli-docs.
I think we should still support |
[workspace]
channels = ["conda-forge"]
description = "A mock project that does ML stuff"
name = "multi-machine"
# The platforms fields can define custom platform features, like `cuda` and `osx` in this example.
# Use the field `name` to easily manage platform specific definitions.
platforms = [
{ name = "cuda-win-64", platform = "win-64", cuda = "12"},
"win-64",
{ name = "cuda-linux-64", platform = "linux-64", cuda = "12" },
"linux-64",
{ platform = "osx-arm64", macos = "13.5" },
]
[tasks]
start = { depends-on = ["train", "test"] }
test = "python test.py"
train = "python train.py"
[dependencies]
ipykernel = "*"
matplotlib-base = "*"
polars = "*"
python = "*"
torchvision = "*"
pytorch = "*"
[target.cuda-win-64.tasks]
test = "python test.py --cuda"
train = "python train.py --cuda"
[target.cuda-linux-64.tasks]
test = "python test.py --cuda"
train = "python train.py --cuda"
[target.osx.dependencies]
mlx = ">=0.16.1,<0.17"
[target.osx.tasks]
test = "python test.py --mlx"
train = "python train.py --mlx"
[target.osx-arm64.dependencies]
mlx = "*"Having this toml, with especially this line: [target.osx-arm64.dependencies]
mlx = "*"It will add the following table after [target.osx-arm64-macos-13-5.dependencies]
mlx = "*" |
|
In the single environment usecase I would leave out the |
|
Right now changing the name in the If we don't sync the lockfile with this name I would use a different identifier for the lockfile platforms. For example changing this: platforms = [
{ name = "cuda-win-64", platform = "win-64", cuda = "12"},
]To this: platforms = [
{ name = "bla", platform = "win-64", cuda = "12"},
]Will keep this value in the lockfile right now: version: 7
platforms:
- name: cuda-win-64
subdir: win-64
virtual-packages:
- __cuda=12
- __win=10.0
- __archspec=0=x86_64 |
|
Blocking issue, we've decided previously that we don't block the run of an environment based on the system requirements but based on the virtual package requirements of the environment. e.g.: [workspace]
channels = ["conda-forge"]
platforms = ["osx-arm64"]
[dependencies]
python = "==3.12.0"
[system-requirements]
macos = "50.0"Even though The current implementation will give this error: Edit: |
|
Here are some more review notes from clanker: Panic in
|
A `--platform` naming a valid workspace platform that the chosen environment does not list now fails with "platform X is not part of environment Y" before any cross-target warning is printed. The warning moves into the install path so it only fires once membership is confirmed.
'pixi workspace platform add' printed "Added ..." even when the platform was already declared. add_platforms now returns the platforms that actually changed (workspace or feature), and the reporter says "Platform X is already present; nothing to do" for the rest. As a side effect, re-adding a platform to a feature no longer appends a duplicate entry to the TOML array.
…slot overrides_from_declared mapped every unrecognized `__name` virtual package to the libc override slot, so a forward-compat escape-hatch entry like `__future_pkg` silently changed the detected libc version. Restrict the catch-all to the libc family (`__glibc`/`__musl`/`__eglibc`); any other raw `__name` now round-trips through TOML but has no detection effect, as the doc comment and the user docs now state.
…m add' 'pixi workspace platform add linux-64 linux-64' silently collapsed the duplicate. It now errors with "platform X was specified more than once on the command line", mirroring the virtual-package dedup path.
Extend the `conda-meta/pixi` marker file with the platform the environment was resolved with and the minimum platform its installed packages actually require. Store each as subdir plus declared virtual packages rather than the platform name, so a synthesised rich-platform name need not be parsed back. Both fields are optional and default-deserialized so marker files written by an older pixi still load without forcing a reinstall.
Read the per-environment conda-meta/pixi marker and surface, for each installed environment, the platform it was resolved for and the minimum platform its installed packages actually require -- alongside the manifest's declared 'Target platforms'. Both appear in the text output and as resolved_platform / minimum_supported_platform in --json, and are omitted when the environment isn't installed. Exposes PlatformData and Environment::installed_platforms from pixi_core to back the new output.
Print a one-line 'Installed for: <platform>' note after the environment header in 'pixi list' and 'pixi tree', read from the conda-meta/pixi marker, with an emulation hint when the installed subdir differs from the host. The note is skipped when the environment isn't installed and is not emitted in 'pixi list --json'.
Accept a '--platform/-p' override on 'pixi run', mirroring 'pixi install': it pins which declared platform the environment is installed and activated for, and is threaded into the on-demand prefix install via the lock file's target_platform. A '--platform' the environment doesn't list reports the same clear membership error as install, and task resolution uses the pinned platform.
…ost_platform
Replace the scattered trio of current_platform_with_override(),
detect_system_virtual_packages(), and Environment::host_platform() with a
single Workspace::host_platform(source, overrides) method that is the sole
source of truth for what pixi treats as this machine's host platform.
Two orthogonal enums control the result:
- PlatformSource::{Defaults, AutoDetected} -- pixi's per-subdir defaults
(deterministic, used at solve time) vs. the virtual packages actually
detected on the machine.
- PlatformOverrides::{NoOverrides, EnvironmentVariableOverrides} -- whether
PIXI_OVERRIDE_PLATFORM and CONDA_OVERRIDE_* are honoured.
apply_environment_variable_overrides() layers CONDA_OVERRIDE_* on top of
any base, matching upstream rattler semantics: unset keeps, non-empty
replaces, empty disables.
Also:
- Rename best_platform -> best_declared_platform: it now returns None for
workspaces with no declared platforms instead of synthesising a fallback,
removing the fallback_platform OnceCell field from Workspace entirely.
Callers that need a host fallback call workspace.host_platform() explicitly.
- Rename pinned_platform -> named_or_best_declared_platform and simplify
its body to use platform_by_name(), correctly handling empty env platform
sets (meaning 'all platforms') as a no-restriction case.
- Remove the duplicate current_platform_with_override() from pixi_cli.
- Migrate the fallback stanzas in activation.rs and lock_file/resolve/pypi.rs.
- Migrate tool-platform detection in Workspace::to_command_dispatcher.
rebuild_with_renames copied each LockedPackage raw into a fresh builder, carrying PackageHandle indices that pointed into the old lock file's package table. Packages only reachable through a source record's build_packages / host_packages are in no environment list, so they were never registered in the new table: the serialized lock file referenced them by URL but lacked their records, and failed its own reparse with MissingPackage. Both rename passes hit this -- shorten_platform_names corrupted every save of a rich-platform workspace and align_platform_names dropped the packages in memory on every load. Route conda packages through LockFileResolver + into_conda_package_data instead, mirroring filter_lock_file, so build/host subtrees are re-registered against the new builder's package table.
…platforms compute_minimal_required_platforms silently dropped specs without a version (e.g. tensorflow's bare `__cuda`), so machines lacking the package entirely looked minimum-compatible while validate_system_meets_environment_requirements rejected them. Treat a bare spec as version 0, render it as "(any version)" instead of ">= 0", and suggest a realistic CONDA_OVERRIDE_* value.
apply_environment_variable_overrides only rewrote packages the machine already provides, so CONDA_OVERRIDE_CUDA on a machine without a GPU was ignored. Match rattler, which runs every family's detector regardless of the host OS, by adding overridden packages that detection missed.
'pixi task list' claimed tasks from environments whose resolved dependencies looked compatible, while 'pixi run' could not even find them: the task search filtered every environment through the default environment's platform, and the run/install gates rejected any environment whose declared platforms don't match the machine before the lock-based minimum check could apply. A task that was filtered out fell through to shell execution and reported 'command not found'. Make the paths agree: - Search each environment with its own platform (installed marker, best declared, first declared); '--platform' still pins strictly. - Error out, naming the defining environments, instead of falling back to a shell command when the name matches a task anywhere. - Let run/install fall back to a declared platform whose lock-resolved minimum requirements the machine meets; verify_run_platform then warns once (runs "by accident") or errors from conda-meta/pixi. - Annotate every task in 'pixi task list' with how this machine runs its environment: "(by design)" when the resolved platform's requirements are met, "(by accident)" when only the minimum is.
A source build for a foreign '--platform' fails with rattler-build's bare 'Script failed to execute', leaving the real cause invisible. When the build target differs from the machine, add a help text naming both platforms and suggesting to retry without '--platform' or install on a matching machine.
VirtualPackageNotFoundError and UnsupportedPlatformError each carried their own virtual-package-name to CONDA_OVERRIDE_* mapping with the same hardcoded example versions. Share one helper that prefers the actually-required version and falls back to a realistic example for version-less requirements. This fixes the old TODO in VirtualPackageNotFoundError: '__glibc >=2.28' now suggests CONDA_OVERRIDE_GLIBC=2.28 instead of always 2.17, and gains hints for __linux, __win, and __archspec.
Lock-file platform names come from rattler's permissive PlatformName but were converted to the stricter PixiPlatformName with `.expect(...)`, so a foreign or hand-edited lock file (e.g. a platform named `linux`, or one containing an underscore) crashed pixi at the boundary. Handle the mismatch per call site: satisfiability and the update path skip such names (the platform is re-solved), the lock-file rebuild keeps their packages verbatim instead of dropping them, and the diff keys on the raw name so foreign platforms still appear. Add a pixi_diff regression test.
Rename the workspace-platform `libc` friendly key to `glibc` (it always mapped to `__glibc`; the old name implied a generic libc) across the TOML model, the `pixi workspace platform` CLI flag, the JSON schema, docs, and tests. Make the libc family behave correctly when a user configures musl/eglibc, which sit beyond CEP-30's glibc-only standard: - `CONDA_OVERRIDE_GLIBC` now governs glibc alone: unset leaves libc packages untouched, an empty value removes `__glibc`, and a version pins `__glibc=<version>=0` and displaces `__musl`/`__eglibc`. It no longer rewrites or drops non-glibc families through rattler's glibc-only slot. - Run-time verification accepts `__musl`/`__eglibc` instead of silently skipping them, so a configured musl requirement is actually checked. - The detected-package display restores the declared libc family rather than showing the glibc label rattler's override slot forces. The subdir-default merge already suppresses the default `__glibc` when a libc family is declared; this adds the missing test coverage for that.
`PixiPlatform`'s `Eq`/`Hash` are by name only, so two platform entries that resolve to the same name silently collapsed into the name-keyed `IndexSet`, keeping whichever appeared first. Deserialize the entries with per-entry spans and emit a `DuplicateKey` error pointing at both occurrences instead.
The derived `Deserialize` (transparent over `String`) accepted any value - empty, reserved family names, malformed - bypassing the validation every other construction path enforces via `TryFrom`. `pixi_api` deserializes `Vec<PixiPlatformName>` from external input, so replace the derive with a hand-written impl that routes through `TryFrom` and rejects invalid names.
After renaming the platform key to `glibc`, three docs still referenced the old `libc` friendly key / a nonexistent `__libc` package in prose mappings and a CLI example; update them to `glibc`/`__glibc`. Also fix doc-comment merge artifacts: an orphaned first line and a sentence truncated mid-word on `print_autodetected_host`, and a marker-file paragraph that documented `installed_platform_data` but sat on `install_platform`.
`store_credentials_from_project` looked up each `env.platforms()` name with
`.expect("Internal error: Unknown platform used")`. A feature whose declared
`platforms` list outlives a workspace platform removal (a dangling reference)
yields a name absent from `workspace.platforms`, panicking the lookup.
Drop the panicking helper and pass the `Option` through to
`combined_dependencies`, matching the sibling
`extract_git_requirements_from_workspace`.
Unify the three divergent virtual-package renderings (pixi info, pixi workspace platform list text, and its --json) onto a single `inline_virtual_package_specs(declared, baseline)`: friendly key when one exists (`cuda`, `glibc`, ...), raw `__name` otherwise, and a bare key for a version-0 entry. The optional `baseline` filters the subdir defaults -- passed for declared platforms, omitted for the computed resolved/minimum sets.
`add_task` and `alias_task` duplicated the resolve-platform + auto-declare + `add_task` block verbatim. Lift it into `declare_platform_and_add_task` so the platform-resolution rules stay in lock-step by construction; the only difference (the feature the task lands on) is a parameter.
`get_macos_platform_tags` accepted a single-segment `__osx` version (`15` -> `15.0`) but `get_pypi_platform_from_virtual_packages` errored on it. Lift the fallback into a shared `macos_major_minor` helper used by both, so a major-only macOS version resolves the same on the tag path and the validation path.
Polish pass over the rich-platforms branch following review: - document: stop creating an empty `[feature.X]` table as a side effect of removing `[system-requirements]` from a feature that has no table - platform CLI: reject a bare `__` as a virtual-package spec - virtual_packages: rewrite the archspec if/else as an early return - environment: sort references (not clones) in `virtual_packages_match`, and expose `installed_resolved_platform` to avoid a second name-based lookup - features_ext: replace the per-feature `Box<dyn Iterator>` with `itertools::Either` - task search: drop the redundant double platform lookup - use `temp_env::with_var` instead of unsafe `set_var` in the platform test - drop a speculative TODO and dead test scaffolding Tests: - platform axis of the `UnrunnableTask` guard - host-only (not just build-only) package preservation across platform rename - `run --platform` membership error fires before any solve - `VirtualPackageArgs::into_specs` subdir-family validation
Embarassing: The most basic thing you could do broke! Fixed now. |
I want to keep all existing But yes, we could keep the old commands for system requirements working normally, but make them error out when rich platforms are used. |
|
@Hofer-Julian: The reordering and also the format-change should be fixes next push. |
`workspace platform edit` rebuilt the whole [workspace].platforms array from a sorted view, which reordered the entries and collapsed the multi-line layout into one line. A single-platform edit now replaces only that entry in place, preserving its surrounding whitespace and the order of the other entries; the full rebuild is kept only for the legacy [system-requirements] migration, where every entry transforms at once.
4 participants
Description
Adds support for declaring workspace platforms as inline tables with a conda subdir plus per-virtual-package shortcut keys. The bare-string form still works and remains the default:
Remove
SystemRequirementsentirely. They get transparently mapped into features. The pixi.toml file stays unchanged -- till someone uses the command line to add a platform to a rich platorm or "enriches" a existing platform. At this point the file gets updated to the new syntax.The
pixi workspace platformcommand group is filled in:add <PLATFORM>...learns six friendly virtual-package flags (--cuda,--archspec,--libc,--linux,--macos,--windows), with--libc/--linux/--macos/--windowseach restricted to their matching subdir family. Trailing__name=version[=build_string]positionals are the raw escape hatch (mirroring the TOML__name = "..."keys).--featurescopes the addition to a feature.edit <NAME>is new — change--subdir, add/upsert virtual packages (friendly flags + trailing__name=valuepositionals), remove (--remove-virtual-package) or clear (--clear-virtual-packages).show [NAME]is new — human or--jsonoutput for one platform,--all, or--current(auto-detect this machine's subdir).removeandlistkeep their existing shape.Lock-file satisfiability now compares both the subdir and the virtual-package set for each workspace platform, and treats the virtual-package list as a set (order-independent). New non-satisfiability fixtures cover changed-subdir and changed-virtual-package; a satisfiability fixture covers virtual-package-permutation invariance.
The workspace JSON schema accepts the new platform-object form and rejects invalid names.
Closes: #4458, #346
How Has This Been Tested?
Unit tests in
pixi_manifestand integration tests intests/integration_python/test_workspace_platform.py, plus manual testing.AI Disclosure
Tools: Claude
Checklist:
schema/model.py.