[build] Move Android SDK + NDK component downloads from xaprepare to MSBuild#11440
Draft
jonathanpeppers wants to merge 11 commits into
Draft
[build] Move Android SDK + NDK component downloads from xaprepare to MSBuild#11440jonathanpeppers wants to merge 11 commits into
jonathanpeppers wants to merge 11 commits into
Conversation
…SBuild Following the pattern established by PR #11399 (aapt2), migrate three more component groups from xaprepare's Step_Android_SDK_NDK into proper MSBuild targets in src/androidsdk/androidsdk.targets: * commandlinetools-{mac,linux,win}-* -> $(AndroidSdkDirectory)/cmdline-tools/$(CommandLineToolsFolder) * platform-tools_r$(XAPlatformToolsVersion)-{darwin,linux,win} -> $(AndroidSdkDirectory)/platform-tools * build-tools_r$(XABuildToolsVersion)_{macosx,linux,windows} -> $(AndroidSdkDirectory)/build-tools/$(XABuildToolsFolder) Each component group follows the same pattern: * Per-HostOS item with hardcoded SHA-256 (Google manifest only has SHA-1) * _Download<X> target with target batching via Outputs="...%(Identity)" * _Extract<X> target depending on _Download<X>, using UnzipDirectoryChildren from $(BootstrapTasksAssembly) with TaskFactory="TaskHostFactory" Runtime="NET" * Inputs/Outputs on both targets for incremental build skip * source.properties used as the extraction sentinel file The _AcceptAndroidSdkLicenses target now depends on _ExtractCmdlineTools so the project is self-sufficient and does not require xaprepare to install cmdline-tools before sdkmanager is invoked. The build-tools full extract reuses the same cached zips that aapt2.targets downloads, but extracts the entire directory contents (versus just the aapt2 binary that aapt2.targets extracts into $(MicrosoftAndroidSdkOutDir) for the .NET SDK pack). ### Configuration.props Add SHA-256 hash properties for cmdline-tools and platform-tools. The build-tools hashes already existed from PR #11399. Remove the unused XAPlatformToolsPackagePrefix property; the only consumer was the corresponding entry in AndroidToolchain.cs, which is removed in this PR. ### xaprepare cleanup Remove the migrated entries from AndroidToolchain.cs, along with the now- unused cltOsTag/altOsTag/pltOsTag partial-class fields and the XAPlatformToolsPackagePrefix wiring in KnownProperties.cs, Properties.Defaults.cs.in, and xaprepare.targets. The remaining Step_Android_SDK_NDK responsibilities (NDK, cmake, emulator, system-images, ~25 platform APIs, m2repository, docs, source-36) will be migrated in follow-up commits. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add cmake-$(AndroidCmakeVersion)-{darwin,linux,windows}.zip download/extract
targets to src/androidsdk/androidsdk.targets following the same pattern as
cmdline-tools/platform-tools/build-tools. The cmake zips have no top-level
subdirectory, so NoSubdirectory="True" is used.
Remove cmake from xaprepare's AndroidToolchain.cs and drop the now-unused
AndroidCmakeUrlPrefix/AndroidCmakeVersion/AndroidCmakeVersionPath
KnownProperties + Properties.Defaults.cs.in + xaprepare.targets wiring.
Add XACmakeHash{MacOS,Linux,Windows} SHA-256 properties to Configuration.props.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR continues the migration of Android SDK component acquisition from xaprepare’s Step_Android_SDK_NDK into MSBuild-driven targets, centralizing download/extract logic in src/androidsdk/androidsdk.targets and removing now-redundant xaprepare wiring.
Changes:
- Added MSBuild targets to download, SHA-256 validate, extract, and clean
cmdline-tools,platform-tools, fullbuild-tools, andcmake. - Added new SHA-256 hash properties for the migrated components in
Configuration.props. - Removed the corresponding component definitions and property plumbing from xaprepare (toolchain component list, known properties, defaults, and replacement tokens).
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| src/androidsdk/androidsdk.targets | Introduces download/extract/clean MSBuild targets for cmdline-tools, platform-tools, build-tools, and cmake; wires license acceptance to extracted cmdline-tools. |
| Configuration.props | Adds SHA-256 hash properties for cmdline-tools/platform-tools/cmake and removes now-unused platform-tools prefix property. |
| build-tools/xaprepare/xaprepare/xaprepare.targets | Removes replacement tokens for cmake and platform-tools prefix that are no longer needed. |
| build-tools/xaprepare/xaprepare/ConfigAndData/Dependencies/AndroidToolchain.Windows.cs | Removes OS-tag helper fields no longer used after component removal. |
| build-tools/xaprepare/xaprepare/ConfigAndData/Dependencies/AndroidToolchain.MacOS.cs | Removes OS-tag helper fields no longer used after component removal. |
| build-tools/xaprepare/xaprepare/ConfigAndData/Dependencies/AndroidToolchain.Linux.cs | Removes OS-tag helper fields no longer used after component removal. |
| build-tools/xaprepare/xaprepare/ConfigAndData/Dependencies/AndroidToolchain.cs | Removes toolchain components for the migrated SDK pieces (build-tools/cmdline-tools/platform-tools/cmake). |
| build-tools/xaprepare/xaprepare/Application/Properties.Defaults.cs.in | Removes defaults initialization for cmake and platform-tools prefix properties. |
| build-tools/xaprepare/xaprepare/Application/KnownProperties.cs | Removes known-property constants for cmake and platform-tools prefix. |
| string XAPlatformToolsVersion = GetRequiredProperty (KnownProperties.XAPlatformToolsVersion); | ||
| string XAPlatformToolsPackagePrefix = Context.Instance.Properties [KnownProperties.XAPlatformToolsPackagePrefix] ?? String.Empty; | ||
| bool isArm64Apple = Context.Instance.OS.Flavor == "macOS" && RuntimeInformation.OSArchitecture == Architecture.Arm64; | ||
| string EmulatorPkgRevision = GetRequiredProperty (KnownProperties.EmulatorPkgRevision);bool isArm64Apple = Context.Instance.OS.Flavor == "macOS" && RuntimeInformation.OSArchitecture == Architecture.Arm64; |
…download to MSBuild
Migrates the remaining $(AndroidSdkDirectory) components from xaprepare's
`Step_Android_SDK_NDK` into `src/androidsdk/androidsdk.targets`:
* Android emulator (per-HostOS and per-arch on Apple Silicon)
* API 29 system image used by the emulator (per-HostOS, with the
arm64-v8a variant on Apple Silicon)
* 25 platform APIs (android-2.3.3..platform-37.0_r01)
* `android_m2repository_r47`, `docs-24_r01`, `source-36_r01`
Refactors the per-component download/extract pattern (previously four
near-identical 30-line target pairs) into a single `_AndroidSdkPackage`
ItemGroup with metadata (`Hash`, `Destination`, `NoSubdirectory`, `Url`,
`GeneratePackageXml`) batched through:
* `_DownloadAndroidSdkPackages` - downloads the zip, validates SHA-256,
deletes + errors on mismatch.
* `_ExtractAndroidSdkPackages` - unzips with `UnzipDirectoryChildren`,
using `source.properties` as the per-component incremental sentinel.
* `_GenerateAndroidPackageXmls` - calls the new
`GenerateAndroidPackageXml` BootstrapTask (the MSBuild port of
`Step_Android_SDK_NDK.WritePackageXmls`) to write the synthetic
`emulator/package.xml`.
Adds a `$(AndroidSdkPlatforms)` MSBuild property (default `latest`,
accepts `all` or a comma-separated list of API levels) that mirrors
xaprepare's `--android-sdk-platforms` flag. Platform items are filtered
into `_AndroidSdkPackage` by a `BeforeTargets` target (`%()` metadata
cannot be used in conditions on `Include="@(... -> ...)"` transforms).
Adds 14 SHA-256 hash properties to `Configuration.props` for the new
zips; Google's manifest only ships SHA-1 which `<GetFileHash/>` does
not support.
`AndroidToolchain.cs` now contains only the NDK entry; the per-OS
partials retain only `osTag`. Everything else under `$(AndroidSdkDirectory)`
is handled by MSBuild.
Validated end-to-end on Windows: 30 components, 51,483 files,
6.087 GB extract from a cold cache; `emulator/package.xml` generated
with the correct schema and `EmulatorPkgRevision=36.4.10`.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
jonathanpeppers
changed the title
Now that all $(AndroidSdkDirectory) components are downloaded by src/androidsdk/androidsdk.targets, the supporting code in xaprepare is dead and can be removed: * AndroidPlatformComponent class * Context.AndroidSdkPlatforms and Main.ParseAndroidSdkPlatformLevels * --android-sdk-platforms CLI flag * Step_Android_SDK_NDK.WritePackageXmls / ReadSourceProperties / GetRevision / XNamespace constants (replaced by the GenerateAndroidPackageXml BootstrapTask) * ShouldInstall (AndroidPlatformComponent ...) and the corresponding branch in Check () setup-test-environment-steps.yaml now passes the platform list to androidsdk.csproj via -p:AndroidSdkPlatforms= instead of to xaprepare. manifest-attribute-codegen/README.md updated to match. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds the Android NDK as another _AndroidSdkPackage row in
src/androidsdk/androidsdk.targets and ports
Step_Android_SDK_NDK.CopyRedistributableFiles to a new
CopyNdkRedistributables BootstrapTask. NDK version info
(BuildInfo.NDKRevision, etc.) is now computed from the
BuildAndroidPlatforms.AndroidNdkPkgRevision constant
instead of reading the NDK's source.properties at runtime.
Deletes (now unused) from xaprepare:
- `Step_Android_SDK_NDK.cs`
- `AndroidToolchain.cs` + `.Linux`/`.MacOS`/`.Windows` partials
- `AndroidToolchainComponent.cs`
- `RefreshableComponent.cs`
- `BuildInfo.GatherNDKInfo`, `NDKVersionTag`, `NDKMinimumApiAvailable`
- `Configurables.Defaults.{AndroidToolchainPrefixes,AbiToRID,AbiToClangArch}`
- `Configurables.Paths.{AndroidToolchainRootDirectory,AndroidClangRootDirectory,AndroidToolchainBinDirectory,AndroidToolchainSysrootLibDirectory}`
- `Configurables.Paths.NdkToolchainOSTag` (per-OS partials)
- `Context.ComponentsToRefresh`
- `Main.ParseRefreshableComponents`, `--refresh` CLI flag
- `Scenario_*.AndroidSdkNdkType`
Net: ~66 added, ~672 removed.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
jonathanpeppers
changed the title
The NDK zip contains many symlinks (e.g. `clang++ -> clang`, `x86_64-linux-android24-clang -> clang`) that LibZipSharp's UnzipDirectoryChildren can't handle: on Linux it crashes with `Mono.Unix.UnixIOException: Operation not permitted [EPERM]`, on macOS it silently materializes them as zero-byte files so `clang++` later fails with `posix_spawn: Exec format error`. xaprepare worked around this by using SevenZipRunner for the NDK; the simplest equivalent here is to shell out to system `unzip` on Unix. The Windows NDK zip uses `.exe` files (no symlinks) so UnzipDirectoryChildren continues to handle it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Match the approach already used by `src/openjdk/openjdk.targets`: shell out to system `tar` (bsdtar / libarchive) for all SDK component extraction. `tar` is available on Windows 10+, Linux, and macOS, handles `.zip` natively, and preserves symlinks correctly. This replaces both the `UnzipDirectoryChildren` BootstrapTask (which can not handle the NDK's many symlinks on Unix) and the short-lived `unzip` + `mv` workaround added in the previous commit. One code path now covers all platforms and all packages. The per-package metadata also switches from `NoSubdirectory` to `StripComponents` (defaults to 1, set to 0 for the cmake zip which has no top-level directory) to map directly onto `tar`'s `--strip-components` flag. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The `CopyNdkRedistributables` BootstrapTask was just shelling files into per-ABI directories - no logic that needed C#. Inline it directly into `androidsdk.targets` using target batching over `_NdkAbi` items + a single `<Copy/>` task per ABI. The LLVM major version is read from `AndroidVersion.txt` via a property function (the same `File.ReadAllText` pattern as `openjdk.targets`). Deletes 117 lines of C# in exchange for ~50 lines of MSBuild and one fewer BootstrapTask to maintain / spawn through TaskHostFactory. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The C# task only needed Pkg.Path, Pkg.Desc, and Pkg.Revision -- all of which are already known at MSBuild evaluation time (static strings and `Configuration.props` properties). Replace the BootstrapTask with: * `package.xml.in` -- template with `@PKG_PATH@`, `@PKG_DESC@`, `@PKG_REVISION_MAJOR@`, `@PKG_REVISION_MINOR@`, `@PKG_REVISION_MICRO@` placeholders. * `_GenerateAndroidPackageXmls` -- reads the template via `File.ReadAllText(...).Replace(...)` and splits the revision via `System.Version.Parse(...).Major/.Minor/.Build`, then writes the result via `<WriteLinesToFile>`. Deletes `GenerateAndroidPackageXml.cs` (~77 lines). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The Exec command passes the path directly to the shell, so '\' on macOS/Linux is not a separator. Use System.IO.Path.Combine so MSBuild produces the right separator for the host. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
Continues the effort started in #11399 to move Android SDK component downloads/extraction out of
xaprepareand into a proper MSBuild project (src/androidsdk/androidsdk.csproj). PR #11399 movedbuild-tools; this PR moves everything else — the rest of the SDK and the NDK.What moves
Each component is now a row in the
_AndroidSdkPackageItemGroup insrc/androidsdk/androidsdk.targets, downloaded byDownloadFile(SHA-256 validated against hashes inConfiguration.props) and extracted byUnzipDirectoryChildren:cmdline-tools,platform-tools,build-toolscmakeemulator+ API 29system-image(per-host, per-arch on macOS arm64)platform-*API levels (filtered by$(AndroidSdkPlatforms)— defaults tolatest, acceptsallor comma-separated list)android_m2repository,docs-24,source-36android-ndk-r28c-{linux,darwin,windows}.zip) → extracts to$(AndroidNdkDirectory)New BootstrapTasks
GenerateAndroidPackageXml— writes the minimalpackage.xmlfiles that some component zips (e.g. emulator) omit. Port ofStep_Android_SDK_NDK.WritePackageXmls.CopyNdkRedistributables— copies CRT, libc++, libunwind, and libclang_rt.builtins for every supported ABI from the extracted NDK into$(NativeRuntimeOutputRootDir)$(_RuntimeRedistDirName)/{rid}/. Port ofStep_Android_SDK_NDK.CopyRedistributableFiles.Build flow
setup-test-environment-steps.yamlnow passes-p:AndroidSdkPlatforms=toandroidsdk.csprojinstead of the old--android-sdk-platformsxaprepare flag.BuildInfo.NDKRevision/NDKVersionMajor/NDKVersionMinor/NDKVersionMicroare now computed from theBuildAndroidPlatforms.AndroidNdkPkgRevisionconstant — no longer needs to read the NDK'ssource.propertiesat runtime, soStep_GenerateFilesstill emits the sameXABuildConfigconstants without an installed NDK.xaprepare deletions
Step_Android_SDK_NDK.cs(entire file)AndroidToolchain.cs+.Linux/.MacOS/.WindowspartialsAndroidToolchainComponent.csRefreshableComponent.cs+Context.ComponentsToRefresh+Main.ParseRefreshableComponents+--refreshCLI flagAndroidPlatformComponent.cs+Context.AndroidSdkPlatforms+--android-sdk-platformsCLI flagBuildInfo.GatherNDKInfo(replaced with computed properties)Configurables.Defaults.{AndroidToolchainPrefixes,AbiToRID,AbiToClangArch}Configurables.Paths.{AndroidToolchainRootDirectory,AndroidClangRootDirectory,AndroidToolchainBinDirectory,AndroidToolchainSysrootLibDirectory,NdkToolchainOSTag}Scenario_*.AndroidSdkNdkTypeNet across the 5 commits: ~800 lines removed from xaprepare vs ~hundreds added in MSBuild + the new BootstrapTasks.
Validation
End-to-end test on Windows from a clean cache:
$(AndroidNdkDirectory)$(NativeRuntimeOutputRootDir)$(_RuntimeRedistDirName))Inputs/Outputs)