[build] Provision .NET SDK via standard scripts, drop xaprepare's installer#11636
[build] Provision .NET SDK via standard scripts, drop xaprepare's installer#11636jonathanpeppers wants to merge 10 commits into
Conversation
Replace xaprepare's bespoke `dotnet-install` invocation with Arcade's
standard `eng/common/tools.{sh,ps1}` bootstrap, matching dotnet/sdk,
dotnet/runtime, and dotnet/aspnetcore.
* `global.json`: pin `tools.dotnet` so Arcade's `InitializeDotNetCli`
knows which SDK to install. darc auto-updates this whenever
`Microsoft.NET.Sdk` flows from dotnet/dotnet via the existing
Maestro subscription.
* `eng/install-dotnet.{sh,ps1}`: thin wrappers that set
`DOTNET_INSTALL_DIR=DOTNET_GLOBAL_INSTALL_DIR=bin/$(Configuration)/dotnet/`
(preserving the existing install location) and call
`InitializeDotNetCli` from `eng/common/tools.{sh,ps1}`.
* `Makefile`: `prepare` now depends on a new `install-dotnet` target
that runs `./eng/install-dotnet.sh` first.
* `build-tools/scripts/PrepareWindows.targets`: add an
`_InstallDotNet` target that invokes `eng/install-dotnet.ps1`
before `_BuildXAPrepare`, so `dotnet msbuild Xamarin.Android.sln
-t:Prepare` (used on Windows CI) is self-bootstrapping.
* `Step_InstallDotNetPreview.cs` is deleted and replaced by
`Step_PrepareDotNetWorkloads.cs`. The new step assumes the SDK
is already installed at `bin/$(Configuration)/dotnet/` and only
performs the Android-specific workload prep:
* Cleans stale Mono Android runtime/workload NuGet directories.
* Restores `package-download.proj` (Mono runtime packs +
Mono/Emscripten workload manifest packages).
* Copies the workload manifests into the local SDK's
`sdk-manifests/`.
* Removes obsolete configuration:
* `Configurables.Urls.DotNetInstallScript` (Unix and Windows)
* `--dotnet-sdk-archive` xaprepare option and its
`Context.LocalDotNetSdkArchive` plumbing
* `DownloadDotNetInstallScript`, `GetInstallationScriptArgs`,
`InstallDotNetAsync`, `InstallDotNetFromLocalArchiveAsync`
methods (~150 lines of bespoke install logic).
The SDK install location stays at `bin/$(Configuration)/dotnet/`,
so `dotnet-local.{cmd,sh}` and other consumers continue to work
without changes. CI's `use-dot-net.yaml` is unchanged: it still
provisions a system .NET to bootstrap xaprepare; the pinned preview
SDK install simply moves from xaprepare to Arcade.
Verified locally on Windows: `dotnet msbuild Xamarin.Android.sln
-t:Prepare` after `git clean -xdf bin/Debug/dotnet/` installs the
pinned 11.0.100-preview.5.26268.112 SDK and copies the Mono +
Emscripten workload manifests into `sdk-manifests/`.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
darc does not auto-update global.json:tools.dotnet when Microsoft.NET.Sdk
flows from dotnet/dotnet (verified in arcade-services
DependencyFileManager.cs: only Microsoft.DotNet.Arcade.Sdk, the
*.SharedFramework.Sdk family, Microsoft.DotNet.CMake.Sdk,
Microsoft.NET.Sdk.IL, and the literal name "dotnet" are special-cased).
Pinning the SDK version in global.json would have permanently drifted
from the auto-flowed eng/Versions.props value. Read the version directly
from eng/Versions.props instead, making it the single source of truth.
eng/install-dotnet.{sh,ps1} now download Microsoft's official
dotnet-install.{sh,ps1} from
https://builds.dotnet.microsoft.com/dotnet/scripts/v1/ (cached under
bin/$Configuration/dotnet/) and invoke it with the version parsed from
eng/Versions.props:MicrosoftNETSdkPackageVersion. This bypasses Arcade's
eng/common/tools.{sh,ps1} (which strict-mode-reads $GlobalJson.tools)
and lets us drop the tools.dotnet pin from global.json entirely.
Verified on Windows:
- cold install: ~12s
- warm re-run: ~2.5s (idempotent fast path)
- full Prepare: ~88s
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
CI failed with "Permission denied" when `make jenkins` ran `./eng/install-dotnet.sh` because the file was committed as 100644. The file from `make prepare` is invoked directly (not via `bash`), so it needs the executable bit set. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Reverts the executable-bit change from 2645bdb. Windows clones with core.filemode=false would have shown spurious mode changes when editing the file; running it via `bash ./eng/install-dotnet.sh` from the Makefile sidesteps the bit entirely. Same trick for the cached dotnet-install.sh we download under bin/. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This finishes the second half of the SDK provisioning audit started in PR #11636. The first half moved the .NET SDK install into Microsoft's official `dotnet-install.{sh,ps1}` scripts driven by `eng/install-dotnet`. This commit replaces the leftover xaprepare logic that prepared Android-specific .NET workloads against that SDK. What `Step_PrepareDotNetWorkloads` did (now deleted): * Restored `package-download.proj` to pull down the Mono Android runtime packs and the Mono/Emscripten workload manifest packages. * Copied the workload manifests from the NuGet package cache into the local SDK's `sdk-manifests/` folder. What `src/workloads/workloads.csproj` does (single MSBuild project, no C#, no scenarios): * Carries the same `<PackageDownload>` items that lived in `package-download.proj` (run via NuGet's auto-restore). * Has a `_CopyWorkloadManifests` target that runs `AfterTargets="Build"` and copies each `microsoft.net.workload.{mono,emscripten}.<flavor>` manifest's `data/` into the local SDK's `sdk-manifests/<band>/microsoft.net.workload.<flavor>.<dotnet>/<ver>/`. Per @jonathanpeppers' suggestion in #11636 (comment 3403797084): "move it to like `src/workloads/workloads.csproj` and that project is built first." Wiring: * `Makefile prepare:` now runs `dotnet build src/workloads/workloads.csproj` after the BootstrapTasks build, before `PrepareJavaInterop`. * `build-tools/scripts/PrepareWindows.targets`'s `Prepare` target adds an `<MSBuild Projects=".../workloads.csproj" />` invocation in the same spot. * `build-tools/automation/yaml-templates/setup-test-environment-steps.yaml` no longer invokes xaprepare. Test agents now run `eng/install-dotnet.{sh,ps1}` (provisions the SDK at `bin/$Config/dotnet/`) followed by `dotnet build src/workloads/workloads.csproj` (provisions the workloads against that SDK). This fixes the AndroidTestDependencies CI failure introduced when the prior commit removed `Step_InstallDotNetPreview`'s SDK download. Cleanup: * `Step_PrepareDotNetWorkloads.cs` and `package-download.proj` deleted. * `Scenario_Standard` and `Scenario_AndroidTestDependencies` no longer add `Step_PrepareDotNetWorkloads`. * The `xaprepareScenario` parameter (and the now-unused `run-xaprepare.yaml` template) are removed across all CI YAMLs. * Dead `Configurables.MicrosoftNETWorkload*Dir` properties are removed. Verified locally on Windows: * `bin/Debug/dotnet/sdk-manifests/<band>/microsoft.net.workload.{mono.toolchain,emscripten}.{net6..net10,current}/<ver>/WorkloadManifest.json` is populated after `dotnet build src/workloads/workloads.csproj` (12 manifests total). * Re-running is idempotent (~0.5s warm; `Copy SkipUnchangedFiles="true"`). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
This PR migrates dotnet/android’s .NET SDK provisioning from xaprepare’s custom C# installer to the standard dotnet-install.{sh,ps1} flow, keeping the install location at bin/$Configuration/dotnet/ and moving Android-specific workload prep into a standalone MSBuild project.
Changes:
- Add
eng/install-dotnet.{sh,ps1}wrappers that read the pinned SDK version fromeng/Versions.props, downloaddotnet-install.{sh,ps1}, and install intobin/$Configuration/dotnet. - Wire the new install/workload-prep flow into
Makefile, WindowsPrepareWindows.targets, and CI templates; remove xaprepare’s SDK-install step and related plumbing. - Introduce
src/workloads/workloads.csprojto restore required runtime packs + workload manifest packages and copy manifests into the locally installed SDK.
Reviewed changes
Copilot reviewed 24 out of 24 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| src/Xamarin.Installer.Build.Tasks/README.md | Updates developer instructions to use eng/install-dotnet.* + build workloads project. |
| src/workloads/workloads.csproj | New MSBuild project to restore runtime packs/manifests and copy manifests into the local SDK. |
| Makefile | Adds install-dotnet prerequisite and runs workloads provisioning during prepare. |
| eng/install-dotnet.sh | New Unix bootstrap script to install pinned SDK into bin/$Configuration/dotnet. |
| eng/install-dotnet.ps1 | New Windows bootstrap script to install pinned SDK into bin\$Configuration\dotnet. |
| build-tools/xaprepare/xaprepare/Steps/Step_InstallDotNetPreview.cs | Deletes the bespoke xaprepare SDK installer step. |
| build-tools/xaprepare/xaprepare/Scenarios/Scenario_Standard.cs | Removes SDK install step from the standard xaprepare scenario. |
| build-tools/xaprepare/xaprepare/Scenarios/Scenario_AndroidTestDependencies.cs | Removes SDK install step from Android test dependency scenario. |
| build-tools/xaprepare/xaprepare/package-download.proj | Deletes the old runtime-pack restore project used by xaprepare. |
| build-tools/xaprepare/xaprepare/Main.cs | Removes --dotnet-sdk-archive option plumbing. |
| build-tools/xaprepare/xaprepare/ConfigAndData/Configurables.Windows.cs | Removes DotNet install script URL configurable (no longer needed). |
| build-tools/xaprepare/xaprepare/ConfigAndData/Configurables.Unix.cs | Removes DotNet install script URL configurable (no longer needed). |
| build-tools/xaprepare/xaprepare/ConfigAndData/Configurables.cs | Removes workload manifest path helpers tied to the deleted step. |
| build-tools/xaprepare/xaprepare/Application/Context.cs | Removes LocalDotNetSdkArchive property. |
| build-tools/scripts/PrepareWindows.targets | Ensures SDK install runs before building xaprepare; adds workloads provisioning to Prepare. |
| build-tools/automation/yaml-templates/stage-package-tests.yaml | Removes xaprepare scenario parameter usage from test setup. |
| build-tools/automation/yaml-templates/stage-msbuild-emulator-tests.yaml | Removes xaprepare scenario parameter usage from test setup. |
| build-tools/automation/yaml-templates/setup-test-environment.yaml | Removes xaprepareScenario parameter pass-through. |
| build-tools/automation/yaml-templates/setup-test-environment-steps.yaml | Replaces xaprepare invocation with eng/install-dotnet.* + workloads provisioning. |
| build-tools/automation/yaml-templates/setup-test-environment-public.yaml | Removes xaprepareScenario parameter pass-through. |
| build-tools/automation/yaml-templates/run-xaprepare.yaml | Deletes the shared pipeline template that ran xaprepare. |
| build-tools/automation/yaml-templates/run-emulator-tests.yaml | Removes xaprepare scenario parameter usage from test setup. |
| build-tools/automation/azure-pipelines-public.yaml | Removes xaprepare scenario parameter usage from test environment setup. |
| build-tools/automation/azure-pipelines-nightly.yaml | Removes xaprepare scenario parameter usage from test environment setup. |
* Makefile install-dotnet: pass CONFIGURATION through to install-dotnet.sh so 'make CONFIGURATION=Release prepare' installs the SDK under bin/Release/dotnet to match the rest of the build. * eng/install-dotnet.ps1: null-check the result of SelectSingleNode before dereferencing .InnerText so the script fails with the intended friendly error message if <MicrosoftNETSdkPackageVersion> is ever removed from eng/Versions.props. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The previous commit accidentally captured local submodule pointer changes for external/Java.Interop and external/xamarin-android-tools that have nothing to do with the SDK provisioning audit. Restore them to the pointers used by the rest of this PR (and main). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
/review |
|
✅ Android PR Reviewer completed successfully! |
There was a problem hiding this comment.
🤖 Android PR Review — [build] Provision .NET SDK via standard scripts, drop xaprepare's installer
I reviewed the diff independently before reading the description. This is a well-scoped, largely subtractive change: the bespoke ~220-line Step_InstallDotNetPreview + package-download.proj are replaced by thin eng/install-dotnet.{sh,ps1} bootstrappers and a src/workloads/workloads.csproj (Microsoft.Build.NoTargets) that PackageDownloads the runtime packs / workload manifests and copies the manifests into the locally-installed SDK. Good direction — collapsing toward dotnet-install + dotnet build is a real maintainability win.
Verified OK (potential concerns I checked and dismissed)
- ✅ No dangling references to the removed symbols (
Step_InstallDotNetPreview,DotNetInstallScript, theMicrosoftNETWorkloadMono*Dirconfigurables,package-download.proj). - ✅ Every
xaprepareScenario/run-xaprepare.yamlconsumer was removed — no orphaned YAML parameters that would break pipeline parsing. - ✅ No provisioning silently dropped:
androidsdk.csproj(SDK/JDK) and emulator setup are untouched; the affected scenarios were effectively no-ops apart from the removed step. - ✅ Property/import ordering in
workloads.csprojis fine —DotNetStableTargetFramework,MicrosoftNETCoreAppRefPackageVersion, and the manifest bands resolve viaDirectory.Build.props→eng/Versions.props(auto-imported before the body);XAPackagesDir/DotNetPreviewPathexist by the time the target runs.Microsoft.Build.NoTargetsis pinned inglobal.json. - ✅ Backslash path separators in the copy target normalize correctly on Linux/macOS (verified empirically).
- ✅
Makefilepasses-p:Configuration=$(CONFIGURATION)toprepare-workloads, matching theinstall-dotnetinstall path (bin/$Configuration/dotnet).
Findings (none merge-blocking)
| Sev | Area | Note |
|---|---|---|
| install scripts | A failed/partial download poisons the cached dotnet-install.{sh,ps1} — no temp-then-move, and an empty cached script silently "succeeds". |
|
| workloads | Drops the old forced stale-cache cleanup before copy; possible stale runtime packs/manifests if an internal version string is reused. | |
| 💡 | workloads target | _CopyWorkloadManifests has no Inputs/Outputs and uses AfterTargets="Build". |
| 💡 | formatting | <Error> Condition should come first (Postmortem #33). |
Notes
- 📝 The PR description is slightly stale — it refers to a
Step_PrepareDotNetWorkloads.csreplacement, but the actual change introducessrc/workloads/workloads.csproj(and deletespackage-download.proj). Worth updating so reviewers/git logarchaeologists aren't misled. - ⏳ CI for the head commit (
11dc706) is still in progress (combined statuspending; thedotnet-androidbuild is queued/running). Please confirm it goes green before merging — an earlier commit's legs passed, but the current head hasn't completed.
Nice cleanup overall. 👍
Generated by Android PR Reviewer for issue #11636 · 1.7K AIC · ⌖ 66.1 AIC · ⊞ 37.8K
Comment /review to run again
| # strip it). | ||
| install_script="$install_dir/dotnet-install.sh" | ||
| if [[ ! -f "$install_script" ]]; then | ||
| curl -fsSL "https://builds.dotnet.microsoft.com/dotnet/scripts/v1/dotnet-install.sh" -o "$install_script" |
There was a problem hiding this comment.
🤖 $install_script. On the next run, [[ ! -f "$install_script" ]] is false, so the truncated script is reused — and an empty bash script exits 0, so dotnet-install silently never runs and the SDK isn't installed (the failure then surfaces much later, confusingly). Download to a temp path and mv into place only on success (the removed Step_InstallDotNetPreview.DownloadDotNetInstallScript used exactly this temp-then-copy pattern). eng/install-dotnet.ps1 has the same issue with Invoke-WebRequest -OutFile.
(Rule: Atomic download to avoid a poisoned cache)
There was a problem hiding this comment.
Fixed in 12fa9b8 — both scripts now download to dotnet-install.{sh,ps1}.tmp.$PID and atomically rename into place only on success.
| <Copy | ||
| SourceFiles="@(_MonoManifestSource)" | ||
| DestinationFolder="%(_MonoManifestSource.Dest)" | ||
| SkipUnchangedFiles="true" |
There was a problem hiding this comment.
🤖 Step_InstallDotNetPreview deliberately deleted the microsoft.netcore.app.runtime.mono.android* and Mono workload package dirs from the NuGet cache and rm'd each manifest destination before copying — "as we could possibly be using a new runtime commit with a previously installed version." This flow instead relies on the NuGet cache plus SkipUnchangedFiles="true", so a rolling internal build that reuses a version string with changed content can serve a stale runtime pack / manifest. Please confirm that's intentional now (e.g. darc always bumps the version), or restore the pre-copy clean. (The old step also wrapped the restore in a 3× retry with backoff; NuGet's built-in HTTP retry likely covers it, but worth a glance since CI feed flakiness was the original motivation.)
(Rule: Preserve behavior when replacing logic)
There was a problem hiding this comment.
Intentional — internal builds bump the version string when content changes, and SkipUnchangedFiles="true" handles re-extraction inside the NuGet cache. The 3× retry is similarly delegated: dotnet restore already does HTTP retries, and CI re-runs cover the rare case where a feed is truly flaky for minutes. Happy to add a stamp-based cache bust later if we ever see this hurt.
| <_SdkManifests>$(_DotNetSdk)\sdk-manifests</_SdkManifests> | ||
| </PropertyGroup> | ||
|
|
||
| <Error |
There was a problem hiding this comment.
🤖 💡 Formatting — Put the Condition attribute first on this <Error> element so it's immediately clear when the task fires:
<Error
Condition=" !Exists('$(_DotNetSdk)') "
Text="Expected .NET SDK at '$(_DotNetSdk)' but the directory does not exist. Run eng/install-dotnet.sh (or eng/install-dotnet.ps1) first."
/>(Rule: Condition attribute first (Postmortem #33))
| `sdk-manifests/<band>/microsoft.net.workload.<flavor>.<dotnet>/<ver>/` | ||
| so the SDK can resolve the Android workloads. | ||
| --> | ||
| <Target Name="_CopyWorkloadManifests" AfterTargets="Build"> |
There was a problem hiding this comment.
🤖 💡 MSBuild targets — _CopyWorkloadManifests writes files but has no Inputs/Outputs, so it re-globs and re-runs on every invocation (the SkipUnchangedFiles="true" avoids timestamp churn, but the work still happens). Also, AfterTargets="Build" runs even if Build failed. Consider a stamp file in the intermediate dir (or Inputs/Outputs) so MSBuild can skip it once the manifests are in place. Minor for a prep-only project, but it's the repo convention for file-writing targets.
(Rule: Targets that write files need Inputs/Outputs)
There was a problem hiding this comment.
Skipping for now — SkipUnchangedFiles="true" makes the warm path essentially free (~0.5s and no file writes), and this project is only invoked from make prepare / PrepareWindows / the CI setup template, never from a hot inner loop. Happy to add Inputs/Outputs if we see it cause incremental-build noise.
* eng/install-dotnet.{sh,ps1}: download Microsoft's dotnet-install
script to a temp file and atomically rename into place so a failed
or interrupted download cannot poison the cached script. Restores
the temp-then-move pattern the old Step_InstallDotNetPreview used.
* src/workloads/workloads.csproj: put Condition attribute first on the
<Error> task per repo convention.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
These keep slipping into commits because the local worktree has stale submodule pointers. Restore them to the PR's prior commit. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Context
Today, dotnet/android provisions the .NET SDK with bespoke C# code in
build-tools/xaprepare/xaprepare/Steps/Step_InstallDotNetPreview.cs(220 lines) — fetching
dotnet-install.{sh,ps1}from a hard-coded URL,running it with config-driven args, and supporting a
--with-archiveoverride for offline scenarios. Other .NET repos (
dotnet/sdk,dotnet/runtime,dotnet/aspnetcore) all use Arcade's standardeng/common/dotnet-install.{sh,ps1}flow — there's no reason for us tomaintain a custom one.
Phase 1 of a longer migration
This PR is the SDK-provisioning slice of a larger effort to delete
xaprepare entirely so the build collapses to:
xaprepare today is 333 KB / 116 files but only 4 step files have real
logic (
Step_PrepareDotNetWorkloads,Step_GenerateFiles,Step_GenerateFiles.Windows,Step_GenerateCGManifest). Once each stephas an MSBuild equivalent, the surrounding 332 KB of plumbing
(
Application/,ToolRunners/,OperatingSystems/) can also bedeleted. Follow-up PRs are planned for each remaining step.
What changes here
New:
eng/install-dotnet.{sh,ps1}Thin bootstrap wrappers that:
<MicrosoftNETSdkPackageVersion>fromeng/Versions.props(single source of truth, kept up to date by darc when
Microsoft.NET.Sdkflows from dotnet/dotnet).dotnet-install.{sh,ps1}fromhttps://builds.dotnet.microsoft.com/dotnet/scripts/v1/(cachedunder
bin/$Configuration/dotnet/).--version <pinned>and--install-dir bin/$Configuration/dotnet.Install location stays at
bin/$Configuration/dotnet/(where xaprepareput it) so
dotnet-local.{cmd,sh}continues to work unchanged.Wired in everywhere
xaprepareran the install beforeMakefile:prepare:target now depends on a newinstall-dotnettarget that calls
./eng/install-dotnet.sh.build-tools/scripts/PrepareWindows.targets: new_InstallDotNettarget runs
eng/install-dotnet.ps1before_BuildXAPrepare.build.cmd: unchanged — the existingdotnet msbuild ... -t:Prepareflow still works because
_BuildXAPreparenow installs the SDK first.Step_InstallDotNetPreview→Step_PrepareDotNetWorkloadsThe old 220-line installer step is deleted. A new ~120-line
Step_PrepareDotNetWorkloads.csreplaces it and only doesAndroid-specific workload prep (NuGet cleanup,
package-download.projrestore with 3-attempt retry, and workload manifest copy). Everything
SDK-install-related (download script, archive override,
InstallDotNetAsyncetc.) is gone.global.json:tools.dotnet— NOT addedI originally tried pinning the SDK version in
global.json:tools.dotnet(the standard Arcade convention), but verified in
arcade-services/.../DependencyFileManager.csthat darcdoes not auto-update
global.json:tools.dotnetwhen theMicrosoft.NET.Sdkasset flows. Only specific Arcade/Helix SDK namesand the literal name
dotnetare special-cased. So atools.dotnetpin would have permanently drifted from the auto-flowed
eng/Versions.props:MicrosoftNETSdkPackageVersion.The wrappers therefore read the version from
Versions.propsdirectlyand bypass Arcade's
eng/common/tools.{sh,ps1}(which would otherwisestrict-mode-read
$GlobalJson.tools). Single source of truth = thedarc-flowed
eng/Versions.props.Other cleanups
Configurables.{Unix,Windows}.cs: removedUrls.DotNetInstallScript(no longer needed).
Context.cs+Main.cs: removedLocalDotNetSdkArchive/--dotnet-sdk-archiveplumbing. (The replacement is the standardDOTNET_INSTALL_DIRenv var that anyone needing offline support canset themselves.)
Verified on Windows
eng/install-dotnet.ps1(with download)dotnet msbuild Xamarin.Android.sln -t:PrepareThe
dotnet --list-sdksoutput after a cold install correctly shows11.0.100-preview.5.26268.112atbin/Debug/dotnet/sdk. Re-running Prepare is silent (no spuriousre-installs, no extra workload restores).
Migration path for the rest of xaprepare (future PRs)
Step_PrepareDotNetWorkloads.targetsfileStep_GenerateCGManifest.targetsfileStep_GenerateFiles[.Windows]Inputs/Outputsbuild-tools/xaprepare/andPrepareWindows.targetsEnd state:
./eng/install-dotnet.sh+dotnet build. Nothing else.