Skip to content

docs: capture how to test the AzDO release pipeline (previewRun + DryRun read-only)#18135

Draft
radical wants to merge 1 commit into
microsoft:mainfrom
radical:ankj/release-pipeline-testing-docs
Draft

docs: capture how to test the AzDO release pipeline (previewRun + DryRun read-only)#18135
radical wants to merge 1 commit into
microsoft:mainfrom
radical:ankj/release-pipeline-testing-docs

Conversation

@radical

@radical radical commented Jun 12, 2026

Copy link
Copy Markdown
Member

What was missing

The AzDO release pipeline (eng/pipelines/release-publish-nuget.yml, def microsoft-aspire-Release-To-NuGet) is never compiled on a GitHub PR — AzDO only compiles it at run time, and no AzDO pipeline runs on a GitHub PR. So two whole classes of defect pass every PR check (and the offline Infrastructure.Tests) and only surface at release time:

  1. Template/compile errors that fail the pipeline before any stage runs.
  2. Dry-run runs that aren't actually read-only.

Both were hit while testing a recent release-pipeline change. A literal template expression inside an inlined PowerShell script block comment compiled the parameters object to a string and broke the pipeline:

/eng/pipelines/release-publish-nuget.yml (Line: 710, Col: 29): Unable to convert from Object to String. Value: Object

And a DryRun=true run still dispatched a real release-github-tasks.yml Actions run, because that stage is gated on SkipGitHubTasks, not DryRun (tracked separately by #18129).

None of our skills or review instructions captured how to catch either before merge.

The change (docs / agent-tooling only)

  • New .github/instructions/release-pipeline.instructions.md (auto-applied to the release pipeline YAML, publish-*.yml, eng/pipelines/scripts/**, and release-github-tasks.yml): a review rule that every side-effecting step must be a no-op under DryRun=true — including the GitHub Tasks dispatch — plus the literal-template-expression-in-inlined-script trap.

  • pr-testing skill ci-infra-testing.md (Track B): new "Testing a release-pipeline change" section — previewRun compile-check first, then a DryRun=true run pinned to a real def-1602 source build; a DryRun read-only audit table; the GitHubTasks-not-gated-by-DryRun caveat; and a fix for a stale claim that the release pipeline "can't be run from a feature branch" (it can, under DryRun).

  • azdo-internal skill SKILL.md: release + unofficial pipelines added to Related Pipelines with their (stable) definition IDs; a previewRun compile-without-queuing recipe; the REST runs API recipe for pinning a consumed source build (az pipelines run can't pin a resource version); and the --api-version 7.1 quirk.

  • AGENTS.md: register the new instructions file.

Surprises / call-outs

  • No product or pipeline code changes — docs and agent skills/instructions only.
  • Definition IDs are now hard-coded (1600 release / 1601 unofficial / 1602 main) and the note that "IDs can change" is corrected — AzDO definition IDs are stable for a pipeline's lifetime.
  • The skills live under .agents/skills/ on main (they were relocated from .github/skills/); these edits target the main layout.

Refs #18129

…ease-pipeline testing guidance

The AzDO release pipeline (release-publish-nuget.yml) never compiles on a
GitHub PR, so template errors and unsafe dry-run side effects pass every PR
check and only surface at release time. Document how to test it and add a
review rule for keeping DryRun read-only.

- New .github/instructions/release-pipeline.instructions.md (auto-applied to
  release-publish-nuget.yml, publish-*.yml, eng/pipelines/scripts/**, and
  release-github-tasks.yml): every side-effecting step must be a no-op under
  DryRun=true, including the GitHub Tasks dispatch; and never write a literal
  template expression inside an inlined script block (the template engine
  evaluates it even in comments/strings -> "Unable to convert from Object to
  String").

- pr-testing/ci-infra-testing.md (Track B): new "Testing a release-pipeline
  change" section -- previewRun compile-check first, then a DryRun run pinned to
  a real def-1602 source build (branch pushed to the mirror); DryRun read-only
  audit table + the GitHubTasks-not-gated-by-DryRun caveat; corrected the stale
  "can't run the release pipeline from a feature branch" claim.

- azdo-internal/SKILL.md: release/unofficial pipelines added to Related
  Pipelines with their (stable) definition IDs; previewRun compile-check recipe;
  pin-the-source-build REST runs API recipe; api-version 7.1 quirk; refined the
  publish/release dry-run validation guidance.

- AGENTS.md: register the new instructions file.

Refs microsoft#18129 (release pipeline DryRun is not fully read-only).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings June 12, 2026 00:28
@github-actions

Copy link
Copy Markdown
Contributor

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 18135

Or

  • Run remotely in PowerShell:
iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 18135"

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds documentation and agent-skill/instructions content to capture how to validate changes to the AzDO release pipeline (release-publish-nuget.yml), which is never compiled during a GitHub PR — meaning template errors and DryRun gaps only surface at release time. The changes are purely documentation/tooling, with no product or pipeline code modifications.

Changes:

  • New .github/instructions/release-pipeline.instructions.md with review rules for DryRun read-only enforcement and template-expression traps in inlined scripts.
  • Extended pr-testing skill (ci-infra-testing.md) with a new "Testing a release-pipeline change" section covering previewRun, DryRun audit, and the known GitHubTasks gap (#18129).
  • Extended azdo-internal skill (SKILL.md) with previewRun compile-check recipe, REST runs API for pinning a source build, hardcoded definition IDs (1600/1601/1602), and updated "Validating publish/release-only changes safely" section.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.

File Description
AGENTS.md Registers new release-pipeline.instructions.md in the pattern-based instructions table
.github/instructions/release-pipeline.instructions.md New instructions file with DryRun read-only rules and template-expression trap warnings
.agents/skills/pr-testing/ci-infra-testing.md Adds release-pipeline testing section (previewRun, DryRun audit table, known gap caveat) and corrects stale "can't run from feature branch" claim
.agents/skills/azdo-internal/SKILL.md Adds release/unofficial pipelines to Related Pipelines table, previewRun recipe, REST runs API recipe for pinning resources, updates validation guidance

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants