backport: extract mage backport targets into standalone cmd/backport CLI sub-module by mrodm · Pull Request #20556 · elastic/integrations

mrodm · 2026-08-05T16:12:38Z

Proposed commit message

Extract all mage backport targets from the root magefile.go into a standalone CLI binary at cmd/backport/ with its own go.mod sub-module.

WHAT:

New sub-module github.com/elastic/integrations/cmd/backport with a self-contained CLI binary (cmd/backport/main.go) exposing all former mage backport targets as subcommands: validate-inventory, validate-branch-name, add-entry, check-active, detect-packages, check-owners, render-checklist, parse-checklist, update-checklist-status, sync-changelog, post-comment, apply, check-changelog-versions
Library packages that the tool depends on moved/copied from dev/ into cmd/backport/ (backports/, citools/, codeowners/, gitutil/). Dead code removed from all copied packages.
cmd/backport/.go-version pins the tool's Go version independently from the root .go-version so the tool always builds with a modern toolchain even when synced onto older backport branches.
New file magefile_daily_jobs.go splits main-branch-only mage targets (ReportFailedTests, RequiresUpdate) that import go-gh/v2 from the rest of magefile.go; backport_branch.sh removes this file when creating backport branches so go mod tidy works with older Go toolchains.
cmd/backport/magefile.go adds Build, Install, Test, Format, Check, Tidy, Clean targets for the sub-module.
All CI scripts (Buildkite + GitHub Actions) updated to call backport <subcommand> instead of mage <Target>.
common.sh: new with_backport() function builds the binary from cmd/backport/.go-version; BACKPORT_BIN exported for reuse across steps.
check_backport_tool.sh: new Buildkite step runs mage check + check_git_diff for the sub-module.
non_package_patterns.txt: added ^cmd/backport/, ^magefile_daily_jobs.go, ^dev/packagenames/ so changes to tooling files do not trigger full package test runs.
Cherry-pick scoping fix in apply: after cherry-picking onto a backport branch that only has a subset of packages, files outside the target package are now reset to HEAD (using git checkout HEAD with a git rm --force fallback for modify/delete conflicts), preventing spurious status: conflict results.

WHY:

Using mage as the entry-point for backport automation required contributors and CI agents to install mage and always run with the root module's Go version. Backport branches frequently carry older Go toolchains (1.19–1.21) that are incompatible with newer dependencies (go-gh/v2, dev/testsreporter, etc.) pulled from main. A standalone binary resolves this: it has its own pinned Go version, its own go.mod, and is copied verbatim onto backport branches by backport_branch.sh, so every branch runs the same tool regardless of its own Go version.

backport: extract mage backport targets into standalone cmd/backport CLI sub-module

Move all backport automation out of the root magefile into a self-contained
cmd/backport binary with its own go.mod and pinned Go version. Update all
Buildkite and GitHub Actions scripts to invoke the binary instead of mage.
Backport branches always receive the latest tool via backport_branch.sh.

Author's Checklist

All former mage backport targets work as backport <subcommand> equivalents
go build -C cmd/backport ./... and go test -C cmd/backport ./... pass
go build ./... and go build ./dev/... still pass (root module unchanged)
backport_branch.sh tested with dry-run mode on a real backport branch https://buildkite.com/elastic/integrations-backport/builds/322
All 4 GitHub Actions backport workflows verified (binary path, flag order)
mage check passes inside cmd/backport/ (format, tidy, test all green)
Run test in my own fork: Test package pr - multiple packages mrodm/integrations#68

How to test this PR locally

# Build the binary
go build -C cmd/backport -o build/backport .
./build/backport help

# Run sub-module tests
go test -C cmd/backport ./...

# Run the full check (format + tidy + test + build)
cd cmd/backport && mage check

# Verify root module still builds
go build ./...
go build ./dev/...

# Smoke-test a few subcommands
./build/backport validate-inventory
./build/backport check-active backport-aws-1.0

Related issues

This PR was generated with the assistance of Claude (claude-sonnet-4-6).

Replaces all mage backport targets with a self-contained Go sub-module at cmd/backport/ that builds a `backport` binary. The sub-module has its own go.mod and cmd/backport/.go-version (pinned to main's Go version) so the tool always compiles with a modern toolchain even on backport branches that carry an older root .go-version (e.g. 1.19/1.20). Key changes: - cmd/backport/main.go: CLI binary with subcommands validate-inventory, check-active, validate-branch-name, check-owners, detect-packages, render/parse/update-checklist, sync-changelog, post-comment, apply, check-changelog-versions, add-entry - cmd/backport/go.mod: standalone sub-module; mage pinned via `tool` directive (Go 1.24+); no replace directive — fully self-contained - Library packages moved from dev/backports/ into the sub-module; dev/citools, dev/codeowners, dev/gitutil copied (originals kept for root module use); dead code pruned from the copies - Binary output: build/backport (gitignored, matching elastic-package convention); BACKPORT_BIN exported in common.sh - with_backport() builds in a subshell so the Go version switch does not leak into the caller (fixes go mod tidy in backport_branch.sh) - All Buildkite scripts and GitHub Actions workflows updated; new check-backport-tool Buildkite step runs cmd/backport tests using cmd/backport/.go-version Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

- non_package_patterns.txt: replace ^dev/backports/ (deleted) with ^cmd/backport/; add check_backport_tool.sh; sort entries alphabetically - pull-requests.json: add check_backport_tool.sh to skip_ci_on_only_changed; cmd/backport/ is intentionally not skipped so tool changes run full CI Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

- dev/gitutil: remove Git.RunToStderr (no callers in root module; cmd/backport/gitutil keeps its own copy used by apply and sync) - dev/citools: make parsePackageManifest private (only called internally by ReadPackageManifest in the same file) - dev/codeowners: remove ParseOwners (no callers outside tests; cmd/backport/codeowners keeps its own copy used by compare.go) - dev/codeowners test: drop TestParseOwners; replace remaining ParseOwners calls with a mustParseOwners helper that calls scanGithubOwners directly Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Move ReportFailedTests and RequiresUpdate (the two mage targets that import go-gh/v2 via dev/testsreporter and dev/requiresupdate) into a new magefile_main_only.go. These targets are only invoked from main-branch pipelines; backport branches never call them. Update backport_branch.sh to exclude this file and the two dev packages when creating a backport branch, so go mod tidy drops go-gh/v2. This allows backport branches to retain their original Go version — .go-version is no longer copied from main. go mod tidy now runs in a subshell that installs Go from the backport branch's own .go-version via gvm; main's Go version is restored in the parent shell afterwards. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Adds Format (go-licenser + goimports) and Check (Build → Format → Tidy → Test) targets to the cmd/backport magefile, matching the root magefile pattern. Tool deps pinned via tools.go (//go:build tools) consistent with the root module. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Change the build output path from $GITHUB_WORKSPACE/backport to $GITHUB_WORKSPACE/build/backport in all four backport workflows and process-checked-branches.sh, consistent with the Buildkite BACKPORT_BIN path and the magefile Build target. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

The run: value started with a double-quoted string, causing YAML to treat it as a quoted scalar and leave the subcommand as invalid trailing content. Convert to a block scalar (run: |) to match the surrounding steps. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Go's flag package stops parsing at the first non-flag argument, so --json must come before the positional SHA arguments. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…onflicts When a commit on main touches multiple packages, cherry-picking it onto a backport branch that doesn't have all the same packages previously reported a spurious conflict instead of succeeding. Add resetNonPackageChanges() which iterates git status --porcelain after the cherry-pick and resets each file outside the target package dir to HEAD: - git checkout HEAD -- file (handles clean staged changes and UU conflicts) - git rm --force -- file fallback (handles DU modify/delete conflicts where HEAD has the file deleted, e.g. packages absent from the backport branch) A note is printed to stderr when files outside the package are reset. Add three integration tests covering all three cases: - modify/delete conflict (file absent from backport branch) - regular UU conflict (file exists on both branches, different edits) - clean apply to other package (file exists, no conflict, still discarded) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Replace `go test` with `mage check` so the CI step exercises Build, Format, Tidy, and Test together. Add check_git_diff at the end to catch any files left dirty by format or tidy. Use add_bin_path + with_mage (from common.sh) to set up Go and install mage from inside cmd/backport/, matching the version pinned in cmd/backport/.go-version. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Changes to the main-only mage targets file should not trigger package tests. Also fixes a stale test path (dev/backports/ no longer exists) replaced with the correct cmd/backport/ path. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Update all references in backport_branch.sh, non_package_patterns.txt, and test_non_package_patterns.sh. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

It is dev tooling (duplicate-name checker), not package content. Consistent with dev/codeowners, dev/gitutil, and similar patterns. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Backport branches must use the backport implementation from cmd/backport.

mrodm · 2026-08-06T07:39:15Z

+    echo "--- Removing dev/testsreporter and dev/requiresupdate (go-gh/v2 not needed on backport branches)..."
+    rm -rf "${DEV_FOLDER}/testsreporter" "${DEV_FOLDER}/requiresupdate"


These packages are just used in daily jobs. That why they are removed, and therefore they will be considered when running go mod tidy in backport branches.

mrodm · 2026-08-06T07:39:57Z

+    echo "--- Removing dev/backports (in favor of cmd/backport)..."
+    # Remove the previous backport code from dev/backport, it should be used from cmd/backport instead.
+    rm -rf "${DEV_FOLDER}/backports"


Removing this as a safety , in case there could be backport branches that include now this folder.

With this PR, all this logic should be taken from cmd/backport folder.

mrodm · 2026-08-06T07:40:51Z

+    (
+      echo "--- Installing Go $(cat "${WORKSPACE}/.go-version") (from backport branch .go-version) for go mod tidy..."
+      eval "$("${BIN_FOLDER}/gvm" "$(cat "${WORKSPACE}/.go-version")")"
+      go version
+      # go-gh/v2 is intentionally absent so this works with the backport branch's
+      # original Go toolchain version.
+      echo "--- Running go mod tidy to update dependencies related to magefile and dev scripts..."
+      go mod tidy
+    )


This go mod tidy is going to be executed in the context of the backport branch.

That's why it uses the same go version from that branch to avoid changing dependencies unexpectedly.

mrodm · 2026-08-06T07:41:18Z


 echo "--- Validating custom backport branch name"
-if ! mage ValidateBackportBranchName "${PACKAGE_NAME}" "${BACKPORT_BRANCH_NAME}"; then
+if ! backport validate-branch-name "${PACKAGE_NAME}" "${BACKPORT_BRANCH_NAME}"; then


Example of use of this new tool.

In Buildkite first, it needs to be built using with_backpotr.

mrodm · 2026-08-06T07:43:11Z

Under dev/ folder all the logic related to backports has been removed.
It is just kept code related to or used by the CI

mrodm · 2026-08-06T07:44:23Z

This new magefile is just to be used in the daily CI jobs.

Splitting this file allows us to not copy these targets to backport branches.
One drawback of copying these targets to backport branches is that they require go-gh/v2 that requires a newer go version that some of our backport branches.

elastic-vault-github-plugin-prod · 2026-08-06T07:49:58Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

mrodm · 2026-08-06T09:46:18Z

 		paths[i] = p.Path
 	}
-	slices.Sort(paths)
+	sort.Strings(paths)


This is changed to avoid errors in backport branches where slices cannot be used. For instances branches like backport-kubernetes-1.39 that sets Golang 1.19.1

mrodm · 2026-08-06T10:58:25Z

Tested to apply these changes in different backport branches:

Old backport branches with golang versions old (v1.19):
- [backport-gcp-2.22] Sync CI configuration with main branch #20555
- [backport-kubernetes-1.39] Sync CI configuration with main branch #20542
Latest backport branches created:
- [backport-aws-6.x] Sync CI configuration with main branch #20414
- [backport-security_detection_engine-9.4] Sync CI configuration with main branch #20416

teresaromero · 2026-08-06T13:58:59Z

i was thinking on how to solve this "file duplication" in a better way... claude suggested to use the replace directive on the go.mod of cmd/backport so it point into /dev/citools (for this file example) . not sure is worth the change... in any case a comment on the file perhaps can give the context that this are originated on /dev/ and where copied into the tool?

Added some comments in those files in this commit: c021e4a

Explains that citools, codeowners, and gitutil are local copies of their dev/ counterparts, and why a replace directive is not used. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

teresaromero

👍🏻 thanks!

mergify · 2026-08-07T08:25:23Z

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

Queue this pull request

This reverts commit 4d81848.

This reverts commit d68b024.

This reverts commit 848d00c.

mrodm · 2026-08-07T10:01:56Z

/test

infra-vault-gh-plugin-prod · 2026-08-07T10:29:37Z

💔 Build Failed

Buildkite Build
Commit: aee2d72

Failed CI Steps

History

💔 Build #47432 failed aee2d72
💚 Build #47374 succeeded c021e4a
💚 Build #47345 succeeded 3f38c30
💚 Build #47329 succeeded 848d00c

cc @mrodm

mrodm · 2026-08-07T13:14:44Z

CI failure related to qualys_vmdr package is unrelated to this change.
It is also failing in daily ci jobs and the issue is reported: #20585

mrodm and others added 19 commits August 4, 2026 17:58

backport: replace slices.Sort with sort.Strings for Go 1.19 compat

0e30e9b

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

backport: fix gomod test fixture for Go < 1.21 compatibility

a3cbf63

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

backport: sync cmd/backport to backport branches from main

125aa19

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Re-order imports in cmd/backports/tools.go

2f9f550

backport: fix --json flag position in detect-packages workflow call

3a5a208

Go's flag package stops parsing at the first non-flag argument, so --json must come before the positional SHA arguments. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

backport: rename magefile_main_only.go -> magefile_daily_jobs.go

96dcb9e

Update all references in backport_branch.sh, non_package_patterns.txt, and test_non_package_patterns.sh. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

backport: exclude dev/packagenames from package test triggers

0bf8b94

It is dev tooling (duplicate-name checker), not package content. Consistent with dev/codeowners, dev/gitutil, and similar patterns. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Merge upstream/main into separate_backport_tool

e15d374

Remove the previous implementation of dev/backports

a7b4047

Backport branches must use the backport implementation from cmd/backport.

mrodm self-assigned this Aug 5, 2026

mrodm mentioned this pull request Aug 5, 2026

backport: deploy backport-branch automation workflows to active backport branches #19262

Open

5 tasks

mrodm added 3 commits August 6, 2026 09:21

Update dependabot configuration for cmd/backport

8d2dd0f

Update commit message

04d9e9c

Test just one package - to be reverted

848d00c

mrodm commented Aug 6, 2026

View reviewed changes

Comment thread .buildkite/scripts/common.sh Outdated

mrodm added 3 commits August 6, 2026 11:54

Test integrations-backport pipeline - to be reverted

4d81848

Restore installation of mage

75fda91

Add also elastic_package_registry to subset of packages

d68b024

mrodm marked this pull request as ready for review August 6, 2026 10:16

mrodm requested a review from a team as a code owner August 6, 2026 10:16

mrodm commented Aug 6, 2026

View reviewed changes

Comment thread .buildkite/scripts/backport_branch.sh Outdated

Remove first cmd/backport folder to ensure the right files are kept

3f38c30

teresaromero reviewed Aug 6, 2026

View reviewed changes

mrodm mentioned this pull request Aug 6, 2026

[CI] Add /sync-changelog comment trigger to retry failed changelog sync PRs #20578

Open

5 tasks

backport: add copy-origin comments to cmd/backport library packages

c021e4a

Explains that citools, codeowners, and gitutil are local copies of their dev/ counterparts, and why a replace directive is not used. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

mrodm requested a review from teresaromero August 6, 2026 15:26

teresaromero approved these changes Aug 7, 2026

View reviewed changes

mrodm added 3 commits August 7, 2026 11:30

Revert "Test integrations-backport pipeline - to be reverted"

1fdfb6c

This reverts commit 4d81848.

Revert "Add also elastic_package_registry to subset of packages"

9336ff2

This reverts commit d68b024.

Revert "Test just one package - to be reverted"

aee2d72

This reverts commit 848d00c.

mrodm merged commit ebcf392 into elastic:main Aug 10, 2026
9 of 10 checks passed

mrodm deleted the separate_backport_tool branch August 10, 2026 07:52

mrodm mentioned this pull request Aug 10, 2026

Test e2e - multiple packages mrodm/integrations#79

Merged

		echo "--- Removing dev/testsreporter and dev/requiresupdate (go-gh/v2 not needed on backport branches)..."
		rm -rf "${DEV_FOLDER}/testsreporter" "${DEV_FOLDER}/requiresupdate"

Uh oh!

Conversation

mrodm commented Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Author's Checklist

How to test this PR locally

Related issues

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elastic-vault-github-plugin-prod Bot commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🚀 Benchmarks report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mrodm commented Aug 6, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

teresaromero left a comment

Choose a reason for hiding this comment

Uh oh!

mergify Bot commented Aug 7, 2026

Uh oh!

mrodm commented Aug 7, 2026

Uh oh!

infra-vault-gh-plugin-prod Bot commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💔 Build Failed

Failed CI Steps

History

Uh oh!

mrodm commented Aug 7, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mrodm commented Aug 5, 2026 •

edited

Loading

elastic-vault-github-plugin-prod Bot commented Aug 6, 2026 •

edited

Loading

infra-vault-gh-plugin-prod Bot commented Aug 7, 2026 •

edited

Loading