TWO-24998: make Magento version-combination CI coverage honest (classifier + skip surfacing)

[dgjlindsay]

Child of TWO-24739. Reference implementation (magento-plugin); magento-abn-plugin adopts the reconciled script in a follow-up PR.

What changed

dev/magento-support-matrix.sh classifies every in-window Magento×PHP combo as run or skip, and the split happens at matrix-assembly time (the discover-magento-matrix job) — not inside a green job.

• --emit-matrix returns runnable combos only. phpstan/di-compile legs are therefore all real tests: a green leg unambiguously means "tested." A skipped combo is never a test leg, so it can never masquerade as a pass.
• --emit-skips returns untestable combos with reasons. The discover job renders them to $GITHUB_STEP_SUMMARY as a coverage table and emits one ::warning:: annotation per skipped combo — a not-run combination shows up yellow on the run summary, clearly distinct from a green pass, at a glance.
• Support window is honest. Current + previous 2 over all published minors — image availability no longer slides the window down. An in-policy minor with no CI image is a surfaced skip, not silently backfilled by an older out-of-policy minor.
• Missing CI image auto-detected via docker manifest inspect (retry + fail-toward-run so a Docker Hub blip can't zero the matrix). Retired the hand-maintained image-exclusion list — which the probe immediately showed was stale: 2.4.9 images exist for php84/php85, only php83 is missing.
• Network hardening: retry + JSON-shape validation on the php.net and per-minor composer.json fetches (were single un-retried SPOFs). GitHub token no longer sent to php.net.

Coverage (this run)

Runnable (real green legs): 2.4.9×{8.4,8.5}, 2.4.8×{8.3,8.4}, 2.4.7×{8.2,8.3}. NOT run (warning + table): 2.4.9×8.3 and 2.4.8×8.2 (no CI image), 2.4.7×8.1 (past upstream PHP support). Window moved {2.4.8,2.4.7,2.4.6} → the true current+prev2 {2.4.9,2.4.8,2.4.7}; SUPPORT_WINDOW is the lever for a wider net.

Raised by Claude (pairing with Doug).

[gemini-code-assist]

Code Review

This pull request refactors the dev/magento-support-matrix.sh script to transition from a filtering approach to a classification-based model. It introduces a fetch_json helper for robust API interaction and a probe_image function to dynamically verify Docker image availability, effectively removing the need for manual exclusion lists. The output format has been updated to explicitly report the status of each Magento × PHP combination, including skip reasons. The review feedback identifies a potential script termination issue caused by grep in a set -e environment and suggests using native Bash parameter expansion for string manipulation to improve efficiency.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

In the subsequent pipeline (lines 244-247), grep -oE is used to parse the PHP constraints:

    php_minors_for_magento=$(echo "$php_constraint" \
        | grep -oE '~[0-9]+\.[0-9]+\.0' \
        | sed -E 's/~([0-9]+\.[0-9]+)\.0/\1/' \
        | sort -V)

Because set -o pipefail and set -e are active, if grep finds no matches (e.g., if the constraint format changes or is invalid), it will exit with status 1. This will cause the entire pipeline to fail and terminate the script immediately, bypassing the defensive check on line 248.

To prevent this silent termination and allow the diagnostic error message on line 249 to be printed, append || true to the assignment:

    php_minors_for_magento=$(echo "$php_constraint" \
        | grep -oE '~[0-9]+\.[0-9]+\.0' \
        | sed -E 's/~([0-9]+\.[0-9]+)\.0/\1/' \
        | sort -V) || true

[gemini-code-assist]

We can optimize this by using native Bash parameter expansion instead of spawning a subshell with echo and tr. This is cleaner and more efficient.

Suggested change

	php_image="php$(echo "$php" | tr -d '.')-fpm"
	php_image="php${php//./}-fpm"

[dgjlindsay]

Addressed the Gemini review in 633b844:

• grep -oE constraint pipeline now ends with || true so a no-match under set -o pipefail can't terminate the script before the empty-check diagnostic.
• php_image uses ${php//./} parameter expansion (no subshell).

by Claude

[brtkwr]

Reviewed adversarially against the honest-CI direction (no fake-green skips). Aligned and happy with it: untestable combos surface as ::warning:: + a step-summary coverage table rather than passing green, the stale intentionally_excluded / hand-maintained exclusion machinery is fully removed with no dangling references, the upstream fetches have retry + JSON-shape validation, and the token is scoped to github.com hosts only. Approving as the source-of-truth classifier.

A few non-blocking notes for your consideration:

1. The discover step invokes the script three times (--emit-matrix, --emit-skips, --emit-php-lint-matrix), and each invocation re-runs the full classification (re-fetching the upstream JSON and re-probing every docker manifest). So the run-list and the skip-list come from two independent runs rather than one classification, which means under a transient docker manifest inspect error a combo could land in one output without being the exact mirror of the other, and it triples the anonymous Docker Hub rate-limit exposure. Consider classifying once into a single JSON and slicing run/skip/lint from that.

2. probe_image fails toward run, so in degraded / rate-limited conditions a genuinely-missing image becomes a real (red) matrix leg rather than the intended yellow skip. Not fake-green, but worth a header note that degraded probes shift skips into red legs by design.

3. Worth confirming the empty-matrix case (run_count == 0 -> include: [] -> zero jobs) doesn't wedge any required branch-protection checks that expect those job names to report.

None of these block; comfortable with it going in first as the parity source of truth.