fix: verify Pantheon push landed; pre-flight SFTP-mode flip; self-pin…

.github/actions/pantheon-push/action.yml

@@ -106,6 +106,42 @@ runs:
           echo "Nothing staged after composer install; skipping pre-commit (upstream push will no-op the commit too)."
         fi
 
+    # Pre-flight: ensure the target Pantheon env is in git mode. If it's been
+    # flipped to SFTP (e.g. someone opened the Dashboard Code view and toggled
+    # it), the push would be rejected by Pantheon's pre-receive hook —
+    # `pantheon-systems/push-to-pantheon@0.9.0` swallows that exit code and
+    # reports the step as successful. We pre-empt the most common cause here;
+    # the verify-after-push step below catches anything else.
+    #
+    # Only applies if the env already exists. New multidevs are created in git
+    # mode by terminus multidev:create (handled inside the upstream action).
+    - name: Ensure Pantheon env is in git mode (pre-flight)
+      shell: bash
+      env:
+        TERMINUS_MACHINE_TOKEN: ${{ inputs.machine_token }}
+        SITE: ${{ inputs.site }}
+        TARGET_ENV: ${{ inputs.target_env }}
+      run: |
+        set -euo pipefail
+        # Use env:list (an authoritative listing) rather than env:info >/dev/null,
+        # which would silently swallow auth/network errors as "env doesn't exist".
+        # If terminus itself errors, set -e fails the step with the real cause.
+        # Matches the existing pattern in reusable-pantheon-deploy-rc-multidev.yml
+        # for the live-env probe.
+        if ! terminus env:list "${SITE}" --format=list | grep -q "^${TARGET_ENV}$"; then
+          echo "Env ${SITE}.${TARGET_ENV} does not exist yet; multidev:create will provision it in git mode. Skipping connection-mode check."
+          exit 0
+        fi
+        # No 2>/dev/null fallback — if connection:info errors here (auth, network),
+        # we want the step to fail with terminus's real error rather than coerce
+        # the result to an empty string and attempt connection:set on bad data.
+        MODE=$(terminus connection:info "${SITE}.${TARGET_ENV}" --field=connection_mode)
+        echo "Current connection mode for ${SITE}.${TARGET_ENV}: ${MODE}"
+        if [[ "${MODE}" != "git" ]]; then
+          echo "::warning::${SITE}.${TARGET_ENV} is in '${MODE}' mode. Switching to git so the push isn't rejected by the pre-receive hook."
+          terminus connection:set "${SITE}.${TARGET_ENV}" git
+        fi
+
     - name: Push to Pantheon
       uses: pantheon-systems/push-to-pantheon@0.9.0
       env:
@@ -127,3 +163,62 @@ runs:
         # multidev:create, so re-pushes to an existing rc-/epr-/pr- multidev
         # update the code instead of failing with "environment already exists".
         skip_build_tools: true
+
+    # Verify-after-push: pantheon-systems/push-to-pantheon@0.9.0 does NOT
+    # propagate the exit code from `git push`. When Pantheon rejects the push
+    # (SFTP mode, branch protection, lost-update race, etc.) the action still
+    # reports the step as success and bobheadxi/deployments@v1 (also invoked
+    # inside the upstream) marks the GitHub deployment "success". We compare
+    # the local HEAD we meant to push against the remote ref Pantheon now has,
+    # and fail the workflow loudly if they don't match. Belt-and-braces:
+    # catches the SFTP case our pre-flight already handles AND any future
+    # silent-fail mode in the upstream action.
+    #
+    # Requires the SSH agent loaded by the upstream's webfactory/ssh-agent
+    # step (SSH_AUTH_SOCK is in GITHUB_ENV, so it persists into this step).
+    - name: Verify push landed on Pantheon
+      shell: bash
+      env:
+        TERMINUS_MACHINE_TOKEN: ${{ inputs.machine_token }}
+        SITE: ${{ inputs.site }}
+        TARGET_ENV: ${{ inputs.target_env }}
+      run: |
+        set -euo pipefail
+        LOCAL_HEAD=$(git rev-parse HEAD)
+        PANTHEON_GIT_URL=$(terminus connection:info "${SITE}.${TARGET_ENV}" --field=git_url)
+        if [[ -z "${PANTHEON_GIT_URL}" ]]; then
+          echo "::error::Could not determine Pantheon git URL for ${SITE}.${TARGET_ENV}; cannot verify push."
+          exit 1
+        fi
+        echo "Local HEAD:    ${LOCAL_HEAD}"
+        echo "Pantheon URL:  ${PANTHEON_GIT_URL}"
+        # Retry ls-remote a few times: for freshly created multidevs there can
+        # be a short window where Pantheon has accepted the push but the ref
+        # isn't yet visible via ls-remote (related to the propagation race the
+        # post-deploy drush step also defends against). Three attempts with
+        # short backoff covers the typical lag without papering over a real
+        # silent-fail.
+        REMOTE_HEAD=""
+        for attempt in 1 2 3; do
+          REMOTE_HEAD=$(git ls-remote "${PANTHEON_GIT_URL}" "refs/heads/${TARGET_ENV}" | awk '{print $1}')
+          if [[ -n "${REMOTE_HEAD}" ]]; then
+            break
+          fi
+          if [[ ${attempt} -lt 3 ]]; then
+            SLEEP=$((attempt * 5))
+            echo "ls-remote returned no ref on attempt ${attempt}; retrying in ${SLEEP}s (Pantheon ref propagation lag)..."
+            sleep "${SLEEP}"
+          fi
+        done
+        echo "Pantheon HEAD: ${REMOTE_HEAD:-<empty>}"
+        if [[ -z "${REMOTE_HEAD}" ]]; then
+          echo "::error::Pantheon ${TARGET_ENV} has no commits on refs/heads/${TARGET_ENV} after 3 attempts. The upstream push silently failed or the branch was never created."
+          echo "::error::Check the previous step's output for 'remote rejected' / 'pre-receive hook declined' / SFTP-mode notices."
+          exit 1
+        fi
+        if [[ "${LOCAL_HEAD}" != "${REMOTE_HEAD}" ]]; then
+          echo "::error::Pantheon ${TARGET_ENV} is at ${REMOTE_HEAD}, expected ${LOCAL_HEAD}. The upstream push silently failed."
+          echo "::error::Check the previous step's output for 'remote rejected' / 'pre-receive hook declined' / SFTP-mode notices."
+          exit 1
+        fi
+        echo "✅ Pantheon ${TARGET_ENV} is at ${LOCAL_HEAD} as expected."

.github/workflows/reusable-pantheon-deploy-dev.yml

@@ -62,7 +62,7 @@ jobs:
   semantic_release:
     name: Semantic Release
     if: ${{ inputs.run_semantic_release }}
-    uses: Square360/shared-workflows/.github/workflows/reusable-semantic-release.yml@v3.2.3
+    uses: Square360/shared-workflows/.github/workflows/reusable-semantic-release.yml@v3.2.4
     secrets:
       CI_GH_TOKEN: ${{ secrets.CI_GH_TOKEN }}
 
@@ -92,7 +92,7 @@ jobs:
           fetch-depth: 1
 
       - name: Install Terminus
-        uses: Square360/shared-workflows/.github/actions/terminus-install@v3.2.3
+        uses: Square360/shared-workflows/.github/actions/terminus-install@v3.2.4
         with:
           machine_token: ${{ secrets.PANTHEON_MACHINE_TOKEN }}
 
@@ -102,7 +102,7 @@ jobs:
       # -----------------------------------------------------------------------
       - name: Push code to Pantheon DEV
         id: deploy_pantheon
-        uses: Square360/shared-workflows/.github/actions/pantheon-push@v3.2.3
+        uses: Square360/shared-workflows/.github/actions/pantheon-push@v3.2.4
         with:
           site: ${{ inputs.pantheon_site }}
           target_env: dev
@@ -159,7 +159,7 @@ jobs:
 
       - name: Run post-deploy drush commands
         id: drush_commands
-        uses: Square360/shared-workflows/.github/actions/pantheon-post-deploy-drush@v3.2.3
+        uses: Square360/shared-workflows/.github/actions/pantheon-post-deploy-drush@v3.2.4
         with:
           site: ${{ inputs.pantheon_site }}
           env: dev

.github/workflows/reusable-pantheon-deploy-epic-multidev.yml

@@ -126,7 +126,7 @@ jobs:
           echo "Epic multidev: ${TARGET_ENV}"
 
       - name: Install Terminus
-        uses: Square360/shared-workflows/.github/actions/terminus-install@v3.2.3
+        uses: Square360/shared-workflows/.github/actions/terminus-install@v3.2.4
         with:
           machine_token: ${{ secrets.PANTHEON_MACHINE_TOKEN }}
 
@@ -148,7 +148,7 @@ jobs:
 
       - name: Push code to Pantheon epic multidev
         id: deploy
-        uses: Square360/shared-workflows/.github/actions/pantheon-push@v3.2.3
+        uses: Square360/shared-workflows/.github/actions/pantheon-push@v3.2.4
         with:
           site: ${{ inputs.pantheon_site }}
           target_env: ${{ steps.derive.outputs.target_env }}
@@ -159,7 +159,7 @@ jobs:
 
       - name: Run post-deploy drush commands
         id: drush_commands
-        uses: Square360/shared-workflows/.github/actions/pantheon-post-deploy-drush@v3.2.3
+        uses: Square360/shared-workflows/.github/actions/pantheon-post-deploy-drush@v3.2.4
         with:
           site: ${{ inputs.pantheon_site }}
           env: ${{ steps.derive.outputs.target_env }}
@@ -234,7 +234,7 @@ jobs:
     # The security workflow detects epr-* internally and runs unconditionally
     # in strict mode (fails on any High-severity finding) — same gate as rc-*,
     # per the Square360 SOP for epic integration multidevs. No force_run needed.
-    uses: Square360/shared-workflows/.github/workflows/reusable-pantheon-security-scan.yml@v3.2.3
+    uses: Square360/shared-workflows/.github/workflows/reusable-pantheon-security-scan.yml@v3.2.4
     with:
       pantheon_site: ${{ inputs.pantheon_site }}
       target_env: ${{ needs.deploy_epic_multidev.outputs.target_env }}

.github/workflows/reusable-pantheon-deploy-pr-multidev.yml

@@ -83,7 +83,7 @@ jobs:
         run: echo "target_env=pr-${{ github.event.pull_request.number }}" >> $GITHUB_OUTPUT
 
       - name: Install Terminus
-        uses: Square360/shared-workflows/.github/actions/terminus-install@v3.2.3
+        uses: Square360/shared-workflows/.github/actions/terminus-install@v3.2.4
         with:
           machine_token: ${{ secrets.PANTHEON_MACHINE_TOKEN }}
 
@@ -105,7 +105,7 @@ jobs:
 
       - name: Push code to Pantheon multidev
         id: deploy
-        uses: Square360/shared-workflows/.github/actions/pantheon-push@v3.2.3
+        uses: Square360/shared-workflows/.github/actions/pantheon-push@v3.2.4
         with:
           site: ${{ inputs.pantheon_site }}
           target_env: ${{ steps.derive.outputs.target_env }}
@@ -116,7 +116,7 @@ jobs:
 
       - name: Run post-deploy drush commands
         id: drush_commands
-        uses: Square360/shared-workflows/.github/actions/pantheon-post-deploy-drush@v3.2.3
+        uses: Square360/shared-workflows/.github/actions/pantheon-post-deploy-drush@v3.2.4
         with:
           site: ${{ inputs.pantheon_site }}
           env: ${{ steps.derive.outputs.target_env }}
@@ -176,7 +176,11 @@ jobs:
           fi
 
       - name: Comment on PR with multidev URL
-        if: ${{ github.event_name == 'pull_request' }}
+        # Gate on success() so the "🚀 Multidev Environment Ready!" comment
+        # doesn't get posted when the deploy actually failed — including the
+        # case where pantheon-push's verify-after-push step caught the
+        # upstream action's silent-fail mode (SFTP-mode rejection, etc.).
+        if: ${{ success() && github.event_name == 'pull_request' }}
         uses: actions/github-script@v8
         with:
           script: |
@@ -224,7 +228,7 @@ jobs:
         contains(github.event.pull_request.body, '[run-security]') ||
         contains(github.event.pull_request.body, '[security]')
       }}
-    uses: Square360/shared-workflows/.github/workflows/reusable-pantheon-security-scan.yml@v3.2.3
+    uses: Square360/shared-workflows/.github/workflows/reusable-pantheon-security-scan.yml@v3.2.4
     with:
       pantheon_site: ${{ inputs.pantheon_site }}
       target_env: ${{ needs.deploy_pr_multidev.outputs.target_env }}
@@ -252,7 +256,7 @@ jobs:
         contains(github.event.pull_request.body, '[run-vrt]') ||
         contains(github.event.pull_request.body, '[vrt]')
       }}
-    uses: Square360/shared-workflows/.github/workflows/reusable-pantheon-vrt.yml@v3.2.3
+    uses: Square360/shared-workflows/.github/workflows/reusable-pantheon-vrt.yml@v3.2.4
     with:
       pantheon_site: ${{ inputs.pantheon_site }}
       target_env: ${{ needs.deploy_pr_multidev.outputs.target_env }}

.github/workflows/reusable-pantheon-deploy-rc-multidev.yml

@@ -88,7 +88,7 @@ jobs:
           echo "Derived RC multidev name: ${TARGET_ENV}"
 
       - name: Install Terminus
-        uses: Square360/shared-workflows/.github/actions/terminus-install@v3.2.3
+        uses: Square360/shared-workflows/.github/actions/terminus-install@v3.2.4
         with:
           machine_token: ${{ secrets.PANTHEON_MACHINE_TOKEN }}
 
@@ -111,7 +111,7 @@ jobs:
 
       - name: Push code to Pantheon multidev
         id: deploy
-        uses: Square360/shared-workflows/.github/actions/pantheon-push@v3.2.3
+        uses: Square360/shared-workflows/.github/actions/pantheon-push@v3.2.4
         with:
           site: ${{ inputs.pantheon_site }}
           target_env: ${{ steps.derive.outputs.target_env }}
@@ -122,7 +122,7 @@ jobs:
 
       - name: Run post-deploy drush commands
         id: drush_commands
-        uses: Square360/shared-workflows/.github/actions/pantheon-post-deploy-drush@v3.2.3
+        uses: Square360/shared-workflows/.github/actions/pantheon-post-deploy-drush@v3.2.4
         with:
           site: ${{ inputs.pantheon_site }}
           env: ${{ steps.derive.outputs.target_env }}
@@ -165,7 +165,7 @@ jobs:
     needs: deploy_rc_multidev
     # The security workflow detects rc-* internally and runs unconditionally
     # in strict mode (fails on any High-severity finding). No force_run needed.
-    uses: Square360/shared-workflows/.github/workflows/reusable-pantheon-security-scan.yml@v3.2.3
+    uses: Square360/shared-workflows/.github/workflows/reusable-pantheon-security-scan.yml@v3.2.4
     with:
       pantheon_site: ${{ inputs.pantheon_site }}
       target_env: ${{ needs.deploy_rc_multidev.outputs.target_env }}
@@ -193,7 +193,7 @@ jobs:
         contains(github.event.head_commit.message, '[run-vrt]') ||
         contains(github.event.head_commit.message, '[vrt]')
       }}
-    uses: Square360/shared-workflows/.github/workflows/reusable-pantheon-vrt.yml@v3.2.3
+    uses: Square360/shared-workflows/.github/workflows/reusable-pantheon-vrt.yml@v3.2.4
     with:
       pantheon_site: ${{ inputs.pantheon_site }}
       target_env: ${{ needs.deploy_rc_multidev.outputs.target_env }}

docs/CHANGELOG-AGENT.md

@@ -2,6 +2,36 @@
 
 This file tracks the work done together with GitHub Copilot on the Square360 Shared Pantheon Workflows repository.
 
+## 2026-05-22
+
+### Fix - `pantheon-push` fails loudly when Pantheon rejects the push (v3.2.4)
+
+- **The problem (YH-571 / Yale-Health-Drupal PR #126):** `pantheon-systems/push-to-pantheon@0.9.0` does not propagate the exit code from `git push`. When Pantheon's pre-receive hook rejects the push (env in SFTP mode, branch protection, lost-update race, etc.), the upstream action reports the step as successful, `bobheadxi/deployments@v1` (invoked inside the upstream) marks the GitHub deployment "success," and the consumer workflow continues — posting "🚀 Multidev Environment Ready!" on the PR while Pantheon still has the old SHA. Discovered when a security fix appeared deployed but `drush deploy` ran on stale code and re-installed a module we'd intentionally removed.
+- **Two defensive layers added to `.github/actions/pantheon-push/action.yml`:**
+  1. **Pre-flight SFTP-mode flip** — before invoking the upstream push, check `terminus connection:info <site>.<env> --field=connection_mode` and call `terminus connection:set <site>.<env> git` if it's anything other than `git`. Catches the most common cause of the silent rejection. Skipped when the env doesn't exist yet (multidev:create provisions in git mode).
+  2. **Verify-after-push** — after the upstream push step, compare `git rev-parse HEAD` (our local artifact commit) against `git ls-remote <pantheon-git-url> refs/heads/<target_env>`. If they differ, fail the workflow with a clear actionable error pointing the developer at the previous step's rejection notice. Belt-and-braces: catches *any* future silent-fail mode in the upstream, not just SFTP.
+- **`if: success()` gate on the PR-comment step** in `.github/workflows/reusable-pantheon-deploy-pr-multidev.yml` so the "Multidev Environment Ready!" PR comment doesn't post when the deploy actually failed.
+- **What this doesn't fix:** the `bobheadxi/deployments@v1` "deployment status: success" call lives inside the upstream `pantheon-systems/push-to-pantheon@0.9.0` action and runs before our verify step. The deployment record on the GitHub Deployments tab will still show success briefly until the workflow itself goes red. Fixing that requires either forking the upstream or adding a follow-up `bobheadxi/deployments@v1` "set status to failure" step in our reusable workflows — deferred. The job-level red state is enough to break the false-positive in CI.
+- **Out-of-scope upstream issue:** the underlying bug should also be filed upstream at `pantheon-systems/push-to-pantheon`, but that won't unblock us on any reasonable timeframe.
+- See `docs/todo/pantheon-push-swallows-rejection.md` (marked RESOLVED in v3.2.4) for the full investigation.
+
+### Maintenance - Self-reference bumps to v3.2.4
+
+- **Bumped all in-repo `uses:` self-references from `@v3.2.3` to `@v3.2.4`** across:
+  - `reusable-pantheon-deploy-dev.yml`
+  - `reusable-pantheon-deploy-pr-multidev.yml`
+  - `reusable-pantheon-deploy-rc-multidev.yml`
+  - `reusable-pantheon-deploy-epic-multidev.yml`
+- Affects: `terminus-install`, `pantheon-push`, `pantheon-post-deploy-drush`, `reusable-semantic-release.yml`, `reusable-pantheon-security-scan.yml`, `reusable-pantheon-vrt.yml`
+- IDE flags these as "Unable to resolve" until the v3.2.4 tag exists — expected; semantic-release tags on merge to `main` and the diagnostics resolve themselves.
+
+### Validation plan (manual, in Yale Health)
+
+1. Merge to `main` → semantic-release tags `v3.2.4`
+2. **Don't yet** bump `pantheon-github-workflows` template SHAs or push the consumer composer plugin — point Yale Health workflows manually at `@v3.2.4` first
+3. Reproduce the original failure mode: `terminus connection:set yalehealth-yale-edu.pr-126 sftp`, re-trigger the workflow, confirm it now fails loudly with the pre-flight warning OR the verify-after-push error (the pre-flight should catch SFTP first; the verify-after-push is the catch-all)
+4. Once verified in Yale Health, do the full release chain: capture SHA → bump `pantheon-github-workflows` templates → Satis rebuild → composer plugin version bump
+
 ## 2026-05-21
 
 ### Feature - `pantheon-post-deploy-drush` switches to `drush deploy` (v3.2.0)