Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions .github/actions/check-smoke-status/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,13 @@ inputs:
set-matrix-result:
description: "`needs.set-matrix.result`."
required: true
resolvers:
description: >-
Extra version-resolver jobs the suites depend on, one per line as
`<label>|<needs.JOB.result>`, checked like `set-matrix-result`, e.g. a
`Connect version resolver` label with `needs.connect-version.result`.
required: false
default: ""
relevant:
description: "`needs.changes.outputs.relevant`: `true` when this run's paths put the suite in scope."
required: true
Expand All @@ -39,6 +46,7 @@ runs:
PRODUCT: ${{ inputs.product }}
CHANGES_RESULT: ${{ inputs.changes-result }}
SET_MATRIX_RESULT: ${{ inputs.set-matrix-result }}
RESOLVERS: ${{ inputs.resolvers }}
RELEVANT: ${{ inputs.relevant }}
SUITES: ${{ inputs.suites }}
IN_SCOPE_HINT: ${{ inputs.in-scope-hint }}
Expand All @@ -53,6 +61,13 @@ runs:
echo "Version-matrix job failed; cannot confirm smoke scope"
exit 1
fi
while IFS='|' read -r LABEL RESULT; do
[ -n "${LABEL}" ] || continue
if [ "${RESULT}" = "failure" ] || [ "${RESULT}" = "cancelled" ]; then
echo "${LABEL} failed; cannot confirm smoke scope"
exit 1
fi
done <<< "${RESOLVERS}"
while IFS='|' read -r LABEL RESULT; do
[ -n "${LABEL}" ] || continue
# If a suite was in scope for this PR (paths matched) it MUST have
Expand Down
8 changes: 7 additions & 1 deletion .github/actions/write-smoke-summary/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,10 @@ inputs:
product:
description: "Product name for the heading and the URL row, e.g. `Connect`."
required: true
title:
description: "Heading text before the version. Defaults to `🧪 <product> Smoke Tests`."
required: false
default: ""
requested-version:
description: "The version the matrix asked for."
required: true
Expand All @@ -31,14 +35,16 @@ runs:
shell: bash
env:
PRODUCT: ${{ inputs.product }}
TITLE: ${{ inputs.title }}
REQUESTED: ${{ inputs.requested-version }}
RESOLVED: ${{ inputs.resolved-version }}
PRODUCT_URL: ${{ inputs.url }}
RESULTS_FILE: ${{ inputs.results-file }}
run: |
RUN_DATE=$(date -u '+%Y-%m-%d %H:%M UTC')
TITLE=${TITLE:-"🧪 ${PRODUCT} Smoke Tests"}
{
echo "## 🧪 ${PRODUCT} Smoke Tests — v${RESOLVED}"
echo "## ${TITLE} — v${RESOLVED}"
echo ""
echo "| | |"
echo "|---|---|"
Expand Down
52 changes: 8 additions & 44 deletions .github/workflows/connect-integration.yml
Original file line number Diff line number Diff line change
Expand Up @@ -129,52 +129,16 @@ jobs:
results.json
if-no-files-found: ignore

# Write a rich job summary with test details and emoji
- name: Write workflow summary
if: always()
run: |
RESOLVED="${{ steps.version.outputs.resolved }}"
CONNECT_URL="${{ steps.connect.outputs.CONNECT_SERVER }}"
REQUESTED="${{ matrix.connect-version }}"
RUN_DATE=$(date -u '+%Y-%m-%d %H:%M UTC')
{
echo "## 🔗 Connect Integration Tests — v${RESOLVED}"
echo ""
echo "| | |"
echo "|---|---|"
echo "| 🏷️ **Requested version** | \`${REQUESTED}\` |"
echo "| ✅ **Resolved version** | \`${RESOLVED}\` |"
echo "| 🌐 **Connect URL** | \`${CONNECT_URL}\` |"
echo "| 📅 **Run date** | ${RUN_DATE} |"
echo ""
} >> "$GITHUB_STEP_SUMMARY"
if [ -f connect-results.xml ]; then
TESTS=$(grep -oP '(?<=tests=")[^"]+' connect-results.xml | head -1 || echo 0)
FAILURES=$(grep -oP '(?<=failures=")[^"]+' connect-results.xml | head -1 || echo 0)
ERRORS=$(grep -oP '(?<=errors=")[^"]+' connect-results.xml | head -1 || echo 0)
SKIPPED=$(grep -oP '(?<=skipped=")[^"]+' connect-results.xml | head -1 || echo 0)
TESTS=${TESTS:-0}; FAILURES=${FAILURES:-0}; ERRORS=${ERRORS:-0}; SKIPPED=${SKIPPED:-0}
PASSED=$(( TESTS - FAILURES - ERRORS - SKIPPED ))
{
echo "### 📊 Test Results"
echo ""
echo "| Result | Count |"
echo "|--------|-------|"
echo "| ✅ Passed | ${PASSED} |"
echo "| ❌ Failed | ${FAILURES:-0} |"
echo "| ⚠️ Errors | ${ERRORS:-0} |"
echo "| ⏭️ Skipped | ${SKIPPED:-0} |"
echo "| 📝 **Total** | **${TESTS:-0}** |"
echo ""
if [ "${FAILURES:-0}" = "0" ] && [ "${ERRORS:-0}" = "0" ]; then
echo "🎉 **All tests passed!**"
else
echo "💥 **Some tests failed — check the logs for details.**"
fi
} >> "$GITHUB_STEP_SUMMARY"
else
echo "> ⚠️ Test results file not found — tests may have been skipped." >> "$GITHUB_STEP_SUMMARY"
fi
uses: ./.github/actions/write-smoke-summary
with:
product: Connect
title: 🔗 Connect Integration Tests
requested-version: ${{ matrix.connect-version }}
resolved-version: ${{ steps.version.outputs.resolved }}
url: ${{ steps.connect.outputs.CONNECT_SERVER }}
results-file: connect-results.xml

# Stop Connect
- name: Stop Connect
Expand Down
42 changes: 14 additions & 28 deletions .github/workflows/mock-idp-e2e.yml
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ on:
- 'Dockerfile'
- '.github/workflows/mock-idp-e2e.yml'
- '.github/workflows/recent-production-versions.yml'
- '.github/actions/check-smoke-status/**'
pull_request:
schedule:
- cron: "30 5 * * *" # daily at 5:30am UTC
Expand Down Expand Up @@ -77,6 +78,7 @@ jobs:
- 'Dockerfile'
- '.github/workflows/mock-idp-e2e.yml'
- '.github/workflows/recent-production-versions.yml'
- '.github/actions/check-smoke-status/**'

# Emit the Workbench version list for the E2E job. PR/push runs the current
# release only; scheduled/manual runs fan out across the Workbench support
Expand Down Expand Up @@ -524,35 +526,19 @@ jobs:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
# The E2E job needs both resolvers: the Workbench matrix and the pinned
# Connect tag. A failed resolver skips (not fails) it, so both are checked.
- name: Check result
run: |
if [ "${{ needs.changes.result }}" = "failure" ] || [ "${{ needs.changes.result }}" = "cancelled" ]; then
echo "Change-detection job failed; cannot confirm E2E scope"
exit 1
fi
# A failed version resolver skips (not fails) mock-idp-e2e, which the
# check below would otherwise read as a clean skip. Both resolvers
# count: the E2E job needs the Workbench matrix and the pinned
# Connect tag.
if [ "${{ needs.set-matrix.result }}" = "failure" ] || [ "${{ needs.set-matrix.result }}" = "cancelled" ]; then
echo "Version-matrix job failed; cannot confirm E2E scope"
exit 1
fi
if [ "${{ needs.connect-version.result }}" = "failure" ] || [ "${{ needs.connect-version.result }}" = "cancelled" ]; then
echo "Connect version resolver failed; cannot confirm E2E scope"
exit 1
fi
# If auth paths changed, the E2E suite was in scope and MUST have
# succeeded — a skip means it was gated out, not legitimately absent.
if [ "${{ needs.changes.outputs.relevant }}" = "true" ] && [ "${{ needs.mock-idp-e2e.result }}" != "success" ]; then
echo "Mock-IdP E2E was in scope but did not succeed (result=${{ needs.mock-idp-e2e.result }})"
exit 1
fi
if [ "${{ needs.mock-idp-e2e.result }}" = "failure" ] || [ "${{ needs.mock-idp-e2e.result }}" = "cancelled" ]; then
echo "Mock-IdP E2E tests failed or were cancelled"
exit 1
fi
echo "Mock-IdP E2E tests passed or were legitimately out of scope"
uses: ./.github/actions/check-smoke-status
with:
product: Mock-IdP E2E
changes-result: ${{ needs.changes.result }}
set-matrix-result: ${{ needs.set-matrix.result }}
resolvers: |
Connect version resolver|${{ needs.connect-version.result }}
relevant: ${{ needs.changes.outputs.relevant }}
suites: |
Mock-IdP E2E tests|${{ needs.mock-idp-e2e.result }}

# Nightly failures have no PR author to notice them; file (or update) a
# tracking issue instead of failing silently. Never fires on push/pull_request
Expand Down
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -349,7 +349,7 @@ Every render also produces `_output/vip-report.pdf` from `report/vip-report.qmd`
- **Shared CI building blocks** -- the smoke workflows (and `connect-integration.yml` and `example-report.yml` where noted) share their repeated steps through local composite actions under `.github/actions/` and two scripts under `scripts/ci/`. A composite cannot read `secrets.*` or `needs`, so callers pass values in as inputs, and every `run:` step in one needs an explicit `shell: bash` (which adds `-eo pipefail`, unlike a workflow's implicit shell). A job that calls one needs `actions/checkout` first, and each smoke workflow lists the composites and scripts it uses in its push `paths:` and its `changes` filter so editing one reruns the suite. Third-party actions inside composites keep the same full-SHA pins zizmor requires.
- `check-license-errors` -- the `if: failure()` step that reads the product container's logs for an expired or invalid license and surfaces it as an annotation and job summary. Takes the product, the secret's name, and either `image` (find the container by image ancestor, which needs the tag) or `container` (a fixed name, as Package Manager uses). Used by Connect, Workbench and Package Manager in the smoke workflows, by `connect-integration.yml`, and by `example-report.yml`.
- `setup-vip` -- `setup-uv`, `uv sync --frozen`, the Playwright-version-keyed browser cache and `vip install --skip-system`. Used by the three smoke workflows, `connect-integration.yml` and `example-report.yml`. The Workbench API-auth job keeps its own setup because it deliberately installs no browser.
- `write-smoke-summary` -- the job summary table and JUnit pass/fail counts. `rename-smoke-job` -- renames the matrix job's check run to the resolved version; it finds the job by the `Smoke test against` prefix plus the requested version, so a smoke job must keep that name. `check-smoke-status` -- the scope-aware rule behind each `* Status` job, which keeps its own `needs:` and passes the results in. All three are used by the three smoke workflows.
- `write-smoke-summary` -- the job summary table and JUnit pass/fail counts. `rename-smoke-job` -- renames the matrix job's check run to the resolved version; it finds the job by the `Smoke test against` prefix plus the requested version, so a smoke job must keep that name. `check-smoke-status` -- the scope-aware rule behind each `* Status` job, which keeps its own `needs:` and passes the results in; an optional `resolvers` input covers extra version jobs, like `mock-idp-e2e.yml`'s `connect-version`. All three are used by the three product smoke workflows. `connect-integration.yml` also uses `write-smoke-summary` (with its own `title`), and `mock-idp-e2e.yml` also uses `check-smoke-status`.
- `scripts/ci/wait-for-package-manager.sh` (`packagemanager-smoke.yml`, `example-report.yml`) and `scripts/ci/resolve-connect-version.sh` (`connect-smoke.yml`, `connect-integration.yml`) hold the two retry loops that were copied between workflows.
- **`connect-integration.yml`** -- push to `main`, `schedule` (daily), and `workflow_dispatch`; deliberately no `pull_request` trigger and not a required check (decided in #421). Runs the full VIP Connect category (`vip verify --api-auth --categories connect`) against two ephemeral Connect servers via `with-connect`: the current production release (resolved by `recent-production-versions.yml`) and `ghcr.io/posit-dev/connect-preview:daily`, the nightly build, so an unreleased upstream build never blocks a PR. It's a drift detector, not a gate: failures post to Slack via the `SLACK_WEBHOOK_CONNECT_CI` secret instead of failing anyone's merge.
- **`install-flow-smoke.yml`** -- push to `main` or PR (paths-gated to `src/vip/install/**`, `src/vip/cli/**`, `pyproject.toml`, `uv.lock`) or `workflow_dispatch`; reproduces the documented default install flow from the README (`uv tool install posit-vip` then `vip install`) on Ubuntu-as-root and macOS. Distinct from the product smoke workflows above: `uv tool install` only puts vip's own entry point on `PATH`, a topology every other install test misses because it runs `uv run vip install` from inside the project venv.
Expand Down
Loading