Skip to content

ci: stop swallowing integration test output on failure - #104

Merged
GuanyiLi-Craig merged 1 commit into
mainfrom
fix/ci-integration-output-swallowing
Jun 23, 2026
Merged

ci: stop swallowing integration test output on failure#104
GuanyiLi-Craig merged 1 commit into
mainfrom
fix/ci-integration-output-swallowing

Conversation

@GuanyiLi-Craig

Copy link
Copy Markdown
Contributor

Problem

When an integration test fails in CI, the job log shows nothing between ##[endgroup] and ##[error]Process completed with exit code 1 — all the diagnostic output (failed test names, tracebacks, API errors) is discarded. This made the 0.0.36 release failure impossible to diagnose from the logs.

Root cause

Each integration step runs:

set -e
output=$(uv run python tests_integration/.../run_*.py)   # stdout captured into $output
echo "$output"                                            # only printed on the NEXT line
if echo "$output" | grep -q "Failed:"; then ...

The runner prints all per-test detail to stdout, which the command substitution captures into $output. With set -e, a non-zero exit from the runner aborts the step at the assignment line, so echo "$output" never executes and the captured output is thrown away.

Fix

Capture the runner's exit status explicitly, always echo the output, and treat a non-zero exit as failure (in addition to the existing "Failed:" grep — the runner already exit 1s on any failed example, so the exit code is the more reliable signal):

set +e
output=$(uv run python tests_integration/.../run_*.py)
status=$?
set -e
echo "$output"
if [ "$status" -ne 0 ] || echo "$output" | grep -q "Failed:"; then ...

Applied uniformly to all 12 integration-test steps.

Notes

This is a pure CI-observability fix — no test or framework code changes. It does not change which tests run or pass; it only ensures failures are visible in the logs.

🤖 Generated with Claude Code

Each integration step ran the test runner under `set -e` with
`output=$(uv run python ...)`. When the runner exits non-zero, `set -e`
aborts the step at the capture line, so the subsequent `echo "$output"`
never runs and all failure detail (failed test names, tracebacks, API
errors) is discarded. Logs jump straight from `##[endgroup]` to
`exit code 1`, making failures impossible to diagnose.

Capture the runner's exit status explicitly (`set +e` around the call),
always echo the captured output, and treat a non-zero exit as failure
in addition to the existing "Failed:" grep. The runner already returns
exit code 1 on any failed example, so this is the more reliable signal.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings June 23, 2026 19:02

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@GuanyiLi-Craig
GuanyiLi-Craig merged commit 45f0089 into main Jun 23, 2026
19 of 20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants