Skip to content

ci: List job failures outside tests in the weekly flaky report - #26460

Merged
s1gr1d merged 2 commits into
sig/test-cloudflare-request-errorsfrom
sig/ci-weekly-report-job-failures
Nov 17, 2026
Merged

s1gr1d merged 2 commits into
sig/test-cloudflare-request-errorsfrom
sig/ci-weekly-report-job-failures

Conversation

@s1gr1d

@s1gr1d s1gr1d commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

What

The weekly flaky test report has a second table. It lists the jobs that failed without a test failure, by job and step, with the result "failed" or "timed out":

Job Step Result Affected runs
Bun Unit Tests Set up Bun failed 2
Check lockfile Check that pnpm-lock.yaml is stable failed 1
Lint Check for circular dependencies timed out 1

Why

The report only printed a count for these jobs, for example "155 failed job attempts; 129 without test annotations". In the last month, they were about half of the failed develop runs: hung Playwright installs, lockfile checks, job timeouts. GitHub reports a job that reached timeout-minutes as cancelled, so the report did not see those jobs at all.

The gate job is excluded, because it fails when any other job fails. No workflow runs pnpm test:scripts, so the new test file only runs locally.

Third PR of a stack of four.

The report only counted the failed jobs without a test annotation. In
one month, these were about half of the failed develop runs: failed
setup steps, and jobs that reached their time limit, which GitHub
reports as cancelled.

The report now lists them by job and step. The gate job is excluded,
because it only repeats the failure of another job.
@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

size-limit report 📦

Compared: 18fd907 (PR test merge using develop at f9f479a)

Against: 9b7265f (develop) · base run #66464

⚠️ Baseline mismatch: 7 develop commit(s) in the PR build are missing from the baseline; 8 develop commit(s) in the baseline are missing from the PR build. Their size changes also affect this comparison. Compare commits. Rebase your PR onto the latest develop to compare against an up-to-date baseline.

Path Size % Change Change
@sentry/browser 30.1 kB - -
@sentry/browser - with treeshaking flags 28.3 kB - -
@sentry/browser - with treeshaking flags tracing without tracing 28.2 kB - -
@sentry/browser (incl. Tracing) 52.28 kB - -
@sentry/browser (incl. Tracing + Span Streaming) 52.29 kB - -
@sentry/browser (incl. Tracing, Profiling) 55.25 kB - -
@sentry/browser (incl. Tracing, Replay) 92.82 kB - -
@sentry/browser (incl. Tracing, Replay) - with treeshaking flags 81.6 kB - -
@sentry/browser (incl. Tracing, Replay with Canvas) 97.21 kB - -
@sentry/browser (incl. Tracing, Replay, Feedback) 110.6 kB - -
@sentry/browser (incl. Feedback) 47.73 kB - -
@sentry/browser (incl. sendFeedback) 30.69 kB - -
@sentry/browser (incl. FeedbackAsync) 35.69 kB - -
@sentry/browser (incl. Metrics) 31.43 kB - -
@sentry/browser (incl. Logs) 31.71 kB - -
@sentry/browser (incl. Metrics & Logs) 32.37 kB - -
@sentry/react 32.11 kB - -
@sentry/react (incl. Tracing) 54.95 kB - -
@sentry/vue 38.29 kB - -
@sentry/vue (incl. Tracing) 55.24 kB - -
@sentry/svelte 30.12 kB - -
@sentry/remix (Remix 3 client bundle) 58.06 kB - -
CDN Bundle 31.87 kB - -
CDN Bundle (incl. Tracing) 52.7 kB - -
CDN Bundle (incl. Logs, Metrics) 34.67 kB - -
CDN Bundle (incl. Tracing, Logs, Metrics) 55.26 kB - -
CDN Bundle (incl. Replay, Logs, Metrics) 75.7 kB - -
CDN Bundle (incl. Tracing, Replay) 90.72 kB - -
CDN Bundle (incl. Tracing, Replay, Logs, Metrics) 93.33 kB - -
CDN Bundle (incl. Tracing, Replay, Feedback) 96.09 kB - -
CDN Bundle (incl. Tracing, Replay, Feedback, Logs, Metrics) 98.69 kB - -
CDN Bundle - uncompressed 92.98 kB - -
CDN Bundle (incl. Tracing) - uncompressed 155.45 kB - -
CDN Bundle (incl. Logs, Metrics) - uncompressed 101.33 kB - -
CDN Bundle (incl. Tracing, Logs, Metrics) - uncompressed 163.08 kB - -
CDN Bundle (incl. Replay, Logs, Metrics) - uncompressed 231.44 kB - -
CDN Bundle (incl. Tracing, Replay) - uncompressed 275.61 kB - -
CDN Bundle (incl. Tracing, Replay, Logs, Metrics) - uncompressed 283.24 kB - -
CDN Bundle (incl. Tracing, Replay, Feedback) - uncompressed 287.4 kB - -
CDN Bundle (incl. Tracing, Replay, Feedback, Logs, Metrics) - uncompressed 295.02 kB - -
@sentry/nextjs (client) 58.57 kB - -
@sentry/sveltekit (client) 53.57 kB - -
@sentry/core/server 44.8 kB - -
@sentry/core/browser 14.4 kB - -
@sentry/node 151.36 kB -0.21% -309 B 🔽
@sentry/node/import (ESM hook with diagnostics-channel injection) 83.3 kB -0.04% -31 B 🔽
@sentry/node - without tracing 88.56 kB -0.04% -35 B 🔽
@sentry/node - without channel injection 140.3 kB -0.23% -318 B 🔽
@sentry/aws-serverless 98.93 kB -0.04% -38 B 🔽
@sentry/cloudflare (withSentry) - minified 206.47 kB -0.12% -240 B 🔽
@sentry/cloudflare (withSentry) 479.22 kB -0.13% -593 B 🔽
@sentry/nextjs/cloudflare (withSentry) - minified 231.83 kB -0.11% -240 B 🔽

View base workflow run

@s1gr1d
s1gr1d added this pull request to stack #26475 November 17, 2026 12:41
@s1gr1d
s1gr1d marked this pull request as ready for review November 17, 2026 12:51

@nicohrubec nicohrubec left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need the tests? else looks good

The tests that stay cover the logic that is easy to break: a failed job without a test
annotation is listed with its failed step, a job that reached its time limit is listed as
timed out, and a job that was cancelled for another reason is not listed. The tests for the
gate job, the E2E job grouping, the fallback step name, and the rendered table go away.
@s1gr1d

s1gr1d commented Nov 17, 2026

Copy link
Copy Markdown
Member Author

I reduced the number of tests and only kept the most important

@github-actions

Copy link
Copy Markdown
Contributor

Optional E2E tests

All optional checks were successful. 2 successful. See the run.

Successful checks
Check Status
✅ CI: Optional E2E Tests / E2E tanstackstart-react-cloudflare Test (optional) Successful in 35s Details
✅ CI: Optional E2E Tests / E2E tanstackstart-react (latest) Test (optional) Successful in 1m Details

@s1gr1d
s1gr1d merged commit a1e62ed into develop Nov 17, 2026
52 checks passed
@s1gr1d
s1gr1d deleted the sig/ci-weekly-report-job-failures branch November 17, 2026 13:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants