Skip to content

Run benchmark workflows on pull_request instead of pull_request_target#4664

Open
tautschnig wants to merge 2 commits into
model-checking:mainfrom
tautschnig:bench-on-pull-request
Open

Run benchmark workflows on pull_request instead of pull_request_target#4664
tautschnig wants to merge 2 commits into
model-checking:mainfrom
tautschnig:bench-on-pull-request

Conversation

@tautschnig

Copy link
Copy Markdown
Member

Since 2026-07-20, the label-gated benchmark jobs (End-to-End Benchmarks / perf-benchcomp, Compiler Benchmarks / compile-timer-{short,long}) fail within seconds on every pull request from a fork, while pushes to main stay green.

Root cause: actions/checkout v6.1.0 was published on 2026-07-20 (with coordinated backports v5.1.0 and v4.4.0) and made the floating v6 tag refuse to check out fork PR code from workflows running in the trusted pull_request_target context (changelog):

##[error]Refusing to check out fork pull request code from a 'pull_request_target' workflow. [...]

Our benchmark workflows are invoked via workflow_call from extra_jobs.yml, which runs on pull_request_target (the auto-labeler needs pull-requests: write). They check out github.event.pull_request.head.sha — fork code — and then build and execute it, which is precisely the pattern the new default blocks. Pushes to main run the same workflows via the push trigger comparing trusted before/after refs, hence unaffected.

Fix: rather than opting out via allow-unsafe-pr-checkout: true, move the benchmark invocations to a new bench-pr.yml workflow triggered by plain pull_request events, so fork code builds and runs in the untrusted, fork-scoped context as GitHub recommends. Label gating (Z-EndToEndBenchCI / Z-CompilerBenchCI) is preserved, now reading the labels from the event payload instead of the labeler job's outputs. The event-name conditions inside the reusable workflows are simplified accordingly, and extra_jobs.yml keeps only the auto-labeler.

Behavioral difference: labels added by the auto-labeler do not re-trigger workflows (events created with the default GITHUB_TOKEN never do), so on freshly opened PRs the benchmarks only start with the next push, or when someone adds the label manually (human-added labels do trigger via the labeled event type). Subsequent pushes to labeled PRs — the common case — behave exactly as before. A concurrency group cancels superseded benchmark runs.

Validated with actionlint 1.7.7 (clean) on all four touched workflow files.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 and MIT licenses.

actions/checkout v6.1.0 (published 2026-07-20, with backports v5.1.0 and
v4.4.0) refuses to check out fork pull-request code from workflows
running in the trusted pull_request_target context:

  Refusing to check out fork pull request code from a
  'pull_request_target' workflow. [...] Fetching and executing a fork's
  code in that trusted context commonly leads to "pwn request"
  vulnerabilities.

Since the floating v6 tag moved to v6.1.0, the label-gated benchmark
jobs (invoked from extra_jobs.yml, which runs on pull_request_target
because the auto-labeler needs write permissions) fail at their
checkout step for every pull request from a fork: they check out and
then build and execute the proposed code, which is exactly the pattern
the new default blocks. Pushes to main are unaffected since the same
workflows then compare trusted before/after refs.

Rather than opting out via allow-unsafe-pr-checkout, move the
benchmark invocations to a new workflow triggered by plain pull_request
events, where fork code runs in the untrusted, fork-scoped context as
GitHub recommends. Gating still uses the Z-EndToEndBenchCI and
Z-CompilerBenchCI labels, now read from the event payload instead of
the labeler job's outputs.

One behavioral difference: labels added by the auto-labeler do not
re-trigger workflows (events created with the default GITHUB_TOKEN
never do), so on freshly opened pull requests the benchmarks only start
with the next push to the pull request, or when a label is added
manually. Subsequent pushes to labeled pull requests (the common case)
behave as before.

Co-authored-by: Kiro <kiro-agent@users.noreply.github.com>
@tautschnig
tautschnig requested a review from a team as a code owner July 21, 2026 19:38
Copilot AI review requested due to automatic review settings July 21, 2026 19:38

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR moves the label-gated benchmark runs off the trusted pull_request_target path (used for auto-labeling) and onto a dedicated pull_request-triggered workflow, aligning with the new actions/checkout safety behavior and avoiding running fork PR code in a trusted context.

Changes:

  • Remove benchmark reusable-workflow invocations from extra_jobs.yml, leaving only the minimal auto-labeler on pull_request_target.
  • Add bench-pr.yml to run the benchmark reusable workflows on pull_request events when the corresponding Z-*BenchCI label is present.
  • Simplify PR-ref selection logic in the benchmark reusable workflows (but see below: the current condition breaks workflow_call use).

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File Description
.github/workflows/extra_jobs.yml Stops invoking benchmark workflows from the trusted pull_request_target labeler workflow.
.github/workflows/bench-pr.yml New pull_request workflow that label-gates and calls the reusable benchmark workflows with concurrency cancellation.
.github/workflows/bench-e2e.yml Updates PR ref-selection condition for setting OLD_REF/NEW_REF (currently problematic under workflow_call).
.github/workflows/bench-compiler.yml Same ref-selection condition update in both compiler benchmark jobs (currently problematic under workflow_call).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread .github/workflows/bench-e2e.yml
Comment thread .github/workflows/bench-compiler.yml
Comment thread .github/workflows/bench-compiler.yml
In reusable workflows, github.event_name and github.event are those of
the caller workflow ('workflow_call' is never a runtime event name), so
the pull_request condition holds when invoked from bench-pr.yml. Add a
comment to preempt confusion.

Co-authored-by: Kiro <kiro-agent@users.noreply.github.com>
@tautschnig

Copy link
Copy Markdown
Member Author

We need to get the changes in #4666 merged first to unblock the CI jobs.

@feliperodri

Copy link
Copy Markdown
Contributor

@tautschnig I just approved #4666 and got it into the merge queue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants