Skip to content

fix: enforce pending timeout when build cluster is unreachable - #1

Open
Prucek wants to merge 203 commits into
mainfrom
fix-pending-timeout-unreachable-cluster
Open

fix: enforce pending timeout when build cluster is unreachable#1
Prucek wants to merge 203 commits into
mainfrom
fix-pending-timeout-unreachable-cluster

Conversation

@Prucek

@Prucek Prucek commented Jul 29, 2026

Copy link
Copy Markdown
Owner

Summary

  • Enforce pending timeout for ProwJobs when the build cluster is unreachable, preventing jobs from staying in pending state indefinitely
  • When a pod lookup fails due to an unreachable cluster and the pending timeout has elapsed, the job is now marked as errored
  • Added comprehensive test coverage for the new timeout behavior with unreachable clusters

vladikkuzn and others added 30 commits March 4, 2026 18:10
Bumps [github.com/tektoncd/pipeline](https://github.com/tektoncd/pipeline) from 1.6.0 to 1.6.1.
- [Release notes](https://github.com/tektoncd/pipeline/releases)
- [Changelog](https://github.com/tektoncd/pipeline/blob/main/releases.md)
- [Commits](tektoncd/pipeline@v1.6.0...v1.6.1)

---
updated-dependencies:
- dependency-name: github.com/tektoncd/pipeline
  dependency-version: 1.6.1
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
…t/go_modules/github.com/tektoncd/pipeline-1.6.1

chore(deps): bump github.com/tektoncd/pipeline from 1.6.0 to 1.6.1
Current project guideance is to not include the year in copyright
headers. This updates the verify-boilerplate check to allow with or
without the year in the header.

Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
This adds scripts and configuration to be able to easily spin up a local
development cluster. It leverages a lot of the infrastructure in place
for integration testing to be able to get the same environment for
running and manually testing code changes without needing to do a full
deployment.

Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
To be able to quickly iterate on changes and try them out, this
introduces Tilt to the local development workflow as an optional way to
speed up this inner loop development. Once a local dev cluster has been
created, users can run `make dev-tilt` or `tilt up` to get tilt managing
the dev cluster and changes to the code for any of the services running
in the cluster.

It defaults to only running the "core" services, but also has a facility
for users to drop in their own custom configuration to be able to do
things like adding the particular non-core thing they are working on.

Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
To send "phony" events the user currently needs to retrieve an HMAC
token, then send the event. This adds a helper script to make that less
confusing and error prone.

Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
Signed-off-by: Amulyam24 <amulmek1@in.ibm.com>
plugins: add new plugin to pin and unpin issues to a repository
…ncements

Enhancements for link issue plugin
…etes-sigs#656)

* feat: check added to detect fixup amend commits

* fix: consolidate to single label and add squash! detection

* fix: aggregate checks into a single comment and simplify label handling

* fix: minor code refactoring
Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.75.0 to 1.79.3.
- [Release notes](https://github.com/grpc/grpc-go/releases)
- [Commits](grpc/grpc-go@v1.75.0...v1.79.3)

---
updated-dependencies:
- dependency-name: google.golang.org/grpc
  dependency-version: 1.79.3
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
…t/go_modules/google.golang.org/grpc-1.79.3

chore(deps): bump google.golang.org/grpc from 1.75.0 to 1.79.3
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
The trusted_apps configuration was not being applied when both
SkipDCOCheckForMembers and SkipDCOCheckForCollaborators were false.
The filterTrustedUsers function (which checks trusted_apps via
trigger.TrustedUser) was only called when one of these skip flags
was enabled, making trusted_apps ineffective on its own.

Add len(config.TrustedApps) > 0 to the condition so filterTrustedUsers
runs whenever trusted apps are configured, regardless of skip flags.

Fixes kubernetes-sigs#606

Signed-off-by: vigneshakaviki <kumarvignesh295@gmail.com>
Add helpers to reduce friction local development
GitHub recommends using the X-Hub-Signature-256 header (HMAC-SHA256)
for webhook validation instead of X-Hub-Signature (HMAC-SHA1), which
is kept only for legacy purposes. Since GitHub already sends both
headers when a webhook secret is configured, this is a safe in-place
algorithm swap with no configuration changes needed on the GitHub side.

- Switch crypto/sha1 to crypto/sha256 in ValidatePayload and
  PayloadSignature
- Read X-Hub-Signature-256 header instead of X-Hub-Signature in
  ValidateWebhook
- Update phony test sender to set the new header
- Recompute all test HMAC signatures using SHA256
- Add test case using GitHub's official HMAC-SHA256 test vector

Assisted-by: Claude:claude-opus-4-6
Missed this test file in the initial SHA1→SHA256 migration. The
githubeventserver package has its own ServeHTTP test that also
references X-Hub-Signature and SHA1 HMAC signatures.

Assisted-by: Claude:claude-opus-4-6
In contrast to the GCS range reader, the S3 range reader returns `io.EOF` at the end
of the desired range. This is compliant with the `io.Reader` specification.
However, the `StorageArtifact.ReadAt` function expects `io.EOF` only when reading up
to the overall end of the artifact.

This leads to problems when expanding skipped lines in an S3-based spyglass build
log. Expanding the skipped lines at the end of the build log works, but expanding
lines at the start or in the middle results in:
```
Failed to retrieve log lines: couldn't read requested bytes: error reading from artifact: EOF
```
mergeRebase runs `git rebase --no-stat <upstream> <branch>`, which
implicitly checks out <branch> before rebasing. When the rebase fails
and is aborted, `git rebase --abort` restores HEAD to <branch> — not
to the pre-rebase HEAD. This leaves callers with HEAD on the wrong
commit, causing silent failures on retry (rebase becomes a no-op when
both arguments resolve to the same SHA).

Add a Checkout(headRev) after successful abort to restore HEAD to
where the caller left it.
…ort-head-restore

Restore HEAD after failed rebase abort in mergeRebase
Removed Kyma link from the overview documentation.
Migrate webhook validation from HMAC-SHA1 to HMAC-SHA256
GitHub App authentication requires a POST request to
/app/installations/{id}/access_tokens to acquire installation tokens,
which are needed for all subsequent API calls including reads. The
existing dry-run mode blocked all non-GET requests, making it impossible
to use dry-run with GitHub App auth.

Add an allowInDryRun flag on the request struct with a hardcoded
allowlist in isDryRunAllowed() that permits only the token acquisition
endpoint. This keeps all actual mutations blocked while allowing the
auth flow to complete.
kubernetes-prow Bot and others added 27 commits June 30, 2026 14:36
…y-records

deck: filter tide history per-record instead of per-pool
…d-commits

git: add SSH commit signing support to the git client factory
…omments, and broaden /override-cancel

- Extract isAuthorized() helper to deduplicate auth logic between handle and handleOverrideCancel
- Handle multiple command types (/override, /override-sticky, /override-cancel) in a single comment
- Make /override-cancel work on both regular and sticky overrides by checking "Overridden by" prefix
- Use correct command name (/override vs /override-sticky) in all error messages
- Update /override-cancel help text to clarify it sets status back to failure
- Add test guarding that sticky descriptions fit within GitHub's 140-char limit
…y validation

validateTideContextPolicy collects branches from job Brancher.Branches
fields, which contain regexes like ^master$. It passes these as literal
branch names to GetTideContextPolicy, which looks up BP config by branch
name. BP config uses literal names like "master", so the lookup silently
fails and context policy collisions between BP-required contexts and
conditionally-triggered jobs are never detected.

Fix by also collecting branches explicitly configured in the
BranchProtection config. These are literal branch names that correctly
match BP lookups.

Ref: kubernetes-sigs#777
…p-branches

checkconfig: include branch-protection branches in tide context policy validation
…licy validation

Branches explicitly configured in tide context_options were not included
in the set of branches validated by validateTideContextPolicy. This meant
a conflict between an optional job and a required-if-present context
defined in context_options for that branch would go undetected.
…ide-context-optional-conflict

checkconfig: include tide context_options branches in tide context policy validation
Ensure that the incoming username actually has the `[bot]` suffix before trimming it and verifying it against the `trusted_apps` list. 

This prevents regular users from registering the base name of a trusted app (e.g., `copybara-service` without the `[bot]` suffix) and gaining unauthorized trusted status.
fix(trigger): verify [bot] suffix before trimming trusted app name
…-cherry-pick

Automate adding kind/* labels to cherry-pick PRs
Extract 'Overridden by' string literal into overrideDescriptionPrefix
constant to keep description() and cancel logic in sync. Add explicit
parentheses in tide baseSHA condition to clarify operator precedence.
…r-head

Add /override-sticky command for persistent overrides
…otification

Per sig-release policy confirmed in the release management Slack discussion
(July 2025), kind/bug PRs may be included in a release during Code Freeze
without requiring full approval — they just need to notify the release team
by tagging @kubernetes/sig-release-leads on the PR.

Update the plugin to:
- Show a softer message for kind/bug PRs: permitted, but please tag the
  release team on GitHub so there's a record on the PR.
- Update the non-bug message to mention GitHub tagging in addition to Slack,
  as explicitly requested by the release team leads.

Signed-off-by: Davanum Srinivas <davanum@gmail.com>
Signed-off-by: Davanum Srinivas <davanum@gmail.com>
Signed-off-by: Davanum Srinivas <davanum@gmail.com>
…d-bug-in-code-freeze

testfreeze: allow kind/bug PRs during code freeze with release team notification
Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.50.0 to 0.52.0.
- [Commits](golang/crypto@v0.50.0...v0.52.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-version: 0.52.0
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.50.0 to 0.52.0.
- [Commits](golang/crypto@v0.50.0...v0.52.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-version: 0.52.0
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
…t/go_modules/hack/tools/golang.org/x/crypto-0.52.0

chore(deps): bump golang.org/x/crypto from 0.50.0 to 0.52.0 in /hack/tools
…t/go_modules/golang.org/x/crypto-0.52.0

chore(deps): bump golang.org/x/crypto from 0.50.0 to 0.52.0
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.54.0 to 0.55.0.
- [Commits](golang/net@v0.54.0...v0.55.0)

---
updated-dependencies:
- dependency-name: golang.org/x/net
  dependency-version: 0.55.0
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
We use dependabot for keeping our Go dependencies up to date. By default
this will propose a PR for each dependency update it finds. There are
some sets of dependencies, like the regularly generated AWS clients and
the golang.org/x deps, that are usually released at the same time. With
the default setting of proposing a single PR for each update, this will
often lead to the first PR in the group causing a merge conflict with
any others in the group, resulting in a lot of extra churn.

This adds the dependabot config to group some of these common sets of
libraries so dependabot will collect all updates within the group and
propose them as a single PR, avoiding merge conflicts and reducing
review and CI overhead.

Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
Bumps gopkg.in/ini.v1 from 1.67.0 to 1.67.3.

---
updated-dependencies:
- dependency-name: gopkg.in/ini.v1
  dependency-version: 1.67.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
…t/go_modules/gopkg.in/ini.v1-1.67.3

chore(deps): bump gopkg.in/ini.v1 from 1.67.0 to 1.67.3
…t/go_modules/golang.org/x/net-0.55.0

chore(deps): bump golang.org/x/net from 0.54.0 to 0.55.0
@Prucek
Prucek force-pushed the fix-pending-timeout-unreachable-cluster branch from 6e36518 to 3ac48b9 Compare August 3, 2026 11:39
When a build cluster becomes unreachable, syncPendingJob enters a retry
loop that never terminates. The pod lookup (informer cache read) returns
not-found, startPod attempts client.Create which fails with a network
error, and because isRequestError only matches 4xx API responses, the
error is returned for requeue — indefinitely.

Check the ProwJob's pendingTime against pod_pending_timeout when
startPod fails with a non-request error. If the timeout is exceeded,
mark the job as errored instead of retrying forever.

Also extract maxPodPendingTimeout helper to deduplicate the timeout
resolution logic between the pod-missing and pod-pending code paths.
@Prucek
Prucek force-pushed the fix-pending-timeout-unreachable-cluster branch from 3ac48b9 to e9b0c93 Compare August 5, 2026 14:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.