Skip to content

Complete planned nodes when provisioning throws an Error - #1364

Open
hshibuya-dd wants to merge 1 commit into
jenkinsci:masterfrom
hshibuya-dd:hshibuya/clean-provisioning-error
Open

Complete planned nodes when provisioning throws an Error#1364
hshibuya-dd wants to merge 1 commit into
jenkinsci:masterfrom
hshibuya-dd:hshibuya/clean-provisioning-error

Conversation

@hshibuya-dd

@hshibuya-dd hshibuya-dd commented Sep 1, 2026

Copy link
Copy Markdown

DockerCloud starts agent provisioning asynchronously and exposes each launch to Jenkins as a PlannedNode backed by a CompletableFuture. The provisioning runnable may catch Exception, completed that and cleaned up its in-progress counter.

But Errors escape that handler. In particular, an OutOfMemoryError from native thread exhaustion could terminate the runnable without completing the planned-node future. Jenkins would then continue to account for a launch that could never finish. Queue items could remain waiting for an executor, Cloud Statistics could retain old PROVISIONING activities, and later provisioning decisions would not recover even after the original resource pressure had disappeared.

Catch Throwable at the existing provisioning boundary so every failed launch completes its planned-node future. Preserve fatal-error semantics by rethrowing Error instances unchanged after recording the failure and attempting the existing agent cleanup. RuntimeException and checked-exception handling remain unchanged, and the finally block still decrements the containers-in-progress count.

Add a regression test that provisions through DockerCloud's public asynchronous path, makes DockerTemplate.provisionNode throw the native-thread OutOfMemoryError seen in practice, and verifies that the PlannedNode future terminates with that exact cause instead of remaining incomplete.

This is a probable fix for the persistent provisioning state reported in #1087.

Testing done

First, run the master jenkins with "mvn hpi:run".
Run this groovy script in console:, which instantiates a DockerCloud with a template which always fails with OutOfMemory error when try to provision a node.
issue1087-repro.groovy.gz

After waiting for a while, you'll get the following results:

`Triggered 'issue1087-repro-1' (label 'issue1087-repro'). NodeProvisioner will try this cloud.
Waiting ~25s for NodeProvisioner to make its first attempt...

=== Cloud Statistics activities created by this run ===
currentPhase=PROVISIONING status=OK

=== Build queue ===
issue1087-repro-1: ‘Jenkins’ doesn’t have label ‘issue1087-repro’
(an activity is open and the job is stuck waiting for an agent)

=== Deterministic check: planned-node future ===
REPRO (unfixed): the planned-node future NEVER completes -> Jenkins accounts for a launch that can never finish. The queue item and cloud-stats activity stay stranded and NodeProvisioner never retries.

Over several MINUTES the two also diverge in cloud-stats: unfixed stays at
exactly one activity frozen in PROVISIONING; fixed accumulates many
COMPLETED/FAIL activities as it retries. Inspect CloudStatistics later to confirm.
`
Now, apply this PR and run the same test:

`Triggered 'issue1087-repro-1' (label 'issue1087-repro'). NodeProvisioner will try this cloud.
Waiting ~25s for NodeProvisioner to make its first attempt...

=== Cloud Statistics activities created by this run ===
currentPhase=PROVISIONING status=OK

=== Build queue ===
issue1087-repro-1: ‘Jenkins’ doesn’t have label ‘issue1087-repro’
(an activity is open and the job is stuck waiting for an agent)

=== Deterministic check: planned-node future ===
FIXED: the planned-node future completed exceptionally (cause=java.lang.OutOfMemoryError: unable to create native thread) -> the launch is released, so cloud-stats resolves it and NodeProvisioner recovers and retries.

Over several MINUTES the two also diverge in cloud-stats: unfixed stays at
exactly one activity frozen in PROVISIONING; fixed accumulates many
COMPLETED/FAIL activities as it retries. Inspect CloudStatistics later to confirm.
`

Submitter checklist

  • [x ] Make sure you are opening from a topic/feature/bugfix branch (right side) and not your main branch!
  • [ x] Ensure that the pull request title represents the desired changelog entry
  • [ x] Please describe what you did
  • [ x] Link to relevant issues in GitHub or Jira
  • Link to relevant pull requests, esp. upstream and downstream changes
  • [ x] Ensure you have provided tests that demonstrate the feature works or the issue is fixed

DockerCloud starts agent provisioning asynchronously and exposes each
launch to Jenkins as a PlannedNode backed by a CompletableFuture. The
provisioning runnable may catch Exception, completed that and cleaned
up its in-progress counter.

But Errors escape that handler. In particular, an OutOfMemoryError from
native thread exhaustion could terminate the runnable without completing
the planned-node future. Jenkins would then continue to account for a
launch that could never finish. Queue items could remain waiting for an
executor, Cloud Statistics could retain old PROVISIONING activities, and
later provisioning decisions would not recover even after the original
resource pressure had disappeared.

Catch Throwable at the existing provisioning boundary so every failed
launch completes its planned-node future. Preserve fatal-error semantics
by rethrowing Error instances unchanged after recording the failure and
attempting the existing agent cleanup. RuntimeException and
checked-exception handling remain unchanged, and the finally block still
decrements the containers-in-progress count.

Add a regression test that provisions through DockerCloud's public
asynchronous path, makes DockerTemplate.provisionNode throw the
native-thread OutOfMemoryError seen in practice, and verifies that the
PlannedNode future terminates with that exact cause instead of remaining
incomplete.

This is a probable fix for the persistent provisioning state reported in
@hshibuya-dd
hshibuya-dd requested a review from a team as a code owner September 1, 2026 19:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant