Complete planned nodes when provisioning throws an Error - #1364
Open
hshibuya-dd wants to merge 1 commit into
Open
Complete planned nodes when provisioning throws an Error#1364hshibuya-dd wants to merge 1 commit into
hshibuya-dd wants to merge 1 commit into
Conversation
DockerCloud starts agent provisioning asynchronously and exposes each launch to Jenkins as a PlannedNode backed by a CompletableFuture. The provisioning runnable may catch Exception, completed that and cleaned up its in-progress counter. But Errors escape that handler. In particular, an OutOfMemoryError from native thread exhaustion could terminate the runnable without completing the planned-node future. Jenkins would then continue to account for a launch that could never finish. Queue items could remain waiting for an executor, Cloud Statistics could retain old PROVISIONING activities, and later provisioning decisions would not recover even after the original resource pressure had disappeared. Catch Throwable at the existing provisioning boundary so every failed launch completes its planned-node future. Preserve fatal-error semantics by rethrowing Error instances unchanged after recording the failure and attempting the existing agent cleanup. RuntimeException and checked-exception handling remain unchanged, and the finally block still decrements the containers-in-progress count. Add a regression test that provisions through DockerCloud's public asynchronous path, makes DockerTemplate.provisionNode throw the native-thread OutOfMemoryError seen in practice, and verifies that the PlannedNode future terminates with that exact cause instead of remaining incomplete. This is a probable fix for the persistent provisioning state reported in
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DockerCloud starts agent provisioning asynchronously and exposes each launch to Jenkins as a PlannedNode backed by a CompletableFuture. The provisioning runnable may catch Exception, completed that and cleaned up its in-progress counter.
But Errors escape that handler. In particular, an OutOfMemoryError from native thread exhaustion could terminate the runnable without completing the planned-node future. Jenkins would then continue to account for a launch that could never finish. Queue items could remain waiting for an executor, Cloud Statistics could retain old PROVISIONING activities, and later provisioning decisions would not recover even after the original resource pressure had disappeared.
Catch Throwable at the existing provisioning boundary so every failed launch completes its planned-node future. Preserve fatal-error semantics by rethrowing Error instances unchanged after recording the failure and attempting the existing agent cleanup. RuntimeException and checked-exception handling remain unchanged, and the finally block still decrements the containers-in-progress count.
Add a regression test that provisions through DockerCloud's public asynchronous path, makes DockerTemplate.provisionNode throw the native-thread OutOfMemoryError seen in practice, and verifies that the PlannedNode future terminates with that exact cause instead of remaining incomplete.
This is a probable fix for the persistent provisioning state reported in #1087.
Testing done
First, run the master jenkins with "mvn hpi:run".
Run this groovy script in console:, which instantiates a DockerCloud with a template which always fails with OutOfMemory error when try to provision a node.
issue1087-repro.groovy.gz
After waiting for a while, you'll get the following results:
`Triggered 'issue1087-repro-1' (label 'issue1087-repro'). NodeProvisioner will try this cloud.
Waiting ~25s for NodeProvisioner to make its first attempt...
=== Cloud Statistics activities created by this run ===
currentPhase=PROVISIONING status=OK
=== Build queue ===
issue1087-repro-1: ‘Jenkins’ doesn’t have label ‘issue1087-repro’
(an activity is open and the job is stuck waiting for an agent)
=== Deterministic check: planned-node future ===
REPRO (unfixed): the planned-node future NEVER completes -> Jenkins accounts for a launch that can never finish. The queue item and cloud-stats activity stay stranded and NodeProvisioner never retries.
Over several MINUTES the two also diverge in cloud-stats: unfixed stays at
exactly one activity frozen in PROVISIONING; fixed accumulates many
COMPLETED/FAIL activities as it retries. Inspect CloudStatistics later to confirm.
`
Now, apply this PR and run the same test:
`Triggered 'issue1087-repro-1' (label 'issue1087-repro'). NodeProvisioner will try this cloud.
Waiting ~25s for NodeProvisioner to make its first attempt...
=== Cloud Statistics activities created by this run ===
currentPhase=PROVISIONING status=OK
=== Build queue ===
issue1087-repro-1: ‘Jenkins’ doesn’t have label ‘issue1087-repro’
(an activity is open and the job is stuck waiting for an agent)
=== Deterministic check: planned-node future ===
FIXED: the planned-node future completed exceptionally (cause=java.lang.OutOfMemoryError: unable to create native thread) -> the launch is released, so cloud-stats resolves it and NodeProvisioner recovers and retries.
Over several MINUTES the two also diverge in cloud-stats: unfixed stays at
exactly one activity frozen in PROVISIONING; fixed accumulates many
COMPLETED/FAIL activities as it retries. Inspect CloudStatistics later to confirm.
`
Submitter checklist