Skip to content

feat(toolkit-lib): diagnose hook failures in the central stack diagnoser - #1786

Merged
iankhou merged 4 commits into
mainfrom
iankhou-hook-error-diagnosis
Jul 30, 2026
Merged

feat(toolkit-lib): diagnose hook failures in the central stack diagnoser#1786
iankhou merged 4 commits into
mainfrom
iankhou-hook-error-diagnosis

Conversation

@iankhou

@iankhou iankhou commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Closes #1788

Description

Details for CloudFormation hook failures are currently provided when hooks fail during cdk deploy, but not in cdk diagnose.

For cdk diagnose, only the short reason would be available (e.g. "The following hook(s) failed: [X]"), since it doesn't have a live activity monitor.

Current state

Hook details available:

  • cdk deploy live activity monitor
  • cdk deploy final output (since we already got it from the love activity monitor)

✨  Synthesis time: 2.23s

HookDemoStack: creating CloudFormation changeset...
Changeset arn:aws:cloudformation:us-east-1:053108159643:changeSet/cdk-deploy-change-set/3e16a1f3-eaf2-4e2c-b43c-c55ca099bdfb created and waiting in review for manual execution (--no-execute)
HookDemoStack: deploying... [1/1]
HookDemoStack | 0/3 | 12:29:26 PM | REVIEW_IN_PROGRESS      | AWS::CloudFormation::Stack | HookDemoStack User Initiated
HookDemoStack | 0/3 | 12:29:33 PM | CREATE_IN_PROGRESS      | AWS::CloudFormation::Stack | HookDemoStack User Initiated
HookDemoStack | 0/3 | 12:29:36 PM | CREATE_IN_PROGRESS      | AWS::CDK::Metadata         | CDKMetadata/Default (CDKMetadata) 
HookDemoStack | 0/3 | 12:29:36 PM | CREATE_IN_PROGRESS      | AWS::S3::Bucket            | NonCompliantBucket (NonCompliantBucketD2660FEF) 
HookDemoStack | 0/3 | 12:29:36 PM | CREATE_IN_PROGRESS      | AWS::CDK::Metadata         | CDKMetadata/Default (CDKMetadata) 
HookDemoStack | 0/3 | 12:29:36 PM | CREATE_IN_PROGRESS      | AWS::S3::Bucket            | NonCompliantBucket (NonCompliantBucketD2660FEF) 
HookDemoStack | 0/3 | 12:29:37 PM | CREATE_IN_PROGRESS      | AWS::CDK::Metadata         | CDKMetadata/Default (CDKMetadata) 
HookDemoStack | 0/3 | 12:29:37 PM | CREATE_IN_PROGRESS      | AWS::S3::Bucket            | NonCompliantBucket (NonCompliantBucketD2660FEF) 
HookDemoStack | 0/3 | 12:29:37 PM | CREATE_IN_PROGRESS      | AWS::CDK::Metadata         | CDKMetadata/Default (CDKMetadata) Hook invocations complete.  Resource creation initiated
HookDemoStack | 0/3 | 12:29:37 PM | CREATE_FAILED           | AWS::S3::Bucket            | NonCompliantBucket (NonCompliantBucketD2660FEF) The following hook(s) failed: [Demo::Guard::VersioningHook] : NonCompliant Rules:

[s3_bucket_versioning_enabled]
• Check was not compliant as property [VersioningConfiguration] is missing. Value traversed to [Path=/Resources/NonCompliantBucketD2660FEF/Properties[L:0,C:82] Value={}].
Remediation: S3 buckets must have versioning enabled. Set VersioningConfiguration.Status to "Enabled".
HookDemoStack | 0/3 | 12:29:37 PM | CREATE_FAILED           | AWS::CDK::Metadata         | CDKMetadata/Default (CDKMetadata) Resource creation cancelled
HookDemoStack | 0/3 | 12:29:37 PM | ROLLBACK_IN_PROGRESS    | AWS::CloudFormation::Stack | HookDemoStack The following resource(s) failed to create: [NonCompliantBucketD2660FEF, CDKMetadata]. Rollback requested by user.
HookDemoStack | 1/3 | 12:29:39 PM | DELETE_COMPLETE         | AWS::CDK::Metadata         | CDKMetadata/Default (CDKMetadata) 
HookDemoStack | 1/3 | 12:32:07 PM | DELETE_SKIPPED          | AWS::S3::Bucket            | NonCompliantBucket (NonCompliantBucketD2660FEF) 
HookDemoStack | 2/3 | 12:32:07 PM | ROLLBACK_COMPLETE       | AWS::CloudFormation::Stack | HookDemoStack 

Failed resources:
HookDemoStack | 12:29:37 PM | CREATE_FAILED           | AWS::S3::Bucket            | NonCompliantBucket (NonCompliantBucketD2660FEF) The following hook(s) failed: [Demo::Guard::VersioningHook] : NonCompliant Rules:

[s3_bucket_versioning_enabled]
• Check was not compliant as property [VersioningConfiguration] is missing. Value traversed to [Path=/Resources/NonCompliantBucketD2660FEF/Properties[L:0,C:82] Value={}].
Remediation: S3 buckets must have versioning enabled. Set VersioningConfiguration.Status to "Enabled".
❌  HookDemoStack failed: DeploymentError: Resource updates failed:
HookDemoStack/NonCompliantBucket/Resource  (AWS::S3::Bucket NonCompliantBucketD2660FEF)
  The following hook(s) failed: [Demo::Guard::VersioningHook]
Source Location: ...new Bucket2 in aws-cdk-lib...
                 <anonymous> (/private/tmp/hook-demo/app/app.js:10:1)
                 ...node internals...

Hook details not available:

  • cdk diagnose final output

Current cdk diagnose example

✨  Synthesis time: 2.6s
❌ Stack HookDemoStack:
Resource updates failed:
HookDemoStack/NonCompliantBucket/Resource  (AWS::S3::Bucket NonCompliantBucketD2660FEF)
  The following hook(s) failed: [Demo::Guard::VersioningHook]
Source Location: ...new Bucket2 in aws-cdk-lib...
                 <anonymous> (/private/tmp/hook-demo/app/app.js:10:1)
                 ...node internals...

State after this change

No changes to cdk deploy output.

cdk diagnose after this change:

✨  Synthesis time: 3.38s
❌ Stack HookDemoStack:
Resource updates failed:
HookDemoStack/NonCompliantBucket/Resource  (AWS::S3::Bucket NonCompliantBucketD2660FEF)
  The following hook(s) failed: [Demo::Guard::VersioningHook]
Source Location: ...new Bucket2 in aws-cdk-lib...
                 <anonymous> (/private/tmp/hook-demo/app/app.js:10:1)
                 ...node internals...

   Hook 'Demo::Guard::VersioningHook' failed: NonCompliant Rules:
   
   [s3_bucket_versioning_enabled]
   • Check was not compliant as property [VersioningConfiguration] is missing. Value traversed to [Path=/Resources/NonCompliantBucketD2660FEF/Properties[L:0,C:82] Value={}].
   Remediation: S3 buckets must have versioning enabled. Set VersioningConfiguration.Status to "Enabled".

In this change, we move hook error detail fetching to a shared helper (fetchHookResultDetails), so that it's available in diagnose without code duplication.

We move hook-error diagnosis into the central diagnoser, which both cdk deploy and cdk diagnose converge into:

  • The GetHookResult fetch is extracted into a shared fetchHookResultDetails helper.
  • The diagnoser fetches hook failure details for stack-event errors and attaches them as additional diagnostic context, and the change-set hook path now uses the same shared helper.
  • The monitor keeps its live enrichment of the activity stream, but now uses the shared helper (fetchHookResultDetails).

Checklist

  • This change contains a major version upgrade for a dependency and I confirm all breaking changes are addressed
    • Release notes for the new version:

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

@iankhou
iankhou requested a review from Copilot July 30, 2026 14:06
@aws-cdk-automation
aws-cdk-automation requested a review from a team July 30, 2026 14:07
@github-actions

Copy link
Copy Markdown
Contributor

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR extends CloudFormation hook-failure diagnostics across the toolkit by centralizing GetHookResult fetching/formatting in a shared helper and wiring that into both deploy monitoring and the central stack diagnoser, so hook failure details are available beyond the live activity stream.

Changes:

  • Introduces a shared fetchHookResultDetails() helper to retrieve and format GetHookResult details (with warning behavior on failures).
  • Correlates failed hook-invocation events with subsequent resource failures via ResourceErrors, and surfaces those details as additional diagnostic context in CloudFormationStackDiagnoser.
  • Updates the stack activity monitor to reuse the shared hook-result helper for live event enrichment.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
packages/@aws-cdk/toolkit-lib/lib/api/stack-events/hook-result-details.ts Adds shared hook result fetching helper and related warning logic.
packages/@aws-cdk/toolkit-lib/lib/api/stack-events/stack-activity-monitor.ts Switches live hook detail enrichment to the shared fetch helper.
packages/@aws-cdk/toolkit-lib/lib/api/stack-events/resource-errors.ts Tracks failed hook events and attaches correlated hook failures to resource errors.
packages/@aws-cdk/toolkit-lib/lib/api/diagnosing/stack-diagnoser.ts Enriches resource-error diagnosis with hook failure details (via GetHookResult or fallback).
packages/@aws-cdk/toolkit-lib/test/api/stack-events/resource-errors.test.ts Adds unit tests for hook failure correlation behavior.
packages/@aws-cdk/toolkit-lib/test/api/diagnosing/stack-diagnoser.test.ts Adds unit tests verifying diagnoser hook-detail enrichment and fallback behavior.
Comments suppressed due to low confidence (1)

packages/@aws-cdk/toolkit-lib/lib/api/stack-events/resource-errors.ts:169

  • hookFailuresFor() also computes the lookup key even when StackId/LogicalResourceId are missing, which again falls back to an empty-stack key. Returning early avoids accidental matches and makes the correlation rules explicit.
  private hookFailuresFor(event: StackEvent): ResourceHookFailure[] {
    const recorded = this.seenHookFailures.get(hookFailureKey(event)) ?? [];
    const reason = event.ResourceStatusReason ?? '';
    return recorded.filter((h) => reason.includes(h.hookType));
  }

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated no new comments.

…Formation Hooks

When a CloudFormation Hook (e.g. a Guard Hook or Lambda Hook) fails a
resource during a deployment, the stack events only carry a terse
'The following hook(s) failed: [X]'. The actual failure details (Guard
rule annotations, remediation messages, Lambda hook reasons) live in
the GetHookResult API. cdk deploy already fetches them for its live
activity stream, but cdk diagnose showed nothing beyond the terse
message.

Move hook-error diagnosis into the central CloudFormationStackDiagnoser:

- ResourceErrors now correlates failed hook events to the resource
  errors they cause (resolving the 'FIXME: Check hooks') and records
  them on a new hookFailures field.
- The GetHookResult fetch (including the bootstrap-version permission
  hint) is extracted into a shared fetchHookResultDetails helper, used
  by the deploy monitor's live enrichment, the diagnoser's change-set
  hook path, and the new stack-event hook path.
- The diagnoser fetches hook failure details for stack-event errors
  and attaches them as additional diagnostic context. This is gated
  behind a new fetchHookFailureDetails option, enabled only by the
  diagnose action: during a deployment the activity stream already
  shows the details live, so deploy output is unchanged.

Also fixes a malformed permissions warning (stray newline and quote)
that was carried over from the monitor code.
const [sourceTrace, additionalContext, hookContext] = await Promise.all([
sourceTracePromise,
this.investigateResourceBestEffort(err),
this.hookFailureContext(err),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This adds a new API call, deserves an integ test.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right, we test the deploy flow but now introduce this API call for diagnose. Will add an integ test.

iankhou added 3 commits July 30, 2026 21:37
Covers the new GetHookResult fetch in the central stack diagnoser: deploys
the guard-hook-app fixture's non-compliant stack with --no-rollback, then
asserts that cdk diagnose output contains the hook failure details that
are only available via the GetHookResult API.
The Diagnosis wrapper doesn't expose 'problems' directly; use the value
returned by assertProblem(), which narrows to the 'problem' variant.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Move diagnosis of Hook errors from cdk deploy to the central diagnosis code

4 participants