(toolkit-lib): throttled final stack-event poll in monitor.stop() fails an already-successful deployment

[polothy]

Describe the bug

FullCloudFormationDeployment.monitorDeployment() ends with an unguarded finally
(packages/@aws-cdk/toolkit-lib/lib/api/deployments/deploy-stack.ts):

let finalState = this.cloudFormationStack;
try {
  const successStack = await waitForStackDeploy(this.cfn, this.ioHelper, this.stackName);
  if (!successStack) { throw new DeploymentError(...); }
  finalState = successStack;   // <-- the deployment has SUCCEEDED here
} catch (e: any) {
  ...
  throw e;
} finally {
  await monitor.stop();        // <-- can throw, and a throw here discards the success
}

monitor.stop() issues one further DescribeStackEvents read:

stop() → finalPollToEnd() → readNewEvents() → StackEventPoller.poll() → doPoll()

On this path readNewEvents() has no try/catch, unlike the periodic tick(), which
catches, emits CDK_TOOLKIT_E5500, and carries on:

} catch (e) {
  await this.ioHelper.notify(IO.CDK_TOOLKIT_E5500.msg(
    util.format('Error occurred while monitoring stack: %s', e), { error: e },
  ));
}

Because a throw inside a finally discards whatever the try was about to return, a
transient Throttling on that final read throws away an already-successful deployment
result and propagates as a stack failure. The stack is UPDATE_COMPLETE in CloudFormation
while the toolkit reports it failed and cdk deploy exits non-zero.

The decisive point is that this final poll is purely presentational — its own comment says so:

// Do a final poll for all events. This is to handle the situation where DescribeStackStatus
// already returned an error, but the monitor hasn't seen all the events yet and we'd end
// up not printing the failure reason to users.
await this.finalPollToEnd(oldMonitorId);

It exists to make console output complete. It cannot affect whether the deployment succeeded,
so it should not be able to change the reported outcome.

The same unguarded shape appears at the destroyStack monitor in the same file, and
Deployments.rollbackStack has a third monitor.stop() call site.

Expected Behavior

A read-only, post-completion, presentation-only poll cannot change a deployment's outcome. If
the final poll fails, the toolkit reports the successful deployment and at most warns that
trailing stack events could not be fetched — the same treatment tick() already gives an
identical failure.

Current Behavior

A stack that CloudFormation reports as UPDATE_COMPLETE is reported by the toolkit as failed,
and the process exits non-zero, aborting the remainder of a multi-stack deploy:

❌  <stack-name> failed: Throttling: Rate exceeded

CloudFormation's own stack-event history showed the stack reaching UPDATE_COMPLETE roughly two
and a half minutes before the toolkit reported it as failed, and the only error in the run was
a CloudFormation read throttle.

The DescribeStackEvents poller was demonstrably being throttled. Below is the non-fatal
CDK_TOOLKIT_E5500 occurrence from the same run, from a global install of aws-cdk@2.1133.0
(stack name omitted; $metadata and Error were collapsed by Node's inspector):

Error occurred while monitoring stack: Throttling: Rate exceeded
    at ProtocolLib.getErrorSchemaOrThrowBaseException (/usr/local/lib/node_modules/aws-cdk/lib/index.js:37185:63)
    at AwsQueryProtocol.handleError (/usr/local/lib/node_modules/aws-cdk/lib/index.js:39041:65)
    at AwsQueryProtocol.deserializeResponse (/usr/local/lib/node_modules/aws-cdk/lib/index.js:39011:22)
    at async /usr/local/lib/node_modules/aws-cdk/lib/index.js:20535:24
    at async /usr/local/lib/node_modules/aws-cdk/lib/index.js:31100:22
    at async /usr/local/lib/node_modules/aws-cdk/lib/index.js:29786:40
    at async /usr/local/lib/node_modules/aws-cdk/lib/index.js:30715:26
    at async _StackEventPoller.doPoll (/usr/local/lib/node_modules/aws-cdk/lib/index.js:90499:26)
    at async _StackEventPoller.poll (/usr/local/lib/node_modules/aws-cdk/lib/index.js:90455:24) {
  '$fault': 'client',
  '$retryable': undefined,
  '$metadata': [Object],
  Type: 'Sender',
  Code: 'Throttling',
  Error: [Object]
}

Those offsets resolve against the published aws-cdk@2.1133.0 bundle, so they are reproducible
from the artifact — npm pack aws-cdk@2.1133.0 and check:

• lib/index.js:90455 → const events = await this.doPoll();, inside poll() (defined at 90454)
• lib/index.js:90499 → await this.cfn.describeStackEvents({ StackName: ..., NextToken: ... }),
  inside doPoll() (defined at 90492)

Note that 90499 is the paginating call inside doPoll's loop, so a single logical poll can
issue several DescribeStackEvents requests, each with its own retry budget.

That occurrence was swallowed by tick() as designed, and the deploy continued. The unguarded
finalPollToEnd path reaches the identical call from stop(), where the same error is fatal
instead.

Reproduction Steps

The defect is in the control flow and does not need real throttling to demonstrate — inject the
failure on the event poll while the stack reaches a successful terminal state. Sketch in the
style of the existing toolkit-lib tests (test/_helpers/mock-sdk.ts):

import {
  DescribeStackEventsCommand, DescribeStacksCommand, StackStatus,
} from '@aws-sdk/client-cloudformation';
import { mockCloudFormationClient, restoreSdkMocksToDefault } from '../_helpers/mock-sdk';

test('a throttled final event poll must not fail an already-successful deployment', async () => {
  restoreSdkMocksToDefault();

  // The stack reaches a successful terminal state, so waitForStackDeploy() resolves.
  mockCloudFormationClient.on(DescribeStacksCommand).resolves({
    Stacks: [{
      StackName: 'my-stack',
      StackStatus: StackStatus.UPDATE_COMPLETE,
      CreationTime: new Date(),
    }],
  });

  // Every DescribeStackEvents call throttles. tick() swallows these by design;
  // the final poll inside monitor.stop() does not.
  mockCloudFormationClient.on(DescribeStackEventsCommand).rejects(
    Object.assign(new Error('Rate exceeded'), {
      name: 'Throttling',
      $metadata: { httpStatusCode: 400 },
    }),
  );

  // Fails: rejects with `Throttling: Rate exceeded` out of the `finally`,
  // even though the deployment succeeded.
  await expect(deployments.deployStack({ stack: testStack({ stackName: 'my-stack' }) }))
    .resolves.toMatchObject({ type: 'did-deploy-stack' });
});

Note the CloudFormation client's own retry strategy will spend its budget on the rejecting mock,
so this wants fake timers (test/_helpers/fake-time.ts) to stay fast.

Organically, the trigger is any deploy that holds enough stacks in flight to saturate the
account-level CloudFormation read quota:

1. cdk deploy --all --concurrency N with N above ~20, all stacks in one account and region.
   The monitor polls DescribeStackEvents every 2 s per in-flight stack, so N / 2 requests
   per second exceeds the account quota for that API.
2. Let a stack complete successfully while the account is throttling.
3. If that stack's final poll exhausts its retry budget, the successful stack is reported as
   failed and the whole deploy exits non-zero.

Possible Solution

Guard inside finalPollToEnd, which covers the deploy, destroy, and rollback monitors from one
place and mirrors what tick() already does:

private async finalPollToEnd(monitorId: string) {
  try {
    if (this.readPromise) {
      await this.readPromise;
    }
    await this.readNewEvents(monitorId);
  } catch (e) {
    // A presentation-only final poll must never change the outcome of the operation
    await this.ioHelper.notify(IO.CDK_TOOLKIT_E5500.msg(
      util.format('Error occurred during final stack event poll: %s', e), { error: e },
    ));
  }
}

Guarding at the finally { await monitor.stop(); } call sites instead would cover deploy and
destroy but miss Deployments.rollbackStack.

Happy to open a PR if you'd prefer this placement, or a different one.

Additional Information/Context

Why this is easy to hit. The DescribeStackEvents quota is 10 requests/second and is not
adjustable — aws service-quotas list-service-quotas --service-code cloudformation reports
Adjustable: false for it (and for DescribeStacks and GetTemplate). With a 2 s interval per
in-flight stack, more than ~20 concurrent stack operations in one account and region
structurally exceeds that quota for this API alone, leaving SDK retries as the only thing
keeping the deploy alive.

Operators cannot widen the retry window. SDK.cloudFormation() hardcodes
retryStrategy: new ConfiguredRetryStrategy(7, cappedExponentialBackoff(1000, 15_000)), which
overrides AWS_MAX_ATTEMPTS and AWS_RETRY_MODE. That also means there is no way to opt into
adaptive mode, whose client-side rate limiter would suit this workload well.

The tolerance for sustained throttling narrowed in 2.1122.0. #1500 changed the
CloudFormation client's retry strategy:

-      retryStrategy: new ConfiguredRetryStrategy(11, (attempt: number) => 1000 * (2 ** attempt)),
+      retryStrategy: new ConfiguredRetryStrategy(7, cappedExponentialBackoff(1000, 15_000)),

That reduces attempts 11 → 7 and lowers the per-delay ceiling from @smithy/util-retry's
MAXIMUM_RETRY_DELAY (20 s) to an explicit 15 s, shortening how long a single call chain can
ride out a throttling burst from roughly 131 s to roughly 45 s. I verified the boundary against
the published artifacts: aws-cdk@2.1121.0 carries the old values, 2.1122.0 the new ones.

That change looks entirely reasonable for the problem it solved — the PR notes an uncapped
backoff on this client could make the SDK spend tens of minutes, which is a real bug. Flagging
it only because it means less headroom for the failure described here, and because the
consequence of running out of headroom is currently an incorrectly failed deployment rather
than a warning. The polling interval and the number of API calls per poll are unchanged, so
this is purely a tolerance change, not a request-rate change.

Relationship to #1709 / #1710. #1710 made the polling interval configurable, which lowers
the request rate. This issue is about the consequence when a poll does fail, which #1710
does not change — and the failure described here was observed on a CLI that already contained
#1710.

stackEventPollingInterval is also unreachable for cdk deploy users. In the published
aws-cdk@2.1133.0 bundle it appears only as stackEventPollingInterval: options.… (four
threading sites), never sourced from argv, and the bundle contains no corresponding CLI
option string. So the CLI still polls at the 2 s default with no way to change it, and the
toolkit-lib escape hatch does not help anyone invoking cdk directly.

Related. #879 is the analogous unguarded-monitor report for cdk watch log monitoring.

CDK CLI Version

2.1133.0

(version confirmed by resolving the stack-trace offsets above against the published bundle;
defect confirmed by inspection of that bundle's finalPollToEnd and of main; also present in
2.1129.0)

Node.js Version

24.x where the failure was observed; v24.18.0 for the source analysis

OS

Observed on Linux arm64 (container). Source analysis and version verification on macOS 26.5.2 (Darwin arm64).

Language

TypeScript

---

Authored by Claude Code (Anthropic's agentic CLI). This is AI-assisted analysis of the public
aws-cdk-cli source, prompted by a failure seen in a CI pipeline running cdk deploy --all at
high --concurrency. The source reading, reasoning, and text above are the model's work; the
reproduction sketch has not been executed. Please review the reasoning on its merits rather
than assuming human verification.