Skip to content

fix: scope Step Functions, Scheduler, Pipes, Logs, SNS, SSM, Secrets Manager, Kinesis, MemoryDB, X-Ray and Resolver state to the request region - #2681

Merged
vieiralucas merged 15 commits into
mainfrom
fix-region-scoped-sfn-scheduler-pipes
Oct 4, 2026
Merged

vieiralucas merged 15 commits into
mainfrom
fix-region-scoped-sfn-scheduler-pipes

Conversation

@vieiralucas

@vieiralucas vieiralucas commented Oct 4, 2026 •

Copy link
Copy Markdown
Member

Summary

Bug audit 2026-10-01 Tier 0.6 / 0.7 sweep: region-scope Step Functions, EventBridge Scheduler, EventBridge Pipes, CloudWatch Logs, SNS, SSM, Secrets Manager, Kinesis, MemoryDB, X-Ray and Route 53 Resolver. Stacked on the shared region-state infra in PR #2675 (branch fix-region-scoped-state, crates/fakecloud-core/src/multi_account.rs); merge that first. Combined into one PR to keep CI load down.

Every service here now keeps its state per (account, region): the same name coexists in two regions, lists and describes see only the request region, an ARN of another region is that service's NotFound, every ARN-addressed lookup or cross-service delivery uses the ARN's account and region, and pre-regional snapshots are split once on load by each record's ARN region (documented per service in reference/persistence.md, "Regional state migration"). No /_fakecloud/* wire shape changed, so no SDK changes.

Step Functions

  • State machines, executions, activities, versions, aliases, map runs and task tokens per (account, region); name uniqueness and List* region-scoped. The interpreter records history, map runs, task tokens and child executions in the execution's own region; activities in the activity's region; StepFunctionsDelivery starts resolve the state machine by its ARN. Snapshot v3.

EventBridge Scheduler / Pipes

  • Scheduler: every region has its own default group (an untouched region still lists/gets it); ticker, fire-once and post-fire deletes keyed by (account, region). Snapshot v2.
  • Pipes: account -> region -> state; settle/recovery and CFN use the request/stack region; runner checkpoints live with their pipe (pipe ARN's account+region, not the source's). Snapshot v2.

CloudWatch Logs

  • Every Logs resource per (account, region). EventBridge Logs targets, ECS/CodeBuild/Route 53 ingestion and the CFN provisioner write into the log group's own region. The segmented manifest migrates pre-regional metadata (v3). CloudwatchLogsDelivery::put_log_events now takes the region.

SNS

  • Topics, subscriptions, platform applications, published messages and SMS settings per (account, region); ListSubscriptions / ListPlatformApplications no longer leak other regions; deliveries resolve the topic by its ARN; CFN subscriptions live in their topic's region; SmsDispatcher::send_sms takes the region (Cognito sends from its pool's region). Snapshot v3.

SSM

  • Parameters, documents, maintenance windows, patch baselines, associations, OpsItems, inventory, sessions, etc. per region. The rewrite_arn_region hack is removed (ARNs stored with their real region, byte-identical on read). /aws/service/... public parameters in every region. Registered default patch baseline per region per OS. /aws/reference/secretsmanager/... reads the caller's account and region. read_parameter_value takes a region (ECS, CodeBuild callers updated). Snapshot v3.

Secrets Manager

  • Secrets per region; another region's ARN -> ResourceNotFoundException. Replication is now real: replica regions get a read-only replica secret kept in sync with the primary; name clash -> Failed unless ForceOverwriteReplicaSecret; RemoveRegionsFromReplication deletes the replica; StopReplicationToReplica promotes it; deleting a primary with replicas is refused. read_secret_value takes a region. Snapshot v3.

Kinesis

  • Streams, consumers, channels, resource policies per region; a stream ARN of another region no longer resolves to a same-name local stream. Delivery (DynamoDB destinations, Logs subscriptions, EventBridge/Scheduler/Pipes targets), the Lambda ESM poller and the Pipes source use the ARN's account+region. Snapshot v3.

MemoryDB / X-Ray / Route 53 Resolver

  • MemoryDB: the retarget_default_arns hack is gone; each region seeds its own default user/ACL/parameter groups; multi-region clusters stay account-wide. Snapshot v2.
  • X-Ray: groups, sampling rules (each region its own Default rule; ensure_default_rule re-pointing removed), encryption config, policies, traces per region. Snapshot v2.
  • Route 53 Resolver: every resource regional; per-VPC configs keyed by VPC within the region; EC2 existence checks go through one region-aware helper. Snapshot v2.

Known pre-existing gap (not changed here)

  • No CloudwatchLogsDelivery implementation is wired in the server, so API Gateway v2 access logs and Step Functions execution logging deliveries are dropped today; this PR only threads the region through the trait.

Test plan

  • Unit (all pass): stepfunctions 262, scheduler 98, pipes 34, logs 347, sns 236, ssm 266, secretsmanager 150, kinesis 184, memorydb 30, xray 22, route53resolver 33, cloudformation 580, cognito, eventbridge, ecs/codebuild secret tests, fakecloud --bin 138. New tests per service: same name in two regions, region-scoped list/describe/delete, other-region ARN not found, untouched-region reads create nothing, legacy snapshot with two regions migrates correctly.
  • E2E (all pass): new kinesis_regional, ssm_secretsmanager_regional; existing stepfunctions (81), sfn_sync, stepfunctions_paths/persistence/task_token, scheduler (+introspection/persistence), pipes (+execution/sources/targets), logs (54/55), logs_introspection, logs_persistence, eventbridge, sns, sns_persistence, ssm, secretsmanager (+kms/persistence/policies/patch tracking), kinesis, kinesis_persistence, xray, cloudformation_kinesis(_consumer), cloudformation_secretsmanager. Docker is unresponsive on this machine, so Docker-backed tests were not runnable: pipes_enrichment, logs_subscription_filter_invokes_lambda (both run real Lambda code), and the RDS step of region_scoped_arns.
  • Conformance: stepfunctions/states/scheduler/pipes 27/27, logs 51/51, sns 42/42, ssm 69, secretsmanager 14, kinesis/memorydb/xray/route53resolver targeted runs pass.
  • cargo clippy --workspace --all-targets -- -D warnings, cargo fmt

@vieiralucas
vieiralucas force-pushed the fix-region-scoped-sfn-scheduler-pipes branch from 6a7e356 to 728e071 Compare October 4, 2026 09:41
@vieiralucas vieiralucas changed the title fix(stepfunctions,scheduler,pipes): scope state to the request region fix: scope Step Functions, Scheduler, Pipes, Logs, SNS, SSM, Secrets Manager, Kinesis, MemoryDB, X-Ray and Resolver state to the request region Oct 4, 2026
@vieiralucas
vieiralucas force-pushed the fix-region-scoped-sfn-scheduler-pipes branch 5 times, most recently from 21a2b5f to 729de8a Compare October 4, 2026 17:31
…gion

CloudWatch Logs state lives per (account, region); EventBridge Logs targets,
ECS/CodeBuild/Route 53 ingestion and the CloudFormation provisioner write
into the log group's own region, and the segmented manifest migrates
pre-regional metadata by each record's ARN.
…settings to the request region

ListSubscriptions and ListPlatformApplications no longer leak other regions,
deliveries resolve a topic by its ARN's account and region, Cognito SMS goes
to its pool's region, and v2 snapshots split by ARN region.
…he request region

SSM (parameters, documents, maintenance windows, patch baselines and the
per-OS registered default baseline, associations, OpsItems, inventory,
managed instances, sessions, data syncs, service settings) and Secrets
Manager secrets now live per (account, region). Parameter ARNs are stored
with their real region (the rewrite_arn_region masking is gone), an ARN of
another region is not found, and the public /aws/service parameters are
seeded in every region. read_parameter_value / read_secret_value take the
caller region (ARNs resolve in the region they name); ECS, CodeBuild, the
CloudFormation provisioner, the SSM Secrets Manager reference and the
rotation scheduler address the right region. Secret replication creates a
real read-only replica in each replica region, kept in sync with the
primary; StopReplicationToReplica promotes it. v2 snapshots migrate each
resource into the region its ARN names.
…, resolver, kinesis

Two aws-sdk-rust clients in different regions: same stream name coexists,
ListStreams is region-scoped, records stay in their region, and a stream
ARN addresses only its own region. Persistence reference lists the new
regional migrations.
The conformance test asserted the removed cross-region ARN tolerance;
it now proves each region resolves only its own same-named stream.
…ets Manager, Scheduler and ECS logs

- Step Functions aws-sdk / optimized integrations (and the ResultWriter
  and TaskScheduled event) use the execution's region, not us-east-1.
- SNS ConfirmSubscription / Unsubscribe resolve the subscription from its
  topic or subscription ARN, wherever the link is opened.
- ECS awslogs writes to awslogs-region (RunTask region by default).
- Pipes v1 checkpoints migrate under the pipe ARN's account.
- Secrets Manager creates replica secrets for a migrated pre-regional
  snapshot (sync_all_replicas on load).
- /_fakecloud/scheduler/schedules entries carry their region and
  /_fakecloud/scheduler/fire takes accountId/region, in the server and all
  seven SDKs.
Main's background export tasks (#2667) and frozen query results now
address the (account, region) the task or query was created in.
…ns and CFN scheduler tags

Cross-account subscriptions (#2676) live with their topic in the topic
owner's account and the topic's region; ListSubscriptions lists the
requester's subscriptions in the request region; Unsubscribe resolves the
subscription's region from its ARN and keeps the legacy caller-account
fallback. The CFN schedule-group tag update addresses the stack region.
Also deflake the SSM SendCommand persistence test (the save follows the
in-memory flip asynchronously).
@vieiralucas
vieiralucas force-pushed the fix-region-scoped-sfn-scheduler-pipes branch from 729de8a to 9a070ed Compare October 4, 2026 20:25
@vieiralucas
vieiralucas merged commit 2b22ac0 into main Oct 4, 2026
159 checks passed
@vieiralucas
vieiralucas deleted the fix-region-scoped-sfn-scheduler-pipes branch October 4, 2026 21:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant