Skip to content

fix(ci): align parity service aliases#68

Draft
JesseKoldewijn wants to merge 21 commits into
developfrom
feat/batch-further-service-coverage
Draft

fix(ci): align parity service aliases#68
JesseKoldewijn wants to merge 21 commits into
developfrom
feat/batch-further-service-coverage

Conversation

@JesseKoldewijn

@JesseKoldewijn JesseKoldewijn commented Apr 23, 2026

Copy link
Copy Markdown
Owner

Summary

Continues the service coverage expansion started in #62. This batch adds 5 new services and significantly deepens 8 existing ones, adding ~120 operations total and raising the supported service count from 24 → 29.

New services

Service Protocol Key operations
ECS json Clusters, task definitions, services, tasks (17 ops)
RDS query (XML) DB instances, snapshots, subnet/parameter groups (15 ops)
Cognito json User pools, app clients, admin user CRUD, AdminInitiateAuth (17 ops)
ElastiCache query (XML) Cache clusters (Redis/Memcached), replication groups, subnet groups (13 ops)
CloudTrail json Trail CRUD, StartLogging/StopLogging, LookupEvents, tags (11 ops)

Deepened existing services

Service Before After Notable additions
EC2 9 ops 30 ops Key pairs, elastic IPs, internet gateways, EBS volumes, DeleteSubnet/DeleteSecurityGroup, StopInstances/StartInstances, CreateTags/DescribeTags, DescribeAvailabilityZones, DescribeRegions
SSM 7 ops 19 ops Documents (CRUD), SendCommand, ListCommands, GetCommandInvocation, GetParameterHistory, tag management
SES 7 ops 16 ops Email templates, GetSendQuota/GetSendStatistics, SendTemplatedEmail, notification attributes
Redshift 5 ops 13 ops Snapshots, subnet groups, parameter groups, EnableLogging/DisableLogging
Route53 7 ops 14 ops Health checks, ListHostedZonesByName, GetHostedZoneCount, ChangeTagsForResource/ListTagsForResource
OpenSearch 6 ops 14 ops DescribeDomains (batch), AddTags/RemoveTags/ListTags, GetCompatibleVersions, ListVersions, software update ops
STS 5 ops 7 ops AssumeRoleWithWebIdentity, AssumeRoleWithSAML

Bug fix

  • EC2 TerminateInstances/StopInstances/StartInstances: collect_indexed_params now checks both query_params and request_body, fixing a regression in the smoke_ec2_instance_lifecycle_with_latency_guardrail integration test where form-body parameters weren't found.

Test plan

  • cargo test --workspace — all tests pass (zero failures)
  • cargo clippy --all-targets --all-features -- -D warnings — zero warnings in changed crates
  • cargo build --workspace — clean build
  • Integration smoke test smoke_ec2_instance_lifecycle_with_latency_guardrail passes
  • README service table updated: ECS, RDS, Cognito, ElastiCache, CloudTrail all show ✅ supported

Summary by CodeRabbit

  • New Features

    • Added CloudTrail, ECS, RDS, Cognito (cognito-idp), and ElastiCache service providers; registration gated by config.
    • Added RDS/ElastiCache/ECS/CloudTrail/Cognito full CRUD and operational flows, plus EC2, OpenSearch, Route53, SES, SSM, Redshift, STS, S3, and other service enhancements.
  • Tests

    • Added extensive integration and performance tests across many services (100-200 item throughput benchmarks and functional suites).

…gap-fills across 8 existing services

New services: ECS, RDS, Cognito, ElastiCache, CloudTrail
Deepened services: SES (+9 ops), Redshift (+8 ops), Route53 (+7 ops),
OpenSearch (+8 ops), EC2 (+21 ops), SSM (+12 ops), STS (+2 ops)

Adds ~120 operations total, bringing the supported service count from
24 to 29 (ECS, RDS, Cognito, ElastiCache, CloudTrail added to README).

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
@github-actions

github-actions Bot commented Apr 23, 2026

Copy link
Copy Markdown
Contributor

CI Results

Checks

Check Result
Rustfmt success
Cargo Check success
Clippy success
Tests success
Build Artifact success
Coverage Validation success

Parity

Profile Scenarios Passed Failed Accepted
core 5 5 0 0
all-services-smoke 34 34 0 105

JesseKoldewijn and others added 5 commits April 23, 2026 11:25
Points AI agents to AGENTS.md as the canonical context file.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
- cargo fmt --all: fix line-length formatting across cloudtrail, cognito,
  ecs, elasticache, and multiple test files
- Add cloudtrail, cognito, ecs, elasticache, rds to service-matrix.json
- Add probe entries for all 5 new services in all-services-smoke.json
- Add guided manifest JSON for cloudtrail, cognito, ecs, elasticache, rds

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
…py lint

- parity.rs: add ecs, rds, cognito, elasticache, cloudtrail to
  all_service_names() and the README→service-id match block so the
  readme_supported_services_match_all_services_smoke_inventory test passes
- studio-ui/explorer.rs: replace manual integer division with
  checked_div to fix the pre-existing `manual checked division` clippy lint

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Add the remaining API Gateway, CloudFormation, and Step Functions lifecycle perf benchmarks from the coverage plan.

Sync the Dockerfile manifest cache layer with the expanded workspace and fix a DynamoDB clippy lint so CI image builds and clippy pass again.
@github-actions

github-actions Bot commented May 20, 2026

Copy link
Copy Markdown
Contributor

Benchmark Gate: ✅ PASS

Profile: default | Mode: docker | Requests: 100 | Concurrency: 6
Timestamp: 2026-06-11T21:51:37Z | Targets: openstack, LocalStack, moto
Thresholds: openstack p95 ≤ 10ms | openstack memory ≤ 125MB | errors = 0

Memory Footprint

Target Idle RSS (MB) Loaded RSS (MB)
openstack 20.0 100.9
LocalStack 381.8 879.0
moto 43.2 119.3

openstack uses 9x less memory than LocalStack under load

ACM

Operation Platform p50 (ms) p95 (ms) p99 (ms) RPS Errors
describe_certificate openstack 2.79 3.99 4.70 2022.3 0
LocalStack 9.12 (-3.3x) 14.32 (-3.6x) 18.80 (-4.0x) 621.3 (-3.3x) 0
moto 16.87 (-6.0x) 19.95 (-5.0x) 22.01 (-4.7x) 343.5 (-5.9x) 0
export_certificate openstack 2.00 2.88 2.97 2843.9 0
LocalStack 8.73 (-4.4x) 11.91 (-4.1x) 16.20 (-5.5x) 674.7 (-4.2x) 100
moto 16.68 (-8.3x) 18.91 (-6.6x) 19.76 (-6.7x) 355.4 (-8.0x) 0
list_certificates openstack 3.23 5.09 6.10 1770.9 0
LocalStack 36.24 (-11.2x) 58.07 (-11.4x) 72.96 (-12.0x) 163.0 (-10.9x) 0
moto 36.75 (-11.4x) 38.53 (-7.6x) 42.29 (-6.9x) 161.7 (-11.0x) 0
request_certificate openstack 1.14 2.91 3.91 261.4 0
LocalStack 571.10 (-501.0x) 693.07 (-238.2x) 695.60 (-177.9x) 9.7 (-26.9x) 0
moto 348.75 (-305.9x) 674.99 (-232.0x) 713.40 (-182.5x) 9.2 (-28.4x) 0

openstack vs LocalStack: 3.3×–501.0× faster on 4 ops (avg 130.0×)
openstack vs moto: 6.0×–305.9× faster on 4 ops (avg 82.9×)

APIGATEWAY

Operation Platform p50 (ms) p95 (ms) p99 (ms) RPS Errors
create_resource openstack 1.53 2.47 3.14 3638.7 0
LocalStack
moto 17.32 (-11.3x) 18.37 (-7.4x) 20.76 (-6.6x) 341.7 (-10.6x) 0
create_rest_api openstack 0.75 2.75 3.12 258.1 0
LocalStack 4.98 (-6.6x) 7.50 (-2.7x) 9.01 (-2.9x) 227.3 (-1.1x) 0
moto 14.74 (-19.7x) 1048.26 (-381.2x) 1060.76 (-340.0x) 30.0 (-8.6x) 0
get_rest_api openstack 1.32 1.96 2.69 4183.7 0
LocalStack 6.57 (-5.0x) 11.53 (-5.9x) 14.88 (-5.5x) 829.5 (-5.0x) 100
moto 16.79 (-12.7x) 17.74 (-9.1x) 23.53 (-8.7x) 349.5 (-12.0x) 0
get_rest_apis openstack 2.78 3.81 4.68 2075.2 0
LocalStack 6.45 (-2.3x) 10.37 (-2.7x) 12.41 (-2.7x) 860.8 (-2.4x) 100
moto 18.85 (-6.8x) 27.60 (-7.2x) 28.29 (-6.0x) 299.3 (-6.9x) 0

openstack vs LocalStack: 2.3×–6.6× faster on 3 ops (avg 4.6×)
openstack vs moto: 6.8×–19.7× faster on 4 ops (avg 12.6×)

CLOUDFORMATION

Operation Platform p50 (ms) p95 (ms) p99 (ms) RPS Errors
create_stack openstack 1.36 2.98 4.08 204.1 0
LocalStack 51.78 (-38.1x) 115.51 (-38.8x) 128.21 (-31.4x) 60.9 (-3.4x) 0
moto 12.19 (-9.0x) 21.29 (-7.1x) 23.82 (-5.8x) 139.4 (-1.5x) 0
describe_stacks openstack 2.56 4.01 4.40 2237.6 0
LocalStack 8.20 (-3.2x) 13.40 (-3.3x) 15.63 (-3.6x) 682.2 (-3.3x) 0
moto 16.94 (-6.6x) 17.88 (-4.5x) 19.61 (-4.5x) 350.3 (-6.4x) 0
get_template openstack 2.19 3.10 3.22 2617.7 0
LocalStack 6.53 (-3.0x) 11.78 (-3.8x) 13.13 (-4.1x) 804.5 (-3.3x) 0
moto 16.91 (-7.7x) 17.75 (-5.7x) 20.01 (-6.2x) 351.3 (-7.5x) 0
list_stacks openstack 3.16 4.78 5.00 1825.6 0
LocalStack 37.44 (-11.8x) 57.20 (-12.0x) 70.29 (-14.1x) 157.3 (-11.6x) 0
moto 16.85 (-5.3x) 17.86 (-3.7x) 20.10 (-4.0x) 351.6 (-5.2x) 0
update_stack openstack 2.31 3.46 3.79 2467.5 0
LocalStack 53.28 (-23.1x) 136.81 (-39.5x) 228.49 (-60.3x) 92.5 (-26.7x) 0
moto 22.06 (-9.5x) 24.40 (-7.1x) 26.48 (-7.0x) 269.2 (-9.2x) 0

openstack vs LocalStack: 3.0×–38.1× faster on 5 ops (avg 15.8×)
openstack vs moto: 5.3×–9.5× faster on 5 ops (avg 7.6×)

CLOUDWATCH

Operation Platform p50 (ms) p95 (ms) p99 (ms) RPS Errors
describe_alarms openstack 0.25 0.90 1.07 16824.8 0
LocalStack 6.96 (-27.8x) 50.86 (-56.5x) 54.90 (-51.3x) 598.5 (-28.1x) 0
moto 17.00 (-68.0x) 20.11 (-22.3x) 20.39 (-19.1x) 342.9 (-49.1x) 0
get_metric_statistics openstack 2.64 3.57 4.25 2196.2 0
LocalStack 52.75 (-20.0x) 55.58 (-15.6x) 61.89 (-14.6x) 112.5 (-19.5x) 0
moto 17.12 (-6.5x) 17.89 (-5.0x) 20.19 (-4.8x) 346.4 (-6.3x) 0
list_metrics openstack 3.53 6.06 6.65 1603.0 0
LocalStack 17.10 (-4.8x) 33.18 (-5.5x) 41.18 (-6.2x) 293.7 (-5.5x) 0
moto 26.60 (-7.5x) 34.28 (-5.7x) 37.10 (-5.6x) 244.4 (-6.6x) 0
put_metric_data openstack 2.69 4.08 4.65 2154.7 0
LocalStack 35.37 (-13.1x) 965.20 (-236.6x) 1317.01 (-283.2x) 23.1 (-93.3x) 0
moto 27.64 (-10.3x) 35.12 (-8.6x) 38.28 (-8.2x) 216.2 (-10.0x) 0

openstack vs LocalStack: 4.8×–27.8× faster on 4 ops (avg 16.5×)
openstack vs moto: 6.5×–68.0× faster on 4 ops (avg 23.1×)

DYNAMODB

Operation Platform p50 (ms) p95 (ms) p99 (ms) RPS Errors
get_item openstack 1.19 1.81 1.85 4697.2 0
LocalStack 15.64 (-13.1x) 27.38 (-15.1x) 31.02 (-16.8x) 350.9 (-13.4x) 0
moto 16.88 (-14.2x) 22.08 (-12.2x) 23.23 (-12.6x) 343.0 (-13.7x) 0
put_item openstack 0.32 1.27 1.60 11866.2 0
LocalStack 25.24 (-78.9x) 99.85 (-78.6x) 127.13 (-79.5x) 157.4 (-75.4x) 0
moto 16.97 (-53.0x) 18.94 (-14.9x) 19.82 (-12.4x) 348.9 (-34.0x) 0
query openstack 1.65 2.58 3.09 3401.5 0
LocalStack 18.83 (-11.4x) 110.40 (-42.8x) 117.34 (-38.0x) 225.5 (-15.1x) 0
moto 16.73 (-10.1x) 17.57 (-6.8x) 19.84 (-6.4x) 355.1 (-9.6x) 0
scan openstack 1.60 2.45 2.63 3535.8 0
LocalStack 20.80 (-13.0x) 48.78 (-19.9x) 56.22 (-21.4x) 261.2 (-13.5x) 0
moto 16.78 (-10.5x) 18.58 (-7.6x) 22.64 (-8.6x) 349.7 (-10.1x) 0

openstack vs LocalStack: 11.4×–78.9× faster on 4 ops (avg 29.1×)
openstack vs moto: 10.1×–53.0× faster on 4 ops (avg 22.0×)

EC2

Operation Platform p50 (ms) p95 (ms) p99 (ms) RPS Errors
create_security_group openstack 0.79 2.91 3.49 259.7 0
LocalStack 11.61 (-14.7x) 29.01 (-10.0x) 31.68 (-9.1x) 163.9 (-1.6x) 40
moto 12.06 (-15.3x) 16.03 (-5.5x) 16.75 (-4.8x) 163.3 (-1.6x) 0
create_vpc openstack 0.86 2.34 2.88 259.7 0
LocalStack 16.08 (-18.7x) 20.20 (-8.6x) 25.51 (-8.9x) 161.9 (-1.6x) 0
moto 11.19 (-13.0x) 17.31 (-7.4x) 21.36 (-7.4x) 159.4 (-1.6x) 0
describe_instances openstack 2.99 4.09 4.99 1923.1 0
LocalStack 191.82 (-64.2x) 718.26 (-175.6x) 896.88 (-179.7x) 24.3 (-79.1x) 0
moto 16.89 (-5.6x) 30.14 (-7.4x) 32.17 (-6.4x) 322.0 (-6.0x) 0
run_instances openstack 0.70 2.13 2.98 242.4 0
LocalStack 24.01 (-34.3x) 34.56 (-16.2x) 34.93 (-11.7x) 115.9 (-2.1x) 0
moto 10.97 (-15.7x) 16.54 (-7.8x) 17.50 (-5.9x) 167.4 (-1.4x) 0

openstack vs LocalStack: 14.7×–64.2× faster on 4 ops (avg 33.0×)
openstack vs moto: 5.6×–15.7× faster on 4 ops (avg 12.4×)

ECR

Operation Platform p50 (ms) p95 (ms) p99 (ms) RPS Errors
batch_get_image openstack 2.64 4.06 4.78 2197.8 0
moto 16.88 (-6.4x) 18.78 (-4.6x) 19.85 (-4.2x) 351.2 (-6.3x) 0
create_repository openstack 0.64 2.61 3.59 263.2 0
moto 11.53 (-18.0x) 17.38 (-6.7x) 18.62 (-5.2x) 170.9 (-1.5x) 0
describe_repositories openstack 2.65 4.11 4.41 2132.0 0
moto 16.94 (-6.4x) 18.37 (-4.5x) 19.46 (-4.4x) 350.7 (-6.1x) 0
list_images openstack 1.58 2.51 2.95 3532.4 0
moto 16.85 (-10.7x) 18.36 (-7.3x) 20.02 (-6.8x) 352.6 (-10.0x) 0

⚠️ LocalStack: seed failed (seed request failed) — excluded from this service's benchmarks
⚠️ LocalStack: seed failed (seed request failed) — excluded from this service's benchmarks

openstack vs moto: 6.4×–18.0× faster on 4 ops (avg 10.4×)

EVENTBRIDGE

Operation Platform p50 (ms) p95 (ms) p99 (ms) RPS Errors
describe_event_bus openstack 1.39 2.03 2.80 3940.5 0
moto 16.78 (-12.1x) 17.83 (-8.8x) 19.66 (-7.0x) 353.2 (-11.2x) 0
describe_rule openstack 2.08 3.46 4.52 2653.6 0
moto 16.75 (-8.1x) 18.14 (-5.2x) 20.06 (-4.4x) 352.5 (-7.5x) 0
list_event_buses openstack 1.71 2.76 3.41 3294.1 0
moto 16.87 (-9.9x) 18.50 (-6.7x) 19.54 (-5.7x) 348.7 (-9.4x) 0
list_rules openstack 2.78 3.77 4.71 2067.9 0
moto 16.83 (-6.1x) 17.66 (-4.7x) 19.77 (-4.2x) 352.7 (-5.9x) 0
list_targets_by_rule openstack 1.46 2.22 2.38 3869.4 0
moto 16.92 (-11.6x) 18.18 (-8.2x) 19.86 (-8.3x) 350.5 (-11.0x) 0
put_rule openstack 0.89 3.35 4.82 274.0 0
moto 11.37 (-12.8x) 16.91 (-5.0x) 18.27 (-3.8x) 168.1 (-1.6x) 0

⚠️ LocalStack: seed failed (seed request failed) — excluded from this service's benchmarks
⚠️ LocalStack: seed failed (seed request failed) — excluded from this service's benchmarks

openstack vs moto: 6.1×–12.8× faster on 6 ops (avg 10.1×)

FIREHOSE

Operation Platform p50 (ms) p95 (ms) p99 (ms) RPS Errors
create_delivery_stream openstack 0.81 2.64 3.40 266.7 0
LocalStack 6.10 (-7.5x) 9.90 (-3.8x) 10.16 (-3.0x) 217.4 (-1.2x) 0
moto 12.40 (-15.3x) 18.11 (-6.9x) 19.36 (-5.7x) 159.4 (-1.7x) 0
describe_delivery_stream openstack 2.38 3.41 3.86 2396.0 0
LocalStack 6.28 (-2.6x) 9.62 (-2.8x) 11.49 (-3.0x) 863.6 (-2.8x) 0
moto 16.83 (-7.1x) 17.88 (-5.2x) 19.43 (-5.0x) 352.8 (-6.8x) 0
list_delivery_streams openstack 2.02 2.99 3.48 2828.7 0
LocalStack 6.50 (-3.2x) 10.93 (-3.7x) 11.85 (-3.4x) 862.9 (-3.3x) 0
moto 16.79 (-8.3x) 18.42 (-6.2x) 19.62 (-5.6x) 352.0 (-8.0x) 0
put_record openstack 1.10 1.52 2.00 5043.8 0
LocalStack 6.55 (-6.0x) 10.17 (-6.7x) 12.07 (-6.0x) 849.5 (-5.9x) 0
moto 16.91 (-15.4x) 17.87 (-11.8x) 19.39 (-9.7x) 351.8 (-14.3x) 0
put_record_batch openstack 1.99 2.72 2.86 2860.5 0
LocalStack 6.89 (-3.5x) 10.92 (-4.0x) 16.22 (-5.7x) 803.1 (-3.6x) 0
moto 16.80 (-8.4x) 18.92 (-7.0x) 19.72 (-6.9x) 352.1 (-8.1x) 0

openstack vs LocalStack: 2.6×–7.5× faster on 5 ops (avg 4.6×)
openstack vs moto: 7.1×–15.4× faster on 5 ops (avg 10.9×)

IAM

Operation Platform p50 (ms) p95 (ms) p99 (ms) RPS Errors
create_role openstack 0.95 2.66 2.94 266.7 0
LocalStack 29.58 (-31.1x) 35.46 (-13.3x) 37.36 (-12.7x) 113.6 (-2.3x) 0
moto 12.66 (-13.3x) 18.40 (-6.9x) 24.46 (-8.3x) 148.7 (-1.8x) 0
create_user openstack 0.76 2.46 2.91 263.2 0
LocalStack 10.51 (-13.8x) 14.26 (-5.8x) 16.18 (-5.6x) 186.9 (-1.4x) 0
moto 11.03 (-14.5x) 16.57 (-6.7x) 17.41 (-6.0x) 170.2 (-1.5x) 0
get_role openstack 2.97 4.45 6.27 1954.3 0
LocalStack 30.20 (-10.2x) 55.10 (-12.4x) 69.46 (-11.1x) 182.2 (-10.7x) 0
moto 16.96 (-5.7x) 18.99 (-4.3x) 19.80 (-3.2x) 348.2 (-5.6x) 0
get_user openstack 0.82 3.21 3.50 264.9 0
LocalStack 9.56 (-11.7x) 13.56 (-4.2x) 14.21 (-4.1x) 194.2 (-1.4x) 0
moto 11.64 (-14.2x) 17.28 (-5.4x) 17.70 (-5.1x) 159.4 (-1.7x) 0
list_users openstack 3.14 4.28 4.81 1817.2 0
LocalStack 29.78 (-9.5x) 49.37 (-11.5x) 61.36 (-12.8x) 195.4 (-9.3x) 0
moto 16.89 (-5.4x) 19.19 (-4.5x) 20.24 (-4.2x) 347.4 (-5.2x) 0

openstack vs LocalStack: 9.5×–31.1× faster on 5 ops (avg 15.3×)
openstack vs moto: 5.4×–14.5× faster on 5 ops (avg 10.6×)

KINESIS

Operation Platform p50 (ms) p95 (ms) p99 (ms) RPS Errors
describe_stream openstack 2.67 4.11 4.51 2164.8 0
LocalStack 54.47 (-20.4x) 68.07 (-16.6x) 80.78 (-17.9x) 107.0 (-20.2x) 0
moto 18.00 (-6.7x) 19.88 (-4.8x) 25.74 (-5.7x) 328.0 (-6.6x) 0
list_streams openstack 1.21 1.89 2.06 4506.2 0
LocalStack 62.90 (-52.0x) 94.55 (-50.0x) 98.64 (-47.9x) 89.3 (-50.5x) 0
moto 17.09 (-14.1x) 18.96 (-10.0x) 19.86 (-9.6x) 347.5 (-13.0x) 0
put_record openstack 1.14 1.76 1.83 4854.6 0
LocalStack 69.72 (-61.2x) 114.95 (-65.3x) 185.72 (-101.5x) 80.6 (-60.2x) 0
moto 17.01 (-14.9x) 18.40 (-10.5x) 20.93 (-11.4x) 348.8 (-13.9x) 0

openstack vs LocalStack: 20.4×–61.2× faster on 3 ops (avg 44.5×)
openstack vs moto: 6.7×–14.9× faster on 3 ops (avg 11.9×)

KMS

Operation Platform p50 (ms) p95 (ms) p99 (ms) RPS Errors
create_key openstack 0.78 2.75 5.25 242.4 0
LocalStack 7.67 (-9.8x) 63.59 (-23.1x) 64.03 (-12.2x) 163.9 (-1.5x) 0
moto 197.29 (-252.9x) 473.14 (-172.1x) 570.08 (-108.6x) 14.1 (-17.2x) 0
describe_key openstack 2.41 3.61 4.49 2348.0 0
LocalStack 6.12 (-2.5x) 10.26 (-2.8x) 11.19 (-2.5x) 879.2 (-2.7x) 0
moto 18.94 (-7.9x) 22.36 (-6.2x) 23.76 (-5.3x) 315.8 (-7.4x) 0
generate_data_key openstack 2.25 3.57 4.26 2542.3 0
LocalStack 9.81 (-4.4x) 16.16 (-4.5x) 16.96 (-4.0x) 592.7 (-4.3x) 0
moto 16.92 (-7.5x) 30.10 (-8.4x) 39.65 (-9.3x) 277.7 (-9.2x) 0
list_keys openstack 3.22 4.44 4.66 1781.7 0
LocalStack 6.91 (-2.1x) 11.34 (-2.6x) 14.17 (-3.0x) 781.5 (-2.3x) 0
moto 17.32 (-5.4x) 37.37 (-8.4x) 39.48 (-8.5x) 268.1 (-6.6x) 0

openstack vs LocalStack: 2.1×–9.8× faster on 4 ops (avg 4.7×)
openstack vs moto: 5.4×–252.9× faster on 4 ops (avg 68.4×)

LAMBDA

Operation Platform p50 (ms) p95 (ms) p99 (ms) RPS Errors
delete_function openstack 0.56 0.56 0.56 807.7 0
LocalStack 7.68 (-13.7x) 7.68 (-13.7x) 7.68 (-13.7x) 119.7 (-6.7x) 0
moto 5.28 (-9.4x) 5.28 (-9.4x) 5.28 (-9.4x) 155.6 (-5.2x) 0
get_function openstack 2.85 4.34 4.61 1997.4 0
LocalStack 8.13 (-2.9x) 14.40 (-3.3x) 18.08 (-3.9x) 686.0 (-2.9x) 0
moto 24.22 (-8.5x) 30.58 (-7.0x) 34.32 (-7.4x) 242.9 (-8.2x) 0
invoke openstack 0.52 1.50 1.58 8628.6 0
LocalStack 10.73 (-20.6x) 18.10 (-12.1x) 26.00 (-16.5x) 504.5 (-17.1x) 100
moto 35.06 (-67.4x) 70.75 (-47.2x) 82.83 (-52.4x) 155.9 (-55.3x) 0
list_functions openstack 2.71 3.72 4.35 2122.1 0
LocalStack 6.14 (-2.3x) 9.95 (-2.7x) 16.22 (-3.7x) 905.9 (-2.3x) 0
moto 17.10 (-6.3x) 22.62 (-6.1x) 23.79 (-5.5x) 336.2 (-6.3x) 0
update_function_code openstack 3.38 7.22 8.93 1597.3 0
LocalStack 58.92 (-17.4x) 87.66 (-12.1x) 121.73 (-13.6x) 96.3 (-16.6x) 100
moto 23.46 (-6.9x) 29.30 (-4.1x) 32.62 (-3.7x) 245.8 (-6.5x) 0
update_function_configuration openstack 3.53 5.77 6.23 1607.5 0
LocalStack 8.85 (-2.5x) 13.44 (-2.3x) 15.99 (-2.6x) 658.0 (-2.4x) 100
moto 18.48 (-5.2x) 28.01 (-4.9x) 30.15 (-4.8x) 291.0 (-5.5x) 100

openstack vs LocalStack: 2.3×–20.6× faster on 6 ops (avg 9.9×)
openstack vs moto: 5.2×–67.4× faster on 6 ops (avg 17.3×)

OPENSEARCH

Operation Platform p50 (ms) p95 (ms) p99 (ms) RPS Errors
create_domain openstack 1.01 3.35 7.28 170.9 0
LocalStack 4.02 (-4.0x) 7.97 (-2.4x) 9.09 (-1.2x) 153.3 (-1.1x) 0
moto 13.58 (-13.4x) 21.28 (-6.4x) 22.78 (-3.1x) 122.0 (-1.4x) 0
describe_domain openstack 3.24 5.06 5.26 1750.0 0
LocalStack 8.78 (-2.7x) 14.83 (-2.9x) 18.18 (-3.5x) 628.1 (-2.8x) 100
moto 22.60 (-7.0x) 29.78 (-5.9x) 33.00 (-6.3x) 256.7 (-6.8x) 0
describe_domain_config openstack 2.96 4.84 6.15 1968.6 0
LocalStack 10.02 (-3.4x) 17.82 (-3.7x) 23.73 (-3.9x) 558.8 (-3.5x) 100
moto 19.15 (-6.5x) 25.05 (-5.2x) 30.71 (-5.0x) 298.9 (-6.6x) 0
list_domain_names openstack 2.28 3.82 4.54 2417.3 0
LocalStack 9.19 (-4.0x) 14.18 (-3.7x) 18.53 (-4.1x) 618.7 (-3.9x) 100
moto 20.95 (-9.2x) 25.41 (-6.7x) 26.66 (-5.9x) 277.2 (-8.7x) 100
update_domain_config openstack 2.90 4.43 5.34 2051.6 0
LocalStack 7.78 (-2.7x) 13.79 (-3.1x) 18.17 (-3.4x) 725.3 (-2.8x) 0
moto 23.05 (-7.9x) 32.31 (-7.3x) 35.62 (-6.7x) 250.6 (-8.2x) 0

openstack vs LocalStack: 2.7×–4.0× faster on 5 ops (avg 3.4×)
openstack vs moto: 6.5×–13.4× faster on 5 ops (avg 8.8×)

REDSHIFT

Operation Platform p50 (ms) p95 (ms) p99 (ms) RPS Errors
create_cluster openstack 0.82 3.31 4.48 223.5 0
LocalStack 13.84 (-16.9x) 19.33 (-5.8x) 21.47 (-4.8x) 154.4 (-1.4x) 0
moto 11.36 (-13.9x) 17.73 (-5.4x) 18.91 (-4.2x) 164.6 (-1.4x) 0
describe_clusters openstack 3.19 4.32 5.40 1808.5 0
LocalStack 126.01 (-39.5x) 670.21 (-155.1x) 717.88 (-132.9x) 37.0 (-48.9x) 0
moto 16.93 (-5.3x) 18.18 (-4.2x) 20.98 (-3.9x) 349.0 (-5.2x) 0
modify_cluster openstack 2.75 3.88 4.78 2070.0 0
LocalStack 13.13 (-4.8x) 22.66 (-5.8x) 24.00 (-5.0x) 428.3 (-4.8x) 0
moto 16.96 (-6.2x) 18.40 (-4.7x) 21.03 (-4.4x) 348.7 (-5.9x) 0
reboot_cluster openstack 2.66 3.82 4.31 2126.5 0
LocalStack 8.91 (-3.3x) 13.50 (-3.5x) 15.84 (-3.7x) 645.5 (-3.3x) 100
moto 16.90 (-6.4x) 20.14 (-5.3x) 20.31 (-4.7x) 348.6 (-6.1x) 0

openstack vs LocalStack: 3.3×–39.5× faster on 4 ops (avg 16.1×)
openstack vs moto: 5.3×–13.9× faster on 4 ops (avg 7.9×)

ROUTE53

Operation Platform p50 (ms) p95 (ms) p99 (ms) RPS Errors
change_resource_record_sets openstack 2.39 3.70 4.00 2379.4 0
LocalStack 5.15 (-2.2x) 8.53 (-2.3x) 11.54 (-2.9x) 1063.6 (-2.2x) 0
moto 17.80 (-7.4x) 18.66 (-5.0x) 22.37 (-5.6x) 332.0 (-7.2x) 0
create_hosted_zone openstack 0.88 2.80 3.18 268.5 0
moto 24.25 (-27.6x) 38.76 (-13.8x) 42.70 (-13.4x) 101.3 (-2.7x) 0
get_hosted_zone openstack 2.50 5.05 7.38 2061.0 0
moto 40.85 (-16.3x) 53.73 (-10.6x) 56.52 (-7.7x) 139.8 (-14.7x) 0
list_hosted_zones openstack 2.96 4.08 4.92 1935.5 0
moto 40.46 (-13.7x) 43.62 (-10.7x) 45.96 (-9.3x) 146.4 (-13.2x) 0
list_resource_record_sets openstack 2.29 3.31 3.33 2515.2 0
moto 64.36 (-28.1x) 68.70 (-20.8x) 70.70 (-21.2x) 92.4 (-27.2x) 0

⚠️ LocalStack: seed failed (seed request failed) — excluded from this service's benchmarks

openstack vs LocalStack: 2.2×–2.2× faster on 1 op (avg 2.2×)
openstack vs moto: 7.4×–28.1× faster on 5 ops (avg 18.6×)

S3

S3 — 1MB

Operation Platform p50 (ms) p95 (ms) p99 (ms) RPS Errors
get_object_1mb openstack 1.46 2.47 2.83 3768.4 0
LocalStack 25.29 (-17.3x) 29.01 (-11.7x) 33.79 (-11.9x) 234.6 (-16.1x) 0
moto 21.49 (-14.7x) 24.80 (-10.0x) 223.48 (-79.0x) 177.8 (-21.2x) 0
head_object_1mb openstack 0.23 0.73 1.03 18145.2 0
LocalStack 6.08 (-26.4x) 9.63 (-13.2x) 12.37 (-12.0x) 896.8 (-20.2x) 0
moto 16.15 (-70.2x) 17.06 (-23.4x) 19.35 (-18.8x) 367.0 (-49.4x) 0
list_objects_v2_1mb openstack 2.59 3.73 4.34 2225.4 0
LocalStack 6.07 (-2.3x) 9.74 (-2.6x) 10.74 (-2.5x) 895.2 (-2.5x) 0
moto 16.63 (-6.4x) 18.99 (-5.1x) 25.12 (-5.8x) 348.8 (-6.4x) 0
put_object_1mb openstack 8.31 17.54 22.52 667.4 0
LocalStack 22.21 (-2.7x) 35.54 (-2.0x) 54.31 (-2.4x) 250.5 (-2.7x) 0
moto 44.93 (-5.4x) 48.88 (-2.8x) 50.22 (-2.2x) 131.7 (-5.1x) 0

openstack vs LocalStack: 2.3×–26.4× faster on 4 ops (avg 12.2×)
openstack vs moto: 5.4×–70.2× faster on 4 ops (avg 24.2×)

S3 — 10MB

Operation Platform p50 (ms) p95 (ms) p99 (ms) RPS Errors
get_object_10mb openstack 14.82 21.04 24.92 389.7 0
LocalStack 184.27 (-12.4x) 191.22 (-9.1x) 193.23 (-7.8x) 32.7 (-11.9x) 0
moto 30.64 (-2.1x) 37.81 (-1.8x) 40.62 (-1.6x) 187.9 (-2.1x) 0
head_object_10mb openstack 0.23 0.77 0.89 14193.7 0
LocalStack 6.15 (-26.7x) 10.81 (-14.0x) 11.90 (-13.4x) 878.8 (-16.2x) 0
moto 16.09 (-70.0x) 16.58 (-21.5x) 18.67 (-21.0x) 368.6 (-38.5x) 0
list_objects_v2_10mb openstack 2.52 3.93 4.38 2114.5 0
LocalStack 6.46 (-2.6x) 10.88 (-2.8x) 11.41 (-2.6x) 853.9 (-2.5x) 0
moto 16.85 (-6.7x) 18.57 (-4.7x) 20.75 (-4.7x) 349.6 (-6.0x) 0
put_object_10mb openstack 112.29 203.92 296.91 47.6 0
LocalStack 154.37 (-1.4x) 166.65 (+1.2x) 219.37 (+1.4x) 37.3 (-1.3x) 0
moto 226.86 (-2.0x) 288.82 (-1.4x) 308.96 25.5 (-1.9x) 4

openstack vs LocalStack: 1.4×–26.7× faster on 4 ops (avg 10.8×)
openstack vs moto: 2.0×–70.0× faster on 4 ops (avg 20.2×)

S3 — 50MB

Operation Platform p50 (ms) p95 (ms) p99 (ms) RPS Errors
get_object_50mb openstack 48.36 58.34 58.34 80.5 0
LocalStack 592.81 (-12.3x) 605.65 (-10.4x) 605.65 (-10.4x) 6.7 (-12.0x) 0
moto 56.92 (-1.2x) 79.73 (-1.4x) 79.73 (-1.4x) 65.1 (-1.2x) 0
head_object_50mb openstack 0.23 0.73 0.73 8298.5 0
LocalStack 4.76 (-20.7x) 6.49 (-8.9x) 6.49 (-8.9x) 808.8 (-10.3x) 0
moto 10.68 (-46.4x) 12.53 (-17.2x) 12.53 (-17.2x) 361.9 (-22.9x) 0
list_objects_v2_50mb openstack 1.95 3.90 3.90 1610.3 0
LocalStack 4.71 (-2.4x) 7.22 (-1.9x) 7.22 (-1.9x) 761.1 (-2.1x) 0
moto 11.36 (-5.8x) 13.75 (-3.5x) 13.75 (-3.5x) 340.0 (-4.7x) 0
put_object_50mb openstack 239.56 938.35 938.35 13.1 0
LocalStack 484.87 (-2.0x) 681.49 (+1.4x) 681.49 (+1.4x) 7.7 (-1.7x) 0
moto 668.08 (-2.8x) 697.83 (+1.3x) 697.83 (+1.3x) 6.0 (-2.2x) 14

openstack vs LocalStack: 2.0×–20.7× faster on 4 ops (avg 9.3×)
openstack vs moto: 1.2×–46.4× faster on 4 ops (avg 14.1×)

S3 — 100MB

Operation Platform p50 (ms) p95 (ms) p99 (ms) RPS Errors
get_object_100mb openstack 51.09 56.92 56.92 37.9 0
LocalStack 541.02 (-10.6x) 545.96 (-9.6x) 545.96 (-9.6x) 3.7 (-10.2x) 0
moto 58.48 (-1.1x) 63.42 (-1.1x) 63.42 (-1.1x) 33.3 (-1.1x) 0
head_object_100mb openstack 0.20 0.62 0.62 4787.5 0
LocalStack 2.38 (-11.9x) 4.18 (-6.7x) 4.18 (-6.7x) 708.0 (-6.8x) 0
moto 5.44 (-27.2x) 6.98 (-11.3x) 6.98 (-11.3x) 344.3 (-13.9x) 0
list_objects_v2_100mb openstack 1.28 1.49 1.49 1331.2 0
LocalStack 2.80 (-2.2x) 4.38 (-2.9x) 4.38 (-2.9x) 656.4 (-2.0x) 0
moto 6.19 (-4.8x) 8.27 (-5.6x) 8.27 (-5.6x) 299.9 (-4.4x) 0
put_object_100mb openstack 307.79 316.37 316.37 6.5 0
LocalStack 490.40 (-1.6x) 678.04 (-2.1x) 678.04 (-2.1x) 3.7 (-1.8x) 0
moto 569.35 (-1.8x) 709.71 (-2.2x) 709.71 (-2.2x) 3.4 (-1.9x) 4

openstack vs LocalStack: 1.6×–11.9× faster on 4 ops (avg 6.6×)
openstack vs moto: 1.1×–27.2× faster on 4 ops (avg 8.8×)

SECRETSMANAGER

Operation Platform p50 (ms) p95 (ms) p99 (ms) RPS Errors
create_secret openstack 0.90 3.21 3.90 264.9 0
LocalStack 8.12 (-9.0x) 11.49 (-3.6x) 12.43 (-3.2x) 201.0 (-1.3x) 0
moto 11.07 (-12.3x) 16.53 (-5.1x) 17.58 (-4.5x) 171.7 (-1.5x) 0
get_secret_value openstack 2.25 3.25 3.66 2548.4 0
LocalStack 6.49 (-2.9x) 9.74 (-3.0x) 12.89 (-3.5x) 840.9 (-3.0x) 0
moto 16.86 (-7.5x) 17.74 (-5.5x) 20.16 (-5.5x) 351.2 (-7.3x) 0
list_secrets openstack 2.77 3.87 4.18 2076.9 0
LocalStack 22.49 (-8.1x) 35.50 (-9.2x) 47.94 (-11.5x) 258.2 (-8.0x) 0
moto 17.13 (-6.2x) 20.21 (-5.2x) 20.50 (-4.9x) 341.1 (-6.1x) 0
put_secret_value openstack 2.08 2.90 3.77 2710.1 0
LocalStack 6.07 (-2.9x) 10.30 (-3.6x) 13.34 (-3.5x) 868.7 (-3.1x) 100
moto 16.83 (-8.1x) 17.65 (-6.1x) 22.12 (-5.9x) 352.6 (-7.7x) 0
update_secret openstack 1.52 2.30 2.49 3690.1 0
LocalStack 6.30 (-4.1x) 9.91 (-4.3x) 12.10 (-4.9x) 877.6 (-4.2x) 100
moto 16.81 (-11.1x) 17.65 (-7.7x) 20.15 (-8.1x) 353.0 (-10.5x) 0

openstack vs LocalStack: 2.9×–9.0× faster on 5 ops (avg 5.4×)
openstack vs moto: 6.2×–12.3× faster on 5 ops (avg 9.0×)

SES

Operation Platform p50 (ms) p95 (ms) p99 (ms) RPS Errors
get_identity_verification_attributes openstack 2.24 3.07 3.83 2567.7 0
LocalStack 6.75 (-3.0x) 11.37 (-3.7x) 13.66 (-3.6x) 801.9 (-3.2x) 0
moto 17.10 (-7.6x) 19.17 (-6.2x) 20.06 (-5.2x) 346.9 (-7.4x) 0
list_identities openstack 2.32 3.64 3.77 2466.8 0
LocalStack 9.92 (-4.3x) 16.42 (-4.5x) 19.51 (-5.2x) 571.0 (-4.3x) 0
moto 16.89 (-7.3x) 17.89 (-4.9x) 19.82 (-5.3x) 351.0 (-7.0x) 0
send_email openstack 2.36 3.53 4.46 2403.3 0
LocalStack 11.60 (-4.9x) 16.85 (-4.8x) 20.87 (-4.7x) 509.0 (-4.7x) 0
moto 17.15 (-7.3x) 19.01 (-5.4x) 20.87 (-4.7x) 345.3 (-7.0x) 0
send_raw_email openstack 2.31 3.66 4.00 2453.7 0
LocalStack 8.44 (-3.7x) 12.59 (-3.4x) 15.97 (-4.0x) 694.8 (-3.5x) 100
moto 17.21 (-7.5x) 18.20 (-5.0x) 20.79 (-5.2x) 344.5 (-7.1x) 0
verify_email_identity openstack 0.73 3.40 5.38 196.1 0
LocalStack 9.23 (-12.6x) 12.51 (-3.7x) 13.14 (-2.4x) 191.4 0
moto 11.95 (-16.4x) 17.50 (-5.1x) 18.19 (-3.4x) 163.3 (-1.2x) 0

openstack vs LocalStack: 3.0×–12.6× faster on 5 ops (avg 5.7×)
openstack vs moto: 7.3×–16.4× faster on 5 ops (avg 9.2×)

SNS

Operation Platform p50 (ms) p95 (ms) p99 (ms) RPS Errors
get_topic_attributes openstack 2.49 3.56 5.05 2296.1 0
LocalStack 9.71 (-3.9x) 459.45 (-129.1x) 465.90 (-92.3x) 157.1 (-14.6x) 0
moto 18.63 (-7.5x) 19.68 (-5.5x) 22.27 (-4.4x) 318.8 (-7.2x) 0
list_subscriptions openstack 2.42 3.92 4.82 2297.7 0
LocalStack 6.70 (-2.8x) 11.20 (-2.9x) 13.61 (-2.8x) 817.2 (-2.8x) 0
moto 18.02 (-7.4x) 18.98 (-4.8x) 21.78 (-4.5x) 328.5 (-7.0x) 0
list_topics openstack 2.59 4.39 4.78 2087.9 0
LocalStack 8.76 (-3.4x) 14.52 (-3.3x) 14.85 (-3.1x) 634.4 (-3.3x) 0
moto 17.73 (-6.8x) 19.52 (-4.4x) 21.73 (-4.5x) 334.3 (-6.2x) 0
publish openstack 2.25 3.09 3.15 2550.8 0
LocalStack 6.85 (-3.0x) 11.91 (-3.9x) 13.69 (-4.3x) 802.2 (-3.2x) 0
moto 18.01 (-8.0x) 25.18 (-8.1x) 26.66 (-8.5x) 320.3 (-8.0x) 0

openstack vs LocalStack: 2.8×–3.9× faster on 4 ops (avg 3.3×)
openstack vs moto: 6.8×–8.0× faster on 4 ops (avg 7.4×)

SQS

Operation Platform p50 (ms) p95 (ms) p99 (ms) RPS Errors
delete_message openstack 2.04 3.14 3.20 2795.0 0
LocalStack 10.79 (-5.3x) 19.71 (-6.3x) 25.30 (-7.9x) 515.0 (-5.4x) 0
moto 21.48 (-10.5x) 23.39 (-7.4x) 27.87 (-8.7x) 289.7 (-9.6x) 0
get_queue_attributes openstack 2.49 3.88 4.07 2291.6 0
LocalStack 7.64 (-3.1x) 13.82 (-3.6x) 16.07 (-3.9x) 706.6 (-3.2x) 0
moto 19.81 (-8.0x) 99.43 (-25.6x) 100.06 (-24.6x) 242.8 (-9.4x) 0
list_queues openstack 2.21 3.42 3.55 2532.2 0
LocalStack 7.25 (-3.3x) 10.96 (-3.2x) 15.20 (-4.3x) 795.6 (-3.2x) 0
moto 17.99 (-8.1x) 19.05 (-5.6x) 21.14 (-6.0x) 330.5 (-7.7x) 0
receive_message openstack 3.45 4.96 5.57 1707.1 0
LocalStack 11.90 (-3.4x) 19.29 (-3.9x) 24.76 (-4.4x) 468.8 (-3.6x) 0
moto 43.71 (-12.7x) 59.75 (-12.0x) 65.32 (-11.7x) 127.3 (-13.4x) 0
send_message openstack 2.40 3.66 3.75 2401.0 0
LocalStack 10.97 (-4.6x) 20.12 (-5.5x) 25.00 (-6.7x) 499.3 (-4.8x) 0
moto 28.99 (-12.1x) 33.89 (-9.3x) 38.28 (-10.2x) 201.8 (-11.9x) 0

openstack vs LocalStack: 3.1×–5.3× faster on 5 ops (avg 3.9×)
openstack vs moto: 8.0×–12.7× faster on 5 ops (avg 10.3×)

SSM

Operation Platform p50 (ms) p95 (ms) p99 (ms) RPS Errors
describe_parameters openstack 2.18 3.18 3.41 2676.0 0
LocalStack 8.37 (-3.8x) 14.01 (-4.4x) 18.33 (-5.4x) 656.7 (-4.1x) 0
moto 20.19 (-9.3x) 22.19 (-7.0x) 23.77 (-7.0x) 294.8 (-9.1x) 0
get_parameter openstack 2.31 3.53 3.69 2478.8 0
LocalStack 12.22 (-5.3x) 29.59 (-8.4x) 64.96 (-17.6x) 408.7 (-6.1x) 0
moto 16.94 (-7.3x) 20.03 (-5.7x) 21.09 (-5.7x) 341.8 (-7.3x) 0
get_parameters openstack 2.94 4.26 4.58 2045.8 0
LocalStack 12.29 (-4.2x) 20.86 (-4.9x) 24.80 (-5.4x) 455.1 (-4.5x) 0
moto 16.81 (-5.7x) 17.81 (-4.2x) 19.69 (-4.3x) 352.0 (-5.8x) 0
get_parameters_by_path openstack 2.33 3.61 4.36 2447.0 0
LocalStack 8.49 (-3.6x) 13.93 (-3.9x) 16.63 (-3.8x) 657.5 (-3.7x) 0
moto 16.90 (-7.3x) 18.08 (-5.0x) 20.04 (-4.6x) 350.1 (-7.0x) 0
put_parameter openstack 0.82 1.19 1.88 6724.2 0
LocalStack 24.42 (-29.8x) 34.61 (-29.1x) 40.31 (-21.4x) 238.6 (-28.2x) 0
moto 16.86 (-20.6x) 18.63 (-15.7x) 20.11 (-10.7x) 350.4 (-19.2x) 0

openstack vs LocalStack: 3.6×–29.8× faster on 5 ops (avg 9.3×)
openstack vs moto: 5.7×–20.6× faster on 5 ops (avg 10.0×)

STEPFUNCTIONS

Operation Platform p50 (ms) p95 (ms) p99 (ms) RPS Errors
create_state_machine openstack 0.79 2.44 2.93 264.9 0
LocalStack 9.78 (-12.4x) 14.95 (-6.1x) 15.42 (-5.3x) 182.6 (-1.5x) 0
moto 11.71 (-14.8x) 16.72 (-6.9x) 17.16 (-5.9x) 166.7 (-1.6x) 0
describe_state_machine openstack 2.59 3.64 4.29 2216.2 0
LocalStack 6.14 (-2.4x) 10.81 (-3.0x) 14.84 (-3.5x) 841.9 (-2.6x) 0
moto 17.13 (-6.6x) 17.92 (-4.9x) 20.42 (-4.8x) 347.6 (-6.4x) 0
list_state_machines openstack 2.64 3.62 3.84 2166.5 0
LocalStack 8.27 (-3.1x) 13.94 (-3.9x) 14.27 (-3.7x) 651.0 (-3.3x) 0
moto 16.87 (-6.4x) 18.36 (-5.1x) 19.81 (-5.2x) 351.0 (-6.2x) 0
start_execution openstack 0.79 2.90 3.51 253.2 0
LocalStack 120.49 (-152.5x) 180.27 (-62.2x) 632.08 (-180.1x) 24.4 (-10.4x) 0
moto 12.42 (-15.7x) 18.20 (-6.3x) 19.64 (-5.6x) 148.7 (-1.7x) 0

openstack vs LocalStack: 2.4×–152.5× faster on 4 ops (avg 42.6×)
openstack vs moto: 6.4×–15.7× faster on 4 ops (avg 10.9×)

STS

Operation Platform p50 (ms) p95 (ms) p99 (ms) RPS Errors
assume_role openstack 2.66 4.14 4.32 2156.4 0
LocalStack 10.37 (-3.9x) 17.67 (-4.3x) 24.53 (-5.7x) 522.1 (-4.1x) 0
moto 16.98 (-6.4x) 19.81 (-4.8x) 20.73 (-4.8x) 344.6 (-6.3x) 0
get_access_key_info openstack 2.64 4.04 4.44 2145.8 0
LocalStack 8.31 (-3.1x) 13.86 (-3.4x) 16.24 (-3.7x) 663.5 (-3.2x) 100
moto 17.30 (-6.6x) 18.58 (-4.6x) 20.68 (-4.7x) 344.2 (-6.2x) 0
get_caller_identity openstack 2.32 3.60 3.73 2462.4 0
LocalStack 8.63 (-3.7x) 17.49 (-4.9x) 21.91 (-5.9x) 614.3 (-4.0x) 0
moto 16.89 (-7.3x) 18.36 (-5.1x) 20.13 (-5.4x) 349.7 (-7.0x) 0
get_session_token openstack 2.62 3.61 4.46 2185.9 0
LocalStack 9.20 (-3.5x) 16.09 (-4.5x) 19.57 (-4.4x) 584.5 (-3.7x) 0
moto 16.82 (-6.4x) 17.99 (-5.0x) 20.30 (-4.6x) 351.6 (-6.2x) 0

openstack vs LocalStack: 3.1×–3.9× faster on 4 ops (avg 3.6×)
openstack vs moto: 6.4×–7.3× faster on 4 ops (avg 6.7×)

Overall Performance

openstack vs LocalStack: 1.4×–501.0× faster on 105 ops (avg 17.1×)
openstack vs moto: 1.1×–305.9× faster on 120 ops (avg 16.7×)

Normalize service alias handling across config, gateway, protocol selection, and the integration harness so parity probes route consistently. Record LocalStack free-tier gaps for cloudtrail, cognito, ecs, elasticache, and rds to unblock all-services parity.
@JesseKoldewijn

Copy link
Copy Markdown
Owner Author

@coderabbitai full review

@coderabbitai

coderabbitai Bot commented May 26, 2026

Copy link
Copy Markdown
✅ Actions performed

Full review triggered.

@coderabbitai

coderabbitai Bot commented May 26, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

Adds five new service crates, canonicalizes service names, wires optional provider registration, extends many providers and their stores, introduces numerous integration and performance tests, and updates CI/benchmark workflow guards.

Changes

AWS services expansion and wiring

Layer / File(s) Summary
Workspace, routing, and registration
Cargo.toml, crates/openstack/src/main.rs, crates/tests/integration/..., crates/gateway/src/server.rs
Adds workspace members/dependencies, registers optional providers (ecs, rds, cognito-idp, elasticache, cloudtrail), adjusts gateway tests to import service_from_target, and extends integration harness service inventory.
Service canonicalization & protocol routing
crates/aws-protocol/..., crates/config/src/services.rs, crates/aws-protocol/tests/*, crates/config/tests/*
Introduces canonical_service_name, uses it for AwsProtocol selection and ServicesConfig parsing/override keys; tests verify alias and underscore/hyphen equivalence (e.g., cognitocognito-idp, esopensearch).
Benchmark & CI workflow guards
.github/workflows/benchmark.yml, .github/workflows/ci.yml
Adds env.ACT guards for local act runs, conditional Docker build select, and tightens artifact/upload/download steps; rewraps a long CI arg line without changing value.
CloudTrail provider, store, and tests
crates/services/cloudtrail/*
New CloudTrail crate: error type, store models, provider dispatch (CreateTrail/Delete/Describe/Get/Status/Start/Stop/Update/LookupEvents/Tags), storage snapshot, and comprehensive tests (unit/integration/perf).
Cognito provider, store, and tests
crates/services/cognito/*
New Cognito crate: error type, store models, provider dispatch (user-pool/client/user/admin flows, AdminInitiateAuth), storage snapshot, and extensive integration/perf tests.
ECS provider, store, and tests
crates/services/ecs/*
New ECS crate: error type, store schema, provider dispatch for clusters/task-defs/services/tasks, storage snapshot, and integration/perf tests.
RDS provider, store, and tests
crates/services/rds/*
New RDS crate: error type, store models, provider dispatch for instances/snapshots/subnet/parameter groups, storage snapshot, and integration/perf tests.
ElastiCache provider, store, and tests
crates/services/elasticache/*
New ElastiCache crate: error type, store models, provider dispatch for cache clusters/replication groups/subnet groups, storage snapshot, and tests.
EC2 provider, store, and tests
crates/services/ec2/*
Expands EC2 store and dispatcher: tags, key pairs, addresses, internet gateways, volumes, instance state codes, tagging APIs, and many tests (unit/integration/perf).
Route53 provider, store, and tests
crates/services/route53/*
Adds hosted-zone helpers, ListHostedZonesByName, GetHostedZoneCount, health check CRUD, resource tagging endpoints, store fields, and tests.
OpenSearch provider, store, and tests
crates/services/opensearch/*
Adds ServiceSoftwareOptions, tagging and version endpoints, domain-status helper, storage of tags, and tests for tags, versions, and service-software update flows.
SES provider, store, and tests
crates/services/ses/*
Adds email templates, templated send, quota/statistics endpoints, identity notification attributes, store additions, and tests.
SSM provider, store, and tests
crates/services/ssm/*
Adds parameter history, document and command models, document tag operations, req-id helper, store extensions, and tests for history/documents/commands.
Redshift provider, store, and tests
crates/services/redshift/*
Adds snapshots, subnet-group, parameter-group types and dispatch, logging enable/disable toggles, store fields, and tests.
STS provider and tests
crates/services/sts/*
Adds AssumeRoleWithWebIdentity and AssumeRoleWithSAML handlers and corresponding tests.
Other perf tests
crates/services/*/tests/*
Adds many throughput/round-trip perf tests and helpers across apigateway, cloudformation, firehose, iam, kinesis, sns, stepfunctions, opensearch, rds, ec2, ssm, and others.
S3 pre-allocation tuning
crates/services/s3/src/object_store.rs
Tightens Linux pre-allocation to use fallocate only when known content_length >= 50 MiB.

Estimated code review effort

🎯 5 (Critical) | ⏱️ ~120+ minutes

Possibly related PRs

Poem

🐰 I hopped through crates and tests so bright,
I stitched new services in the night.
Trails, pools, clusters — tags in tow,
I chased down bugs and watched them go.
A carrot deploy — the CI's delight.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/batch-further-service-coverage

Align protocol selection with gateway service alias handling and recognize CloudTrail target variants used by guided flows. Add regression tests for Cognito alias routing and CloudTrail target parsing.
@JesseKoldewijn JesseKoldewijn changed the title feat(coverage): expand service coverage — 5 new services, 8 existing deepened fix(ci): align parity service aliases May 27, 2026
Apply rustfmt's import ordering in gateway tests and add a per-operation benchmark threshold for opensearch/list_domain_names to match observed CI variance. The PR title was updated separately to satisfy the conventional title gate.
@JesseKoldewijn

Copy link
Copy Markdown
Owner Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jun 11, 2026

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 57

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (4)
crates/services/s3/src/object_store.rs (1)

213-218: 🛠️ Refactor suggestion | 🟠 Major | ⚡ Quick win

Update the doc comment to reflect the Linux 50 MiB pre-allocation threshold.

The doc comment states that pre-allocation happens "when size is known", but after the change at line 610, Linux only pre-allocates for objects >= 50 MiB. The current documentation is misleading for callers trying to understand the performance characteristics of this method.

📝 Proposed documentation update
     /// Write object data from a reader (async), streaming to disk.
     ///
     /// All objects use [`write_via_copy`]: an async read loop with a 512 KiB
     /// `BufReader` feeding an adaptive `BufWriter` (2–8 MiB depending on
-    /// `content_length`).  Pre-allocates the file with `set_len` when size
-    /// is known to avoid block-level fragmentation.
+    /// `content_length`).  On Linux, pre-allocates the file with `fallocate`
+    /// for known sizes >= 50 MiB to avoid block-level fragmentation. On other
+    /// platforms, pre-allocates with `set_len` when size is known and > 0.
     ///
     /// Returns `(final_path, bytes_written)`.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/services/s3/src/object_store.rs` around lines 213 - 218, Update the
doc comment for the async writer (the block describing write_via_copy) to note
that pre-allocation via set_len is conditional on platform and size: on Linux
the code only pre-allocates for objects >= 50 MiB; otherwise pre-allocation is
skipped. Edit the comment that currently says "when size is known" to explicitly
mention the 50 MiB Linux threshold and that behavior may differ on other
platforms.
crates/services/opensearch/src/store.rs (1)

19-51: ⚠️ Potential issue | 🟠 Major

Add parity assertion coverage for ServiceSoftwareOptions

  • ServiceSoftwareOptions is already exercised in crates/services/opensearch/src/provider.rs and covered by unit tests (UpdateStatus/Cancellable assertions in crates/services/opensearch/tests/opensearch_tests.rs).
  • The parity scenario currently hit (tests/parity/scenarios/all-services-smoke.json) only runs opensearch list-domain-names and has no assertions/captured JSON validating ServiceSoftwareOptions.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/services/opensearch/src/store.rs` around lines 19 - 51, Add parity
assertion coverage for ServiceSoftwareOptions by ensuring the parity test
captures and asserts its fields: update the parity scenario or test harness that
produces the JSON for opensearch domains so that service_software_options
(fields current_version, new_version, update_available, cancellable,
update_status, description, automated_update_date) are emitted and validated;
modify the parity scenario (tests/parity/scenarios/all-services-smoke.json) or
the parity capture logic in the opensearch provider code paths (see
ServiceSoftwareOptions struct and Default impl in store.rs and usages in
provider.rs) to include these fields in the recorded output, and add assertions
in the parity test suite (similar to the unit assertions in
tests/opensearch_tests.rs) that verify expected values/types for those
ServiceSoftwareOptions properties.

Source: Coding guidelines

crates/services/rds/tests/rds_perf_tests.rs (1)

1-131: ⚠️ Potential issue | 🟠 Major

Add missing perf coverage for RDS provider operations
crates/services/rds/src/provider.rs dispatches 15 operations, but crates/services/rds/tests/rds_perf_tests.rs only has perf tests for CreateDBInstance, DescribeDBInstances, and CreateDBSnapshot. Add perf test entries (with timing assertions) for the remaining operations: CreateDBParameterGroup, CreateDBSubnetGroup, DeleteDBInstance, DeleteDBParameterGroup, DeleteDBSnapshot, DeleteDBSubnetGroup, DescribeDBParameterGroups, DescribeDBSnapshots, DescribeDBSubnetGroups, ModifyDBInstance, RebootDBInstance, RestoreDBInstanceFromDBSnapshot.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/services/rds/tests/rds_perf_tests.rs` around lines 1 - 131, Tests are
missing perf coverage for the other RDS operations; add new tokio::test
functions that mirror the existing patterns (use RdsProvider::new(), make_ctx,
body_str) to exercise the remaining operations and assert status_code == 200 and
timing bounds. Specifically add tests named perf_create_db_parameter_group,
perf_create_db_subnet_group, perf_delete_db_instance,
perf_delete_db_parameter_group, perf_delete_db_snapshot,
perf_delete_db_subnet_group, perf_describe_db_parameter_groups,
perf_describe_db_snapshots, perf_describe_db_subnet_groups,
perf_modify_db_instance, perf_reboot_db_instance, and
perf_restore_db_instance_from_snapshot; for mutating ops use n=100 loop with
unique identifiers and assert elapsed.as_millis() < 2000, for describe ops
perform the query after creating sample resources and assert elapsed.as_millis()
< 500, reusing make_ctx and body_str helpers and creating any prerequisite
resources (e.g., DB instances/snapshots/subnet/parameter groups) before the
measured loop.

Source: Coding guidelines

crates/services/ec2/src/store.rs (1)

73-159: ⚠️ Potential issue | 🟠 Major

EC2 store: add parity coverage for new KeyPair/Address/InternetGateway/Volume resources

  • Unit tests in crates/services/ec2/tests/ec2_tests.rs cover KeyPair, Elastic IP (Allocate/Associate/Release/Describe), InternetGateway (create/attach/detach/delete/describe), and Volume (create/attach/delete/describe).
  • No parity scenarios under tests/parity/scenarios/ match CreateKeyPair, AllocateAddress, CreateInternetGateway, or CreateVolume, so the “at least one parity scenario” requirement for these store changes is not met.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/services/ec2/src/store.rs` around lines 73 - 159, The new store types
KeyPair, Address, InternetGateway, and Volume were added (in structs KeyPair,
Address, InternetGateway, Volume and aggregated in Ec2Store) but there are no
parity scenarios exercising CreateKeyPair, AllocateAddress,
CreateInternetGateway, or CreateVolume under tests/parity/scenarios; add at
least one parity scenario file for each resource mapping to the corresponding
API flows (CreateKeyPair, AllocateAddress, CreateInternetGateway, CreateVolume)
so the parity test suite covers those new store changes and references the same
resource IDs/fields used by the unit tests in
crates/services/ec2/tests/ec2_tests.rs.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/benchmark.yml:
- Line 211: The workflow uses mutable action tags like actions/checkout@v4 (and
other occurrences flagged) which should be pinned to specific commit SHAs;
update each unpinned uses: replace the tag (e.g., actions/checkout@v4) with the
corresponding commit SHA for that action, keeping an inline comment with the
human-friendly tag (e.g., "# v4") for clarity, and ensure you pin every other
unpinned action occurrence flagged in the file so all actions are referenced by
exact commit SHAs.
- Around line 262-276: The checkout step using actions/checkout@v4 currently
lacks persist-credentials: false; update the checkout step (the job step that
uses actions/checkout@v4 and sets ref: ${{ needs.check-trigger.outputs.pr_sha ||
github.sha }}) to include persist-credentials: false so credentials are not
persisted to the workspace, matching the change applied at lines 210-213.
- Around line 210-213: Update the checkout step that uses actions/checkout@v4 to
explicitly set persist-credentials: false so Git credentials are not left in the
workspace; locate the checkout block (the step using actions/checkout@v4 and the
ref: ${{ needs.check-trigger.outputs.pr_sha || github.sha }}) and add the
persist-credentials: false input to that step.

In `@crates/aws-protocol/src/protocol.rs`:
- Line 21: The canonical_service_name function is case-sensitive and only
special-cases "es" and "cognito*" so mixed-case inputs like "SQS" fall through
and get misclassified as RestJson; fix it by normalizing the incoming service
identifier to lowercase at the start of canonical_service_name (e.g., let s =
service.to_ascii_lowercase()) and then perform the existing matches against that
normalized string (handle starts_with checks for "cognito" and exact matches for
services such as "es", "sqs", "s3", etc.), ensuring the match branches that
currently return RestJson only apply to true unknowns after the lowercase
comparison.

In `@crates/config/src/services.rs`:
- Around line 53-55: The provider override suffix is being canonicalized to use
underscores (via canonical_service_name) but lookups expect hyphenated service
names (e.g., "cognito-idp"), so change the mapping to normalize to the
hyphenated form: after calling canonical_service_name(service) convert any
underscores to hyphens (e.g., .map(|service|
(canonical_service_name(service).replace('_', "-"), value))). This ensures keys
produced by key.strip_prefix("PROVIDER_OVERRIDE_") match lookup names used
elsewhere.

In `@crates/services/cloudtrail/src/provider.rs`:
- Around line 38-60: The json_error function in provider.rs bypasses the service
error module; replace the inline JSON error construction by using a
CloudTrail-specific error type in the error module and its helper constructor
(mirror the CognitoError/ElastiCacheError pattern). Add CloudTrail error
variants and a helper (e.g., CloudTrailError::new or
CloudTrailError::to_dispatch_response) to
crates/services/cloudtrail/src/error.rs, then update provider.rs to call that
helper from places that currently call json_error (and keep json_ok for success
responses). Ensure the resulting return type is DispatchResponse and that
content_type/headers are produced by the new error helper.
- Around line 187-188: Extract the repeated ARN-to-key parsing into a small
helper, e.g., fn extract_key_from_name(name: &str) -> String, placed near the
top of the module (next to other helper functions). Replace each occurrence of
name.rsplit('/').next().unwrap_or(&name).to_string() in provider.rs (the places
where a local variable key is being computed) with a call to
extract_key_from_name(name). Ensure the helper handles both ARN and non-ARN
inputs and returns the same String result so callers like the key assignments in
the provider logic remain unchanged.
- Line 12: Remove the unused uuid import and the suppression helper: delete the
`use uuid::Uuid;` import and the `_use_uuid()` function (the dummy function that
references Uuid at the bottom of the file) so there is no unused-import
workaround; ensure no other references to `Uuid` remain in provider.rs before
committing.
- Around line 101-584: Add explicit parity scenarios for CloudTrail covering
each operation handled in provider.rs—CreateTrail, DeleteTrail, DescribeTrails,
GetTrail, GetTrailStatus, StartLogging, StopLogging, UpdateTrail, LookupEvents,
AddTags, ListTags, and RemoveTags—into the parity scenario sets (core.json and
extended.json). For each scenario, invoke the corresponding CLI/API action that
maps to the operation string (e.g., cloudtrail create-trail → "CreateTrail",
cloudtrail delete-trail → "DeleteTrail", cloudtrail describe-trails →
"DescribeTrails", etc.), include any necessary setup/teardown steps (create a
trail before testing GetTrail/StartLogging/UpdateTrail/DeleteTrail and add tags
before ListTags/RemoveTags), assert the expected responses (presence/absence of
trail, logging_enabled toggles, tags present), and ensure the scenarios exercise
the code paths used by trail_to_json, trail_arn creation,
StartLogging/StopLogging mutating logging_enabled, LookupEvents returning
Events, and the AddTags/ListTags/RemoveTags tag map manipulation so parity tests
cover all twelve operations.

In `@crates/services/cloudtrail/src/store.rs`:
- Around line 55-56: The events Vec on the store (pub events:
Vec<CloudTrailEvent>) is unbounded despite the comment "last N events"; update
the provider code paths that append to store.events (the method(s) that
create/persist events used by LookupEvents) to enforce a fixed capacity eviction
policy: add a MAX_EVENTS constant and after pushing a new CloudTrailEvent, trim
the oldest entries so store.events.len() <= MAX_EVENTS (e.g., drain the front
excess). Ensure the constant is documented and used consistently where events
are mutated so LookupEvents reads a bounded collection.

In `@crates/services/cloudtrail/tests/cloudtrail_perf_tests.rs`:
- Around line 1-91: Add performance tests for the missing CloudTrail operations
by modeling new async #[tokio::test] functions after the existing
perf_create_trail_throughput and perf_describe_trails_many tests: create tests
for DeleteTrail, GetTrail, GetTrailStatus, StartLogging, StopLogging,
UpdateTrail, LookupEvents, AddTags, ListTags, and RemoveTags that instantiate
CloudTrailProvider::new(), use make_ctx(...) to build appropriate RequestContext
bodies, call p.dispatch(...).await.unwrap(), assert resp.status_code == 200 (or
expected code) and check relevant response bodies with body_json(&resp) where
applicable, and include throughput assertions similar to the existing
elapsed.as_millis() checks to satisfy perf thresholds.

In `@crates/services/cloudtrail/tests/cloudtrail_tests.rs`:
- Around line 1-363: Add tests that cover the missing error paths by using
CloudTrailProvider::new() and dispatching the relevant operations via make_ctx
with non-existent or invalid resource identifiers and asserting the proper error
status and __type: specifically add a GetTrailStatus not-found test (e.g.,
test_get_trail_status_not_found) that dispatches "GetTrailStatus" for a missing
trail and expects 404 + TrailNotFoundException, StartLogging and StopLogging
not-found tests (test_start_logging_not_found, test_stop_logging_not_found) that
dispatch "StartLogging"/"StopLogging" for a missing trail and expect error
status + TrailNotFoundException, an UpdateTrail not-found test
(test_update_trail_not_found) that dispatches "UpdateTrail" for a missing trail
and expects error + TrailNotFoundException, an AddTags invalid-resource test
(test_add_tags_invalid_resource) that calls "AddTags" with a bogus ResourceId
and asserts an error type for invalid resource, and tests for RemoveTags (both
happy and error path, e.g., test_remove_tags and test_remove_tags_not_found)
that dispatch "RemoveTags" for an existing trail and for a non-existent resource
respectively, asserting correct success and error responses; use make_ctx to
build request bodies and body_json to validate __type and status_code.

In `@crates/services/cognito/src/provider.rs`:
- Around line 133-893: The parity and perf coverage for Cognito is incomplete:
add parity scenarios for all missing operations (CreateUserPool, DeleteUserPool,
DescribeUserPool, ListUserPools, UpdateUserPool, CreateUserPoolClient,
DeleteUserPoolClient, DescribeUserPoolClient, ListUserPoolClients,
AdminCreateUser, AdminDeleteUser, AdminGetUser, ListUsers, AdminSetUserPassword,
AdminEnableUser, AdminDisableUser, AdminUpdateUserAttributes, AdminInitiateAuth)
into tests/parity/scenarios/core.json and tests/parity/scenarios/extended.json
so the test runner exercises the corresponding operation strings handled in
provider.rs (e.g., "CreateUserPool", "AdminCreateUser", "AdminInitiateAuth",
etc.), and extend crates/services/cognito/tests/cognito_perf_tests.rs to add
timing/throughput assertions for each of those operations (mirroring existing
tests for CreateUserPool, AdminCreateUser, ListUsers) by issuing the same client
calls the provider implements and asserting expected latency/throughput
thresholds. Ensure scenario entries use the correct request shape/parameters to
hit the branches in provider.rs and that perf tests cover success and relevant
error states where applicable.
- Around line 133-893: Add the missing error-path unit tests, parity scenarios,
and perf coverage for the Cognito operations: create unit tests in
crates/services/cognito/tests/cognito_tests.rs that exercise the failure
branches for the match arms "CreateUserPool", "DeleteUserPool", "ListUserPools",
"UpdateUserPool", "CreateUserPoolClient", "DeleteUserPoolClient",
"DescribeUserPoolClient", "ListUserPoolClients", "AdminDeleteUser",
"AdminSetUserPassword", "AdminEnableUser", "AdminDisableUser", and
"AdminUpdateUserAttributes" (e.g., missing params -> InvalidParameterException,
non-existent pool/client/user ->
ResourceNotFoundException/UsernameExistsException as appropriate) by calling the
provider dispatch logic in provider.rs; add parity scenarios for these
operations into tests/parity/scenarios/core.json and
tests/parity/scenarios/extended.json describing request input and expected error
responses so parity runner covers them; and extend
crates/services/cognito/tests/cognito_perf_tests.rs to add perf measurements
(using std::time::Instant like existing tests) for the omitted operations
(include CreateUserPoolClient, DeleteUserPoolClient, AdminDeleteUser,
AdminSetUserPassword, AdminEnableUser/AdminDisableUser,
AdminUpdateUserAttributes, ListUserPoolClients) to ensure regression coverage —
keep tests deterministic and reuse existing helpers to create
pools/clients/users where needed.

In `@crates/services/cognito/src/store.rs`:
- Around line 38-39: Rename the misspelled struct fields callback_ur_ls and
logout_ur_ls to callback_urls and logout_urls respectively; update all
references (constructors, pattern matches, field access, tests, Serde
attributes) to use the new names, and if needed add #[serde(rename =
"callback_ur_ls")] / #[serde(rename = "logout_ur_ls")] temporarily to preserve
backward compatibility during de-serialization until callers are updated.

In `@crates/services/ec2/src/provider.rs`:
- Around line 270-287: Add the missing unit and perf tests and expand parity
scenarios: add unit tests in crates/services/ec2/tests/ec2_tests.rs that
exercise AuthorizeSecurityGroupEgress and DeleteTags covering both happy paths
and error paths (e.g., validate parameter errors and expected xml_error
responses), add perf benchmarks in crates/services/ec2/tests/ec2_perf_tests.rs
using std::time::Instant for DeleteSubnet, DeleteSecurityGroup, StopInstances,
StartInstances, etc., measuring and asserting elapsed time thresholds, and
update tests/parity/scenarios/ to include dedicated scenario files that exercise
each modified operation against both OpenStack and LocalStack (ensuring handlers
like the DeleteSubnet branch that uses store.subnets.remove and returns
xml_ok/xml_error are actually hit).

In `@crates/services/ec2/tests/ec2_tests.rs`:
- Around line 550-569: Add a failing-path test that attempts to ReleaseAddress
with a non-existent AllocationId to exercise the error path: extend or add a
test near test_release_address that calls p.dispatch(&make_ctx("ReleaseAddress",
rel)) where rel contains a made-up AllocationId (not returned by
AllocateAddress), assert the response is an error (non-200) and/or contains the
expected error message/code, and verify DescribeAddresses still does not contain
that fake allocation; reference test_release_address, Ec2Provider::dispatch,
make_ctx, "ReleaseAddress", and "DescribeAddresses" to locate and implement the
new test.
- Around line 397-424: Extend the existing test_stop_and_start_instances test
suite by adding separate test cases (or additional assertions) that call
p.dispatch(&make_ctx("StopInstances", params)) and
p.dispatch(&make_ctx("StartInstances", params)) with a non-existent InstanceId
(e.g., "i-doesnotexist") to exercise the error paths; verify the provider
returns the appropriate error HTTP status and error body (assert status_code !=
200 and that body_str contains the expected error code/message for
InstanceNotFound or similar). Use the same Ec2Provider::new, dispatch, make_ctx
helpers and mirror how run/start/stop responses are inspected (body_str,
status_code) so the tests fail if the provider doesn't return the expected error
behavior.
- Around line 366-391: Add a new test that exercises the error path for
DeleteSecurityGroup by attempting to delete a non-existent GroupId: create a new
async test (e.g., test_delete_security_group_not_found) that instantiates
Ec2Provider::new(), calls p.dispatch(&make_ctx("DeleteSecurityGroup", params))
with a bogus GroupId (not created earlier) and then asserts the provider returns
the expected error outcome (e.g., response status_code != 200 and body_str
contains the AWS-style error like "InvalidGroup.NotFound" or that dispatch
returned an Err). Reuse make_ctx, body_str, and Ec2Provider::dispatch as in
test_delete_security_group and keep assertions specific to DeleteSecurityGroup's
error signal.
- Around line 718-737: Add a new test function (e.g.,
test_delete_volume_not_found) alongside test_delete_volume that constructs an
Ec2Provider via Ec2Provider::new(), calls p.dispatch(&make_ctx("DeleteVolume",
del)) with a HashMap where "VolumeId" is a non-existent id (e.g.,
"vol-00000000"), awaits the result, and asserts the error path: check
resp.status_code is not 200 and that body_str(&resp) contains the expected error
identifier (e.g., "InvalidVolume.NotFound" or other service-specific not-found
string); also keep the existing DescribeVolumes check separate — reference
functions/methods: test_delete_volume, Ec2Provider::new, p.dispatch, make_ctx,
"DeleteVolume", and body_str.
- Around line 807-813: The extract_tag function currently uses unwrap_or(0)
which hides missing tags and produces incorrect slice bounds; update extract_tag
to explicitly handle absent tags by checking xml.find(&open) and
xml.find(&close) for None and returning a clear failure (preferably
Option<String> returning None for missing tags or a Result with an error)
instead of computing invalid start/end; compute start = pos_open + open.len()
and end = pos_close, ensure end >= start, then return
Some(xml[start..end].to_string()) (or Err) so callers can fail fast; reference
the function name extract_tag and the variables open, close, start, end when
making the change.

In `@crates/services/ecs/src/provider.rs`:
- Around line 165-885: The test suite is missing coverage for many ECS
operations (notably DescribeServices), lacks error-path assertions per
operation, misses perf timing checks for several operations, and lacks
per-operation parity scenarios; update tests to add unit/integration tests in
crates/services/ecs/tests/ecs_tests.rs to include a happy-path and at least one
error-path assertion (expecting 400) for each operation string (e.g.,
"DescribeServices", "CreateService", "UpdateService", "DeleteService",
"RunTask", "StopTask", "DescribeTasks", "ListTasks", etc.), add
timing/regression assertions in crates/services/ecs/tests/ecs_perf_tests.rs for
the operations currently missing perf checks (add Instant-based assertion blocks
similar to existing CreateCluster/ListClusters/RegisterTaskDefinition/RunTask
checks), and add parity scenario JSONs under tests/parity/scenarios/* (core.json
or extended.json) to exercise each operation against both OpenStack and
LocalStack so parity runs cover per-operation flows; ensure tests exercise the
provider code paths that handle operation strings in provider.rs (match arm
names like "DescribeServices", "ListServices", "CreateService", "DeleteService",
"UpdateService", "RunTask", "StopTask", "DescribeTasks", "ListTasks",
"RegisterTaskDefinition", "DeregisterTaskDefinition", "DescribeTaskDefinition",
"ListTaskDefinitions") and include assertions for expected responses and status
codes.
- Around line 1-13: The file declares a local json_error helper and doesn't
import the crate's centralized error helpers from the service error module;
replace usage of the inline json_error with the project's standard error
constructors by adding a use crate::error (or the appropriate error module name
exported in lib.rs) and update all places that call json_error to return the
service's error wrapper functions (e.g., the error module's functions/structs
for formatting DispatchError/ResponseBody) so error responses follow the
established pattern; locate references to json_error and any direct
DispatchError/ResponseBody constructions in provider.rs and swap them to the
error module helpers.

In `@crates/services/ecs/src/store.rs`:
- Around line 88-100: EcsStore's five HashMaps (clusters, task_definitions,
task_def_revisions, services, tasks) are unbounded and need eviction; add a
bounded/TTL cleanup strategy by introducing constants (e.g. MAX_CLUSTERS,
MAX_SERVICES, MAX_TASKS or TTL_MS) and implement a method on EcsStore like
prune_evicted(&mut self) or ensure_capacity_on_insert(&mut self, key: &str) that
is called from all insertion/update paths (where
Cluster/Service/Task/TaskDefinition are created) to evict oldest/stale entries
(LRU, FIFO, or age-based) or trim to the max size; also consider replacing
HashMap with an LRU cache type for specific maps (or maintain an auxiliary
VecDeque/timestamps map to track insertion times) and update
serialize/deserialize logic to preserve behavior.

In `@crates/services/elasticache/src/provider.rs`:
- Line 304: The Delete/Modify/Reboot handlers (DeleteCacheCluster,
ModifyCacheCluster, RebootCacheCluster, DeleteReplicationGroup,
ModifyReplicationGroup, DeleteCacheSubnetGroup) currently call
self.store.get_or_create(account_id, region) which creates and leaks empty
ElastiCacheStore entries when the targeted resource is missing; change those
calls to self.store.get(account_id, region) and explicitly handle the None case
by returning NotFound without creating or persisting a store entry. Update the
code paths inside the referenced functions (e.g., the blocks around the
get_or_create call in DeleteCacheCluster, ModifyCacheCluster,
RebootCacheCluster, DeleteReplicationGroup, ModifyReplicationGroup,
DeleteCacheSubnetGroup) to check for None, perform no store mutation or
creation, and ensure any cleanup logic is applied so empty account/region stores
are not left behind.

In `@crates/services/elasticache/tests/elasticache_tests.rs`:
- Around line 185-208: Add a negative test for ModifyCacheCluster: create a new
async test (e.g., test_modify_cache_cluster_not_found) that constructs an
ElastiCacheProvider, dispatches a ModifyCacheCluster request with CacheClusterId
set to a non-existent id (e.g., "nonexistent") and a CacheNodeType, then assert
the response status_code is 400 and that body_str(&resp) contains
"CacheClusterNotFound"; this complements the existing happy-path
test_modify_cache_cluster and verifies the provider correctly returns the
not-found error.
- Around line 210-229: Add a negative test for the RebootCacheCluster operation:
create a new async test (e.g., test_reboot_cache_cluster_not_found) that
instantiates ElastiCacheProvider, calls svc.dispatch with action
"RebootCacheCluster" and CacheClusterId set to a nonexistent id, awaits the
response, and asserts that resp.status_code == 400 and body_str(&resp) contains
"CacheClusterNotFound"; place this alongside the existing
test_reboot_cache_cluster to ensure the error path for RebootCacheCluster is
covered.
- Around line 402-426: Add a test that exercises the error path for
DeleteCacheSubnetGroup by dispatching a DeleteCacheSubnetGroup request for a
non-existent group name (e.g., "nonexistent-sg") using the same
ElastiCacheProvider::dispatch and make_ctx helpers; assert the response
indicates failure (error status code and/or expected error code/message in
body_str) rather than 200, and keep it alongside test_delete_cache_subnet_group
to ensure the negative-case behavior of DeleteCacheSubnetGroup is covered.
- Around line 335-365: Add a negative test that covers the primary error path
for ModifyReplicationGroup by dispatching a ModifyReplicationGroup request
against a non-existent ReplicationGroupId and asserting the service returns the
expected error response (e.g., non-200 status, proper error code/message like
"ReplicationGroupNotFound" or similar). In the test file add a tokio::test
(similar to test_modify_replication_group) that creates
ElastiCacheProvider::new(), calls
svc.dispatch(&make_ctx("ModifyReplicationGroup",
[p("ReplicationGroupId","no-such"),
p("ReplicationGroupDescription","x")].into())).await.unwrap() (or handle the
Result) and then assert on resp.status_code and the body_str(resp) contains the
expected error element/string; use the same helpers (make_ctx, p, body_str) so
the test mirrors the existing pattern.
- Around line 235-265: The test test_create_and_describe_replication_group only
covers the success path; add an error-path test that attempts to create a
duplicate replication group (call ElastiCacheProvider::new() and dispatch
CreateReplicationGroup twice using make_ctx with the same "ReplicationGroupId"
like "my-rg"), then assert the second response is an error (non-200 status_code)
and its body contains the expected duplicate error text (mirror the duplicate
cache cluster test pattern: check error code/message for an already-existing
replication group). Ensure you use the same helper functions (make_ctx, p,
body_str, and ElastiCacheProvider::dispatch) so the new assertions align with
existing tests.
- Around line 371-400: Add an error-path test that attempts to create the same
CacheSubnetGroup twice using ElastiCacheProvider::dispatch with the
CreateCacheSubnetGroup parameters (reuse test_create_and_describe_subnet_group
setup), then assert the second response is an error: verify non-200 status or
that the body contains the ElastiCache error identifier (e.g.,
"CacheSubnetGroupAlreadyExistsFault" or an appropriate error message) and a
descriptive message; ensure the test name reflects duplicate creation (e.g.,
test_create_subnet_group_duplicate_error) and uses the same request shape
(CacheSubnetGroupName, SubnetIds.*) so it exercises the duplicate-path in
CreateCacheSubnetGroup handling.

In `@crates/services/opensearch/src/provider.rs`:
- Around line 270-293: Add missing perf, parity, and error-path tests for the
new OpenSearch operations: add perf_* benchmarks in
crates/services/opensearch/tests/opensearch_perf_tests.rs for DescribeDomains,
AddTags, RemoveTags, ListTags, GetCompatibleVersions, ListVersions,
StartServiceSoftwareUpdate, and CancelServiceSoftwareUpdate; add corresponding
service:"opensearch" entries in tests/parity/scenarios/core.json and the
extended.json parity scenarios to cover DescribeDomains, GetCompatibleVersions,
ListVersions, StartServiceSoftwareUpdate, CancelServiceSoftwareUpdate and the
tag operations (AddTagsToStream, ListTagsForStream, RemoveTagsFromStream) using
the non-stream API names; and add unit tests in
crates/services/opensearch/tests/opensearch_tests.rs that exercise error paths
(e.g., DescribeDomains when store missing or domain not found, returning
appropriate empty list or 404/ResourceNotFoundException depending on API
contract; AddTags/RemoveTags/ListTags when domain missing -> 404
ResourceNotFoundException; GetCompatibleVersions/ListVersions when target
domain/state invalid -> test expected error responses; and a
CancelServiceSoftwareUpdate error test similar to
test_start_service_software_update_domain_not_found) — reference existing test
helper patterns and functions (store.get, domain_status_json, json_ok, and
test_start_service_software_update_domain_not_found) to implement these tests
consistently.

In `@crates/services/opensearch/src/store.rs`:
- Around line 61-62: DeleteDomain currently removes the domain from
store.domains but leaves stale entries in OpenSearchStore.tags; update the
DeleteDomain handler (the function/method that removes domains) to also remove
the tags entry for that domain (e.g., call store.tags.remove(&d.arn) or
equivalent using the domain ARN) so tags are cleaned up on deletion, and add a
unit test that creates a domain, calls AddTags, deletes the domain via
DeleteDomain, then asserts ListTags returns an empty TagList for that domain to
prevent regressions.

In `@crates/services/opensearch/tests/opensearch_tests.rs`:
- Around line 490-632: Add tests that exercise the error paths for tag ops:
create new async tests (e.g., test_add_tags_error_invalid_arn and
test_remove_tags_error_nonexistent_domain) that use OpenSearchProvider::new(),
call p.dispatch(&make_ctx(...)) with the "AddTags" and "RemoveTags" actions and
invalid inputs (bad ARN format for AddTags and an ARN that doesn't correspond to
any created domain for RemoveTags), then assert non-200 status_code and that
body_json(&resp) contains the expected error structure/code/message; reuse
make_ctx, dispatch, and body_json helpers and mirror the existing happy-path
tests (test_add_and_list_tags, test_remove_tags) for locating where to add them.

In `@crates/services/rds/src/provider.rs`:
- Around line 428-446: The code currently uses unwrap_or_else on
store.instances.get(&db_id) to supply hardcoded defaults for (engine,
engine_version, allocated_storage, master_username) which masks a missing DB;
change this so snapshot creation fails when the instance is not found: replace
the unwrap_or_else branch with an explicit match on store.instances.get(&db_id)
and return an Err(...) (or propagate a suitable error) if None, including the
db_id in the message, otherwise extract the tuple from the found db; update the
function's return path to propagate that error instead of proceeding with
defaults.
- Around line 208-766: Add comprehensive parity, unit, and perf coverage for all
RDS operations: extend the parity scenarios to include rds actions (at minimum
CreateDBInstance, DeleteDBInstance, DescribeDBInstances, ModifyDBInstance,
RebootDBInstance, CreateDBSnapshot, DeleteDBSnapshot, DescribeDBSnapshots,
RestoreDBInstanceFromDBSnapshot, CreateDBSubnetGroup, DeleteDBSubnetGroup,
DescribeDBSubnetGroups, CreateDBParameterGroup, DeleteDBParameterGroup,
DescribeDBParameterGroups); update the unit test module rds_tests.rs to add
happy-path tests and at least one error-path test per operation (use the
existing patterns for DBInstanceAlreadyExists/DBInstanceNotFound to assert
xml_error responses and instance_xml/snapshot_xml outputs for successful flows,
exercising functions that handle these ops in provider.rs); and expand the perf
test module rds_perf_tests.rs to add Instant perf/regression tests for the
missing operations (CreateDBInstance, DescribeDBInstances, CreateDBSnapshot
already present—add tests for Delete*, ModifyDBInstance, RebootDBInstance,
RestoreDBInstanceFromDBSnapshot, subnet/parameter group create/delete/describe)
using the same test harness and request sequences as existing perf tests.

In `@crates/services/rds/tests/rds_tests.rs`:
- Around line 323-356: Add a negative test case to cover the error path for
CreateDBSubnetGroup by duplicating the call inside the existing
test_create_and_describe_subnet_group flow (or as a new #[tokio::test]) that
calls RdsProvider::dispatch with the same "CreateDBSubnetGroup" parameters a
second time and asserts a non-200 response or that the error body contains the
expected duplicate-name error; locate the existing
test_create_and_describe_subnet_group, reuse the same params map used for the
first CreateDBSubnetGroup call, invoke
p.dispatch(&make_ctx("CreateDBSubnetGroup", params)) again, await unwrap_err or
check resp.status_code != 200 and assert the returned error message matches the
duplicate-subnet-group condition.
- Around line 411-435: Add a new test case that exercises the error path for
DeleteDBParameterGroup by calling RdsProvider::dispatch via make_ctx with a
DBParameterGroupName that was never created (e.g., "pg-missing") and assert the
provider returns the expected error response (non-200 status_code and/or error
body indicating the parameter group does not exist). Implement this alongside
the existing test_delete_parameter_group (or as a separate async #[tokio::test])
and use DescribeDBParameterGroups or the response body to verify the correct
error handling for DeleteDBParameterGroup.
- Around line 388-409: Add an error-path test to
test_create_and_describe_parameter_group that attempts to create the same
DBParameterGroup twice: use RdsProvider::new(), call dispatch with
make_ctx("CreateDBParameterGroup", params) to create "my-pg" as before, then
call dispatch again with the same params and assert the response indicates
failure (non-200 status_code) and the body contains a duplicate-group error
message (e.g., mentions "DBParameterGroupAlreadyExists" or similar); ensure you
reference the same parameter map and use the existing helper functions
(dispatch, make_ctx, body_str) to validate the error path.
- Around line 182-202: Add a negative-path test for RebootDBInstance: create a
new async test (e.g., test_reboot_db_instance_not_found) that calls
RdsProvider::dispatch with make_ctx("RebootDBInstance", params) where
"DBInstanceIdentifier" is a non-existent id (do not create it first), await the
response and assert the status_code indicates an error (e.g., 404) and the
response body contains the expected error structure/message (e.g.,
"DBInstanceNotFound" or "ErrorResponse"). Use the same helpers (make_ctx,
body_str) and ensure you cover the primary error path for RebootDBInstance.
- Around line 221-248: Extend test_create_and_describe_snapshot to cover
CreateDBSnapshot error paths: add a test case that calls
p.dispatch(make_ctx("CreateDBSnapshot", params)) twice with the same
DBSnapshotIdentifier ("snap-001") and assert the second response returns an
error status (non-200) and contains an appropriate duplicate-snapshot error
message; also add a test where you call CreateDBSnapshot with a
DBSnapshotIdentifier but with DBInstanceIdentifier set to a non-existent
instance (e.g., "no-such-db") and assert the response is an error with the
expected not-found message; use the same RdsProvider instance, make_ctx helper
and body_str() to inspect responses and assert status_code and body contents.
- Around line 277-317: Add a new async test (e.g.,
test_restore_db_instance_from_nonexistent_snapshot) alongside
test_restore_db_instance_from_snapshot that calls RdsProvider::dispatch with
action "RestoreDBInstanceFromDBSnapshot" using a valid DBInstanceIdentifier but
a non-existent DBSnapshotIdentifier (e.g., "no-such-snap"); assert the provider
returns the error path (non-200 status or the specific error XML/message your
provider uses) and that the body contains the expected error code/text about the
missing snapshot. Use the same make_ctx helper and HashMap param construction
pattern as in test_restore_db_instance_from_snapshot to locate where to add the
test and to build the request for RestoreDBInstanceFromDBSnapshot.
- Around line 250-275: Add a test covering the error path for DeleteDBSnapshot
by creating a new async test (similar to test_delete_snapshot) that uses
RdsProvider::new() and calls dispatch with a DeleteDBSnapshot request for a
non-existent DBSnapshotIdentifier (e.g., "no-such-snap"); assert that the
response indicates failure (non-200 or contains the expected error/exception
information) and verify via DescribeDBSnapshots that the snapshot is not
present; place the test alongside test_delete_snapshot and reference the same
request name strings "DeleteDBSnapshot" and "DescribeDBSnapshots" when
constructing the contexts.
- Around line 156-180: Add a test that covers the primary error path for
ModifyDBInstance by asserting the provider returns the expected error when the
target DB instance does not exist: create a new async test (similar to
test_modify_db_instance) that calls RdsProvider::dispatch with
make_ctx("ModifyDBInstance", params) where "DBInstanceIdentifier" is a
non-existent name, then assert the response status_code and body contain the
not-found error (e.g., ResourceNotFound or equivalent message); reference
test_modify_db_instance, RdsProvider::dispatch, make_ctx, and the
"ModifyDBInstance" operation when implementing.
- Around line 358-382: Add a negative test covering the error path for
DeleteDBSubnetGroup by creating a new async test (e.g.,
test_delete_subnet_group_not_found) that uses RdsProvider::new() and calls
p.dispatch(&make_ctx("DeleteDBSubnetGroup", params)) where params contains a
non-existent DBSubnetGroupName; assert the response represents the expected
error (non-200 status and/or error body indicating not found) instead of a
successful 200, similar to how test_delete_subnet_group asserts success, so the
DeleteDBSubnetGroup error path is covered.

In `@crates/services/redshift/src/provider.rs`:
- Around line 364-424: The repo is missing performance regression tests for the
11 new Redshift operations; add timing-based tests to
crates/services/redshift/tests/redshift_perf_tests.rs that exercise each API
operation (CreateClusterSnapshot, DeleteClusterSnapshot,
DescribeClusterSnapshots, CreateClusterSubnetGroup, DeleteClusterSubnetGroup,
DescribeClusterSubnetGroups, CreateClusterParameterGroup,
DeleteClusterParameterGroup, DescribeClusterParameterGroups, EnableLogging,
DisableLogging) by invoking the provider endpoints (the handler that routes on
the operation name in provider.rs) and asserting latency using
std::time::Instant (measure start/elapsed and fail if above an acceptable
threshold), create one test per operation, reuse existing test helpers for
request construction and account/region setup, and name tests clearly (e.g.,
perf_create_cluster_snapshot) so they run with cargo test.
- Around line 364-719: The parity test suite lacks scenarios exercising the
newly implemented Redshift actions (CreateClusterSnapshot,
DeleteClusterSnapshot, DescribeClusterSnapshots, CreateClusterSubnetGroup,
DeleteClusterSubnetGroup, DescribeClusterSubnetGroups,
CreateClusterParameterGroup, DeleteClusterParameterGroup,
DescribeClusterParameterGroups, EnableLogging, DisableLogging) in provider.rs;
add entries to tests/parity/scenarios/core.json (and
extended.json/all-services-smoke.json as appropriate) that call these API
actions with representative parameters (e.g.,
SnapshotIdentifier/ClusterIdentifier, ClusterSubnetGroupName/SubnetIds,
ParameterGroupName/ParameterGroupFamily, ClusterIdentifier for logging) so
parity tests exercise the match arms handling those operations (match arms
labeled by the action strings in provider.rs).

In `@crates/services/redshift/tests/redshift_tests.rs`:
- Around line 577-605: Add two unit tests that cover the ClusterNotFound error
path for EnableLogging and DisableLogging: create tests named like
test_enable_logging_not_found and test_disable_logging_not_found that
instantiate RedshiftProvider, call dispatch with make_ctx("EnableLogging",
params) and make_ctx("DisableLogging", params) where params contains a
non-existent ClusterIdentifier (e.g., "no-cluster"), await the response, and
assert the response is an error with the ClusterNotFound payload similar to
test_reboot_cluster_not_found (use body_str(&resp) and assert it contains
"ClusterNotFound" and the appropriate XML error structure); mirror the structure
and assertions used in test_reboot_cluster_not_found so the provider.rs error
branches for EnableLogging and DisableLogging are exercised.
- Around line 508-534: Add a new test that asserts the duplicate-create error
path for CreateClusterParameterGroup: call RedshiftProvider::dispatch with
make_ctx("CreateClusterParameterGroup", params) twice (same params as in
test_create_and_describe_parameter_group) and verify the second response returns
the ClusterParameterGroupAlreadyExists error (similar to
test_create_snapshot_duplicate_fails). Ensure the test name indicates duplicate
failure (e.g., test_create_parameter_group_duplicate_fails), awaits the dispatch
calls, and asserts the error response status/body matches the provider's
ClusterParameterGroupAlreadyExists output.
- Around line 432-465: Add a new async test (modeled after
test_create_snapshot_duplicate_fails and located near
test_create_and_describe_subnet_group) that uses RedshiftProvider::new(), calls
dispatch with make_ctx("CreateClusterSubnetGroup", params) to create "my-sg"
once, then calls dispatch a second time with the same params and asserts the
response is an error for the duplicate path: check resp.status_code is the
expected error code (e.g., 400) and that body_str(&resp) contains
"ClusterSubnetGroupAlreadyExists" (or the exact error string returned by the
provider implementation); reuse the same param keys ("ClusterSubnetGroupName",
"Description", "VpcId", "SubnetIds.SubnetIdentifier.1",
"SubnetIds.SubnetIdentifier.2") as in test_create_and_describe_subnet_group to
reproduce the duplicate creation scenario.

In `@crates/services/route53/src/provider.rs`:
- Around line 469-774: The PR is missing parity scenario entries, performance
assertions, and error-path unit tests for the eight Route53 operations; add
parity scenarios for "ListHostedZonesByName", "GetHostedZoneCount",
"CreateHealthCheck", "GetHealthCheck", "DeleteHealthCheck", "ListHealthChecks",
"ChangeTagsForResource", and "ListTagsForResource" to
tests/parity/scenarios/core.json and extended.json; add Instant-based perf
assertions in crates/services/route53/tests/route53_perf_tests.rs exercising
those handlers (same pattern used for hosted-zone/rrset control-plane ops); and
add unit tests in crates/services/route53/tests/route53_tests.rs that exercise
the error paths for each handler (e.g., missing store or missing resource)
verifying the XML error responses produced by the handler branches (look at
handlers matching "ListHostedZonesByName", "GetHostedZoneCount",
"CreateHealthCheck", "DeleteHealthCheck", "ListHealthChecks",
"ChangeTagsForResource", "ListTagsForResource" and reuse the existing test
pattern from test_get_health_check_not_found to assert xml_error outputs and
status codes).

In `@crates/services/route53/tests/route53_tests.rs`:
- Around line 522-568: Add error-path tests alongside
test_create_and_get_health_check: (1) Add a test (e.g.,
test_create_health_check_missing_fields) that uses Route53Provider::dispatch
with make_ctx("CreateHealthCheck", <invalid XML missing
CallerReference/Port/Type>, "/2013-04-01/healthcheck", "POST") and assert
resp.status_code == 400 and body contains an error element/validation message;
(2) Add a test (e.g., test_delete_nonexistent_health_check) that calls dispatch
with make_ctx("DeleteHealthCheck", "",
&format!("/2013-04-01/healthcheck/nonexistent-id"), "DELETE") and assert
resp.status_code == 404 and body contains "NoSuchHealthCheck"; follow the same
pattern as test_create_and_get_health_check for using body_str/xml_text to
inspect response contents.

In `@crates/services/ses/src/provider.rs`:
- Around line 288-578: Add missing parity, perf, and error-path tests for the
SES operations: implement scenario entries referencing GetSendQuota,
GetSendStatistics, CreateTemplate, DeleteTemplate, GetTemplate, ListTemplates,
SendTemplatedEmail, SetIdentityFeedbackForwardingEnabled, and
GetIdentityNotificationAttributes in the parity JSONs (core.json, extended.json,
all-services-smoke.json); update crates/services/ses/tests/ses_perf_tests.rs to
dispatch/measure these operations (use the existing dispatch helper used by
other perf tests rather than adding new timing primitives) so perf coverage
includes them; and add unit tests in crates/services/ses/tests/ses_tests.rs
covering the missing error paths (e.g., GetSendQuota/GetSendStatistics when no
store, DeleteTemplate/GetTemplate/ListTemplates when template missing,
SendTemplatedEmail when template absent, and GetIdentityNotificationAttributes
for unknown identities) by asserting xml_error responses from the provider
handlers (targeting the same operation names used in provider.rs like
"GetTemplate", "CreateTemplate", "SendTemplatedEmail", etc.).

In `@crates/services/ses/tests/ses_tests.rs`:
- Around line 532-576: Add negative-path tests in the same test module using
SesProvider, dispatch and make_ctx: create one test (e.g.,
test_set_identity_feedback_forwarding_enabled_unverified) that calls
SetIdentityFeedbackForwardingEnabled with an identity that wasn't
VerifyEmailIdentity-verified and assert the response is an error (non-200) and
contains the expected error element/message; create another test (e.g.,
test_get_identity_notification_attributes_nonexistent) that calls
GetIdentityNotificationAttributes with an identity that doesn't exist and assert
it returns an error response and contains the appropriate error element/message;
reuse the existing test patterns (SesProvider::new(), make_ctx, body_str) and
mirror assertions style from test_set_identity_feedback_forwarding_enabled and
test_get_identity_notification_attributes to keep consistency.

In `@crates/services/ssm/src/provider.rs`:
- Around line 325-769: The PR is missing unit, perf, and parity coverage for
several new SSM operations; add unit tests in
crates/services/ssm/tests/ssm_tests.rs that exercise the DeleteDocument error
path (DeleteDocument when document missing),
AddTagsToResource/ListTagsForResource/RemoveTagsFromResource (verify tags
added/listed/removed on Document), and ensure CreateDocument duplicate error and
GetCommandInvocation not-found remain covered; add performance benchmarks in
crates/services/ssm/tests/ssm_perf_tests.rs using std::time::Instant for
GetParameterHistory, a representative document flow (CreateDocument,
SendCommand, ListCommands, GetCommandInvocation), and tag operations
(Add/List/RemoveTagsFromResource); finally, add parity scenario entries under
tests/parity/scenarios (core/extended/all-services) to include probes for
get-parameter-history, create/delete/describe/get/list documents, send/list/get
commands, and tag operations so CI parity tests cover them. Ensure tests
reference the handler operation names exactly (GetParameterHistory,
CreateDocument, DeleteDocument, DescribeDocument, GetDocument, ListDocuments,
SendCommand, ListCommands, GetCommandInvocation, AddTagsToResource,
ListTagsForResource, RemoveTagsFromResource) so they exercise the code paths
shown.

In `@crates/services/sts/src/provider.rs`:
- Around line 151-214: The PR is missing parity scenarios and negative/perf
tests for the AssumeRoleWithWebIdentity and AssumeRoleWithSAML handlers: add
parity scenario JSON entries (core.json or extended.json) under
tests/parity/scenarios that invoke "AssumeRoleWithWebIdentity" and
"AssumeRoleWithSAML" so they run against OpenStack and LocalStack; add
error-path unit tests in crates/services/sts/tests/sts_tests.rs that exercise
failure branches for those handlers (use the repo's error helper/formatting) and
assert error XML/status; and add Instant-timed performance tests in
crates/services/sts/tests/sts_perf_tests.rs that call the handlers (e.g., using
the same request construction as in existing success tests) and assert they
complete within expected durations. Ensure test names reference
AssumeRoleWithWebIdentity and AssumeRoleWithSAML so they’re discoverable.

In `@crates/services/sts/tests/sts_tests.rs`:
- Around line 174-243: The tests add happy-path coverage for
AssumeRoleWithWebIdentity and AssumeRoleWithSAML but lack error-path tests; add
new tokio::test functions (e.g., test_assume_role_with_web_identity_error,
test_assume_role_with_web_identity_missing_token,
test_assume_role_with_saml_error) that use StsProvider::new() and
p.dispatch(&make_ctx(...)) with invalid/missing parameters (missing/invalid
RoleArn, missing/invalid WebIdentityToken or SAMLAssertion, and malformed
params) and assert the response.status_code is an error (400-range) and the body
contains the appropriate STS error code/message; reuse existing helpers body_str
and make_ctx to construct requests and assert specific error codes/messages
returned by the provider for the given failure scenarios.

---

Outside diff comments:
In `@crates/services/ec2/src/store.rs`:
- Around line 73-159: The new store types KeyPair, Address, InternetGateway, and
Volume were added (in structs KeyPair, Address, InternetGateway, Volume and
aggregated in Ec2Store) but there are no parity scenarios exercising
CreateKeyPair, AllocateAddress, CreateInternetGateway, or CreateVolume under
tests/parity/scenarios; add at least one parity scenario file for each resource
mapping to the corresponding API flows (CreateKeyPair, AllocateAddress,
CreateInternetGateway, CreateVolume) so the parity test suite covers those new
store changes and references the same resource IDs/fields used by the unit tests
in crates/services/ec2/tests/ec2_tests.rs.

In `@crates/services/opensearch/src/store.rs`:
- Around line 19-51: Add parity assertion coverage for ServiceSoftwareOptions by
ensuring the parity test captures and asserts its fields: update the parity
scenario or test harness that produces the JSON for opensearch domains so that
service_software_options (fields current_version, new_version, update_available,
cancellable, update_status, description, automated_update_date) are emitted and
validated; modify the parity scenario
(tests/parity/scenarios/all-services-smoke.json) or the parity capture logic in
the opensearch provider code paths (see ServiceSoftwareOptions struct and
Default impl in store.rs and usages in provider.rs) to include these fields in
the recorded output, and add assertions in the parity test suite (similar to the
unit assertions in tests/opensearch_tests.rs) that verify expected values/types
for those ServiceSoftwareOptions properties.

In `@crates/services/rds/tests/rds_perf_tests.rs`:
- Around line 1-131: Tests are missing perf coverage for the other RDS
operations; add new tokio::test functions that mirror the existing patterns (use
RdsProvider::new(), make_ctx, body_str) to exercise the remaining operations and
assert status_code == 200 and timing bounds. Specifically add tests named
perf_create_db_parameter_group, perf_create_db_subnet_group,
perf_delete_db_instance, perf_delete_db_parameter_group,
perf_delete_db_snapshot, perf_delete_db_subnet_group,
perf_describe_db_parameter_groups, perf_describe_db_snapshots,
perf_describe_db_subnet_groups, perf_modify_db_instance,
perf_reboot_db_instance, and perf_restore_db_instance_from_snapshot; for
mutating ops use n=100 loop with unique identifiers and assert
elapsed.as_millis() < 2000, for describe ops perform the query after creating
sample resources and assert elapsed.as_millis() < 500, reusing make_ctx and
body_str helpers and creating any prerequisite resources (e.g., DB
instances/snapshots/subnet/parameter groups) before the measured loop.

In `@crates/services/s3/src/object_store.rs`:
- Around line 213-218: Update the doc comment for the async writer (the block
describing write_via_copy) to note that pre-allocation via set_len is
conditional on platform and size: on Linux the code only pre-allocates for
objects >= 50 MiB; otherwise pre-allocation is skipped. Edit the comment that
currently says "when size is known" to explicitly mention the 50 MiB Linux
threshold and that behavior may differ on other platforms.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: fcab6e1b-996c-4d40-852c-921db2a8827f

📥 Commits

Reviewing files that changed from the base of the PR and between 02a94c6 and 382659f.

⛔ Files ignored due to path filters (23)
  • .opencode/package-lock.json is excluded by !**/package-lock.json and included by none
  • CLAUDE.md is excluded by none and included by none
  • Cargo.lock is excluded by !**/*.lock and included by Cargo.lock
  • Dockerfile is excluded by none and included by none
  • crates/openstack/Cargo.toml is excluded by none and included by none
  • crates/services/cloudtrail/CHANGELOG.md is excluded by none and included by none
  • crates/services/cloudtrail/Cargo.toml is excluded by none and included by none
  • crates/services/cognito/CHANGELOG.md is excluded by none and included by none
  • crates/services/cognito/Cargo.toml is excluded by none and included by none
  • crates/services/ecs/CHANGELOG.md is excluded by none and included by none
  • crates/services/ecs/Cargo.toml is excluded by none and included by none
  • crates/services/elasticache/CHANGELOG.md is excluded by none and included by none
  • crates/services/elasticache/Cargo.toml is excluded by none and included by none
  • crates/services/rds/CHANGELOG.md is excluded by none and included by none
  • crates/services/rds/Cargo.toml is excluded by none and included by none
  • crates/tests/integration/Cargo.toml is excluded by none and included by none
  • manifests/guided/cloudtrail.guided.json is excluded by none and included by none
  • manifests/guided/cognito.guided.json is excluded by none and included by none
  • manifests/guided/ecs.guided.json is excluded by none and included by none
  • manifests/guided/elasticache.guided.json is excluded by none and included by none
  • manifests/guided/rds.guided.json is excluded by none and included by none
  • tests/bench/bench_services.sh is excluded by none and included by none
  • tests/harness/service-matrix.json is excluded by none and included by none
📒 Files selected for processing (72)
  • .github/workflows/benchmark.yml
  • .github/workflows/ci.yml
  • Cargo.toml
  • crates/aws-protocol/src/protocol.rs
  • crates/aws-protocol/tests/protocol_integration.rs
  • crates/config/src/services.rs
  • crates/gateway/src/server.rs
  • crates/openstack/src/main.rs
  • crates/services/apigateway/tests/apigateway_perf_tests.rs
  • crates/services/cloudformation/tests/cloudformation_perf_tests.rs
  • crates/services/cloudtrail/src/error.rs
  • crates/services/cloudtrail/src/lib.rs
  • crates/services/cloudtrail/src/provider.rs
  • crates/services/cloudtrail/src/store.rs
  • crates/services/cloudtrail/tests/cloudtrail_perf_tests.rs
  • crates/services/cloudtrail/tests/cloudtrail_tests.rs
  • crates/services/cognito/src/error.rs
  • crates/services/cognito/src/lib.rs
  • crates/services/cognito/src/provider.rs
  • crates/services/cognito/src/store.rs
  • crates/services/cognito/tests/cognito_perf_tests.rs
  • crates/services/cognito/tests/cognito_tests.rs
  • crates/services/ec2/src/provider.rs
  • crates/services/ec2/src/store.rs
  • crates/services/ec2/tests/ec2_perf_tests.rs
  • crates/services/ec2/tests/ec2_tests.rs
  • crates/services/ecs/src/error.rs
  • crates/services/ecs/src/lib.rs
  • crates/services/ecs/src/provider.rs
  • crates/services/ecs/src/store.rs
  • crates/services/ecs/tests/ecs_perf_tests.rs
  • crates/services/ecs/tests/ecs_tests.rs
  • crates/services/elasticache/src/error.rs
  • crates/services/elasticache/src/lib.rs
  • crates/services/elasticache/src/provider.rs
  • crates/services/elasticache/src/store.rs
  • crates/services/elasticache/tests/elasticache_perf_tests.rs
  • crates/services/elasticache/tests/elasticache_tests.rs
  • crates/services/firehose/tests/firehose_perf_tests.rs
  • crates/services/iam/tests/iam_perf_tests.rs
  • crates/services/kinesis/tests/kinesis_perf_tests.rs
  • crates/services/opensearch/src/provider.rs
  • crates/services/opensearch/src/store.rs
  • crates/services/opensearch/tests/opensearch_perf_tests.rs
  • crates/services/opensearch/tests/opensearch_tests.rs
  • crates/services/rds/src/error.rs
  • crates/services/rds/src/lib.rs
  • crates/services/rds/src/provider.rs
  • crates/services/rds/src/store.rs
  • crates/services/rds/tests/rds_perf_tests.rs
  • crates/services/rds/tests/rds_tests.rs
  • crates/services/redshift/src/provider.rs
  • crates/services/redshift/src/store.rs
  • crates/services/redshift/tests/redshift_perf_tests.rs
  • crates/services/redshift/tests/redshift_tests.rs
  • crates/services/route53/src/provider.rs
  • crates/services/route53/src/store.rs
  • crates/services/route53/tests/route53_tests.rs
  • crates/services/s3/src/object_store.rs
  • crates/services/ses/src/provider.rs
  • crates/services/ses/src/store.rs
  • crates/services/ses/tests/ses_tests.rs
  • crates/services/sns/tests/sns_perf_tests.rs
  • crates/services/ssm/src/provider.rs
  • crates/services/ssm/src/store.rs
  • crates/services/ssm/tests/ssm_perf_tests.rs
  • crates/services/ssm/tests/ssm_tests.rs
  • crates/services/stepfunctions/tests/stepfunctions_perf_tests.rs
  • crates/services/sts/src/provider.rs
  • crates/services/sts/tests/sts_tests.rs
  • crates/tests/integration/src/harness.rs
  • crates/tests/integration/src/parity.rs

Comment thread .github/workflows/benchmark.yml
Comment thread .github/workflows/benchmark.yml Outdated
Comment thread .github/workflows/benchmark.yml
Comment thread crates/aws-protocol/src/protocol.rs Outdated
Comment thread crates/config/src/services.rs
Comment on lines +469 to +774
// ----------------------------------------------------------------
// ListHostedZonesByName GET /2013-04-01/hostedzonesbyname
// ----------------------------------------------------------------
"ListHostedZonesByName" => {
let Some(store) = self.store.get(account_id, ROUTE53_REGION) else {
let body = format!(
"<?xml version=\"1.0\" encoding=\"UTF-8\"?>\
<ListHostedZonesByNameResponse xmlns=\"{ROUTE53_NS}\">\
<HostedZones></HostedZones>\
<IsTruncated>false</IsTruncated>\
<MaxItems>100</MaxItems>\
</ListHostedZonesByNameResponse>"
);
return Ok(xml_ok(body));
};
let dns_name_filter = ctx
.query_params
.get("dnsname")
.map(|s| s.as_str())
.unwrap_or("");
let mut zones: Vec<&HostedZone> = store
.zones
.values()
.filter(|z| {
dns_name_filter.is_empty()
|| z.name
.trim_end_matches('.')
.ends_with(dns_name_filter.trim_end_matches('.'))
})
.collect();
zones.sort_by(|a, b| a.name.cmp(&b.name));
let zones_xml: String = zones.iter().map(|z| zone_xml(z)).collect();
let body = format!(
"<?xml version=\"1.0\" encoding=\"UTF-8\"?>\
<ListHostedZonesByNameResponse xmlns=\"{ROUTE53_NS}\">\
<HostedZones>{zones_xml}</HostedZones>\
<IsTruncated>false</IsTruncated>\
<MaxItems>100</MaxItems>\
</ListHostedZonesByNameResponse>"
);
Ok(xml_ok(body))
}

// ----------------------------------------------------------------
// GetHostedZoneCount GET /2013-04-01/hostedzonecount
// ----------------------------------------------------------------
"GetHostedZoneCount" => {
let count = self
.store
.get(account_id, ROUTE53_REGION)
.map(|s| s.zones.len())
.unwrap_or(0);
let body = format!(
"<?xml version=\"1.0\" encoding=\"UTF-8\"?>\
<GetHostedZoneCountResponse xmlns=\"{ROUTE53_NS}\">\
<HostedZoneCount>{count}</HostedZoneCount>\
</GetHostedZoneCountResponse>"
);
Ok(xml_ok(body))
}

// ----------------------------------------------------------------
// CreateHealthCheck POST /2013-04-01/healthcheck
// ----------------------------------------------------------------
"CreateHealthCheck" => {
let raw = String::from_utf8_lossy(ctx.raw_body_bytes());
let caller_reference = xml_text(&raw, "CallerReference").unwrap_or_else(req_id);
let health_check_type =
xml_text(&raw, "Type").unwrap_or_else(|| "HTTP".to_string());
let ip_address = xml_text(&raw, "IPAddress");
let port: u16 = xml_text(&raw, "Port")
.and_then(|s| s.parse().ok())
.unwrap_or(80);
let resource_path = xml_text(&raw, "ResourcePath");
let fqdn = xml_text(&raw, "FullyQualifiedDomainName");
let request_interval: u32 = xml_text(&raw, "RequestInterval")
.and_then(|s| s.parse().ok())
.unwrap_or(30);
let failure_threshold: u32 = xml_text(&raw, "FailureThreshold")
.and_then(|s| s.parse().ok())
.unwrap_or(3);

let hc_id = short_id();
let hc = HealthCheck {
id: hc_id.clone(),
caller_reference,
config: HealthCheckConfig {
ip_address,
port,
health_check_type,
resource_path,
fully_qualified_domain_name: fqdn,
request_interval,
failure_threshold,
},
health_check_version: 1,
};

let mut store = self.store.get_or_create(account_id, ROUTE53_REGION);
store.health_checks.insert(hc_id.clone(), hc.clone());

let body = format!(
"<?xml version=\"1.0\" encoding=\"UTF-8\"?>\
<CreateHealthCheckResponse xmlns=\"{ROUTE53_NS}\">\
<HealthCheck>{}</HealthCheck>\
</CreateHealthCheckResponse>",
health_check_xml(&hc)
);
Ok(xml_created(
body,
&format!("/2013-04-01/healthcheck/{hc_id}"),
))
}

// ----------------------------------------------------------------
// GetHealthCheck GET /2013-04-01/healthcheck/{Id}
// ----------------------------------------------------------------
"GetHealthCheck" => {
let hc_id = ctx.path.split('/').next_back().unwrap_or("").to_string();
let Some(store) = self.store.get(account_id, ROUTE53_REGION) else {
return Ok(xml_error(
"NoSuchHealthCheck",
&format!("No health check with ID: {hc_id}"),
404,
));
};
match store.health_checks.get(&hc_id) {
Some(hc) => {
let body = format!(
"<?xml version=\"1.0\" encoding=\"UTF-8\"?>\
<GetHealthCheckResponse xmlns=\"{ROUTE53_NS}\">\
<HealthCheck>{}</HealthCheck>\
</GetHealthCheckResponse>",
health_check_xml(hc)
);
Ok(xml_ok(body))
}
None => Ok(xml_error(
"NoSuchHealthCheck",
&format!("No health check with ID: {hc_id}"),
404,
)),
}
}

// ----------------------------------------------------------------
// DeleteHealthCheck DELETE /2013-04-01/healthcheck/{Id}
// ----------------------------------------------------------------
"DeleteHealthCheck" => {
let hc_id = ctx.path.split('/').next_back().unwrap_or("").to_string();
let mut store = self.store.get_or_create(account_id, ROUTE53_REGION);
if store.health_checks.remove(&hc_id).is_none() {
return Ok(xml_error(
"NoSuchHealthCheck",
&format!("No health check with ID: {hc_id}"),
404,
));
}
let body = format!(
"<?xml version=\"1.0\" encoding=\"UTF-8\"?>\
<DeleteHealthCheckResponse xmlns=\"{ROUTE53_NS}\"></DeleteHealthCheckResponse>"
);
Ok(xml_ok(body))
}

// ----------------------------------------------------------------
// ListHealthChecks GET /2013-04-01/healthcheck
// ----------------------------------------------------------------
"ListHealthChecks" => {
let Some(store) = self.store.get(account_id, ROUTE53_REGION) else {
let body = format!(
"<?xml version=\"1.0\" encoding=\"UTF-8\"?>\
<ListHealthChecksResponse xmlns=\"{ROUTE53_NS}\">\
<HealthChecks></HealthChecks>\
<IsTruncated>false</IsTruncated>\
<MaxItems>100</MaxItems>\
</ListHealthChecksResponse>"
);
return Ok(xml_ok(body));
};
let hcs_xml: String = store
.health_checks
.values()
.map(|hc| format!("<HealthCheck>{}</HealthCheck>", health_check_xml(hc)))
.collect();
let body = format!(
"<?xml version=\"1.0\" encoding=\"UTF-8\"?>\
<ListHealthChecksResponse xmlns=\"{ROUTE53_NS}\">\
<HealthChecks>{hcs_xml}</HealthChecks>\
<IsTruncated>false</IsTruncated>\
<MaxItems>100</MaxItems>\
</ListHealthChecksResponse>"
);
Ok(xml_ok(body))
}

// ----------------------------------------------------------------
// ChangeTagsForResource POST /2013-04-01/tags/{ResourceType}/{ResourceId}
// ----------------------------------------------------------------
"ChangeTagsForResource" => {
// Path: /2013-04-01/tags/{resourcetype}/{resourceid}
let parts: Vec<&str> = ctx.path.split('/').collect();
let (resource_type, resource_id) = parts
.iter()
.enumerate()
.find(|(_, p)| *p == &"tags")
.map(|(i, _)| {
let rtype = parts.get(i + 1).copied().unwrap_or("");
let rid = parts.get(i + 2).copied().unwrap_or("");
(rtype.to_string(), rid.to_string())
})
.unwrap_or_default();

let raw = String::from_utf8_lossy(ctx.raw_body_bytes());
let mut store = self.store.get_or_create(account_id, ROUTE53_REGION);
let tag_map = store.tags.entry((resource_type, resource_id)).or_default();

// Parse <AddTags><Tag><Key>...</Key><Value>...</Value></Tag></AddTags>
let mut rest = raw.as_ref();
while let Some(start) = rest.find("<Tag>") {
let chunk = &rest[start..];
let end = chunk.find("</Tag>").unwrap_or(chunk.len());
let tag_block = &chunk[..end];
if let (Some(key), Some(value)) =
(xml_text(tag_block, "Key"), xml_text(tag_block, "Value"))
{
tag_map.insert(key, value);
}
rest = &rest[start + end..];
}

// Parse <RemoveTagKeys><Key>...</Key></RemoveTagKeys>
let mut remove_rest = raw.as_ref();
if let Some(remove_start) = remove_rest.find("<RemoveTagKeys>") {
let remove_chunk = &remove_rest[remove_start..];
let remove_end = remove_chunk
.find("</RemoveTagKeys>")
.unwrap_or(remove_chunk.len());
let remove_block = &remove_chunk[..remove_end];
remove_rest = remove_block;
while let Some(ks) = remove_rest.find("<Key>") {
let after = &remove_rest[ks + 5..];
if let Some(ke) = after.find("</Key>") {
let k = &after[..ke];
tag_map.remove(k);
}
remove_rest = &remove_rest[ks + 5..];
}
}

let body = format!(
"<?xml version=\"1.0\" encoding=\"UTF-8\"?>\
<ChangeTagsForResourceResponse xmlns=\"{ROUTE53_NS}\"></ChangeTagsForResourceResponse>"
);
Ok(xml_ok(body))
}

// ----------------------------------------------------------------
// ListTagsForResource GET /2013-04-01/tags/{ResourceType}/{ResourceId}
// ----------------------------------------------------------------
"ListTagsForResource" => {
let parts: Vec<&str> = ctx.path.split('/').collect();
let (resource_type, resource_id) = parts
.iter()
.enumerate()
.find(|(_, p)| *p == &"tags")
.map(|(i, _)| {
let rtype = parts.get(i + 1).copied().unwrap_or("");
let rid = parts.get(i + 2).copied().unwrap_or("");
(rtype.to_string(), rid.to_string())
})
.unwrap_or_default();

let tags_xml = self
.store
.get(account_id, ROUTE53_REGION)
.and_then(|store| {
store
.tags
.get(&(resource_type.clone(), resource_id.clone()))
.map(|tags| {
tags.iter()
.map(|(k, v)| {
format!(
"<Tag><Key>{}</Key><Value>{}</Value></Tag>",
xml_escape(k),
xml_escape(v)
)
})
.collect::<String>()
})
})
.unwrap_or_default();

let body = format!(
"<?xml version=\"1.0\" encoding=\"UTF-8\"?>\
<ListTagsForResourceResponse xmlns=\"{ROUTE53_NS}\">\
<ResourceTagSet>\
<ResourceType>{resource_type}</ResourceType>\
<ResourceId>{resource_id}</ResourceId>\
<Tags>{tags_xml}</Tags>\
</ResourceTagSet>\
</ListTagsForResourceResponse>"
);
Ok(xml_ok(body))
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Description: Verify Route53 test coverage for new operations.

echo "=== Checking Route53 unit tests ==="
if [[ -f crates/services/route53/tests/route53_tests.rs ]]; then
  rg -n 'health_check|ListHostedZonesByName|GetHostedZoneCount|ChangeTagsForResource|ListTagsForResource' crates/services/route53/tests/route53_tests.rs | head -10
else
  echo "Route53 unit test file not found or incomplete"
fi

echo "=== Checking Route53 parity scenarios ==="
fd -e json . tests/parity/scenarios/ -x cat {} \; | grep -i 'route53.*health\|ChangeTagsForResource' || echo "No parity scenarios found for new operations"

Repository: JesseKoldewijn/openstack

Length of output: 626


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "=== Route53 tests directory ==="
ls -la crates/services/route53/tests || true

echo
echo "=== Search unit tests for 8 operations ==="
for f in crates/services/route53/tests/*.rs; do
  [ -f "$f" ] || continue
  echo "--- $f ---"
  rg -n "ListHostedZonesByName|GetHostedZoneCount|CreateHealthCheck|GetHealthCheck|DeleteHealthCheck|ListHealthChecks|ChangeTagsForResource|ListTagsForResource" "$f" || true
done

echo
echo "=== Search perf tests for std::time::Instant + operations ==="
for f in crates/services/route53/tests/*_perf_tests.rs; do
  [ -f "$f" ] || continue
  echo "--- $f ---"
  rg -n "Instant|ListHostedZonesByName|GetHostedZoneCount|CreateHealthCheck|GetHealthCheck|DeleteHealthCheck|ListHealthChecks|ChangeTagsForResource|ListTagsForResource" "$f" || true
done

echo
echo "=== Search parity scenarios for Route53 and operations ==="
# list a few scenario files
fd -e json . tests/parity/scenarios/ | head -n 50

echo
echo "Route53-related parity files (by keyword 'route53'):"
rg -n -i "route53" tests/parity/scenarios/ --glob "*.json" || true

echo
echo "Parity scenarios mentioning each operation: "
for op in ListHostedZonesByName GetHostedZoneCount CreateHealthCheck GetHealthCheck DeleteHealthCheck ListHealthChecks ChangeTagsForResource ListTagsForResource; do
  echo "--- $op ---"
  rg -n "$op" tests/parity/scenarios/ --glob "*.json" || true
done

Repository: JesseKoldewijn/openstack

Length of output: 5741


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "=== Inspect route53_perf_tests.rs ==="
ls -la crates/services/route53/tests/route53_perf_tests.rs
wc -l crates/services/route53/tests/route53_perf_tests.rs
rg -n "health[-_ ]?check|healthcheck|health[-_ ]?checks|HostedZonesByName|hosted[-_ ]?zones[-_ ]?by[-_ ]?name|hosted[-_ ]?zone[-_ ]?count|tags[-_ ]?for[-_ ]?resource|list[-_ ]?tags[-_ ]?for[-_ ]?resource|ChangeTagsForResource|ListTagsForResource" crates/services/route53/tests/route53_perf_tests.rs || true

echo
echo "=== Show perf test sections around Instant::now() ==="
# print small windows around each occurrence
perl -ne 'if(/Instant::now\(\)/){ $line=$.; print "---- around $line ----\n"; }' crates/services/route53/tests/route53_perf_tests.rs >/tmp/lines.txt
cat /tmp/lines.txt || true
while read -r L; do
  # L looks like "---- around 123 ----"
  n=$(echo "$L" | rg -o '[0-9]+' | tail -1)
  if [[ -n "${n:-}" ]]; then
    echo "---- $n ----"
    sed -n "$((n-20)),$((n+20))p" crates/services/route53/tests/route53_perf_tests.rs
  fi
done < <(rg -n "Instant::now\(\)" crates/services/route53/tests/route53_perf_tests.rs | head -n 20 | awk -F: '{print "---- around "$1" ----"}')

echo
echo "=== Search parity scenarios for CLI-like names ==="
for f in tests/parity/scenarios/core.json tests/parity/scenarios/extended.json; do
  echo "--- $f ---"
  rg -n -i "route53.*(list-hosted-zones-by-name|get-hosted-zone-count|create-health-check|get-health-check|delete-health-check|list-health-checks|change-tags-for-resource|list-tags-for-resource|listhealthchecks|change.*tags.*resource|list.*tags.*resource|healthcheck)" "$f" || true
done

echo
echo "=== Search parity scenarios for API-like names ==="
for f in tests/parity/scenarios/core.json tests/parity/scenarios/extended.json; do
  echo "--- $f ---"
  rg -n "ListHostedZonesByName|GetHostedZoneCount|CreateHealthCheck|GetHealthCheck|DeleteHealthCheck|ListHealthChecks|ChangeTagsForResource|ListTagsForResource" "$f" || true
done

Repository: JesseKoldewijn/openstack

Length of output: 498


🏁 Script executed:

#!/bin/bash
set -euo pipefail

PERF="crates/services/route53/tests/route53_perf_tests.rs"
UNIT="crates/services/route53/tests/route53_tests.rs"
CORE="tests/parity/scenarios/core.json"
EXT="tests/parity/scenarios/extended.json"

echo "=== route53_perf_tests.rs (first 220 lines) ==="
sed -n '1,220p' "$PERF"

echo
echo "=== route53_perf_tests.rs (last 80 lines) ==="
sed -n '181,261p' "$PERF"

echo
echo "=== route53_perf_tests.rs: lines containing key identifiers ==="
rg -n "dispatch\(&make_ctx|make_ctx\(|Instant::now\(\)|ListHostedZonesByName|GetHostedZoneCount|CreateHealthCheck|GetHealthCheck|DeleteHealthCheck|ListHealthChecks|ChangeTagsForResource|ListTagsForResource" "$PERF" || true

echo
echo "=== route53_tests.rs: locate operation-specific unit tests ==="
rg -n "ListHostedZonesByName|GetHostedZoneCount|CreateHealthCheck|GetHealthCheck|DeleteHealthCheck|ListHealthChecks|ChangeTagsForResource|ListTagsForResource" "$UNIT"

echo
echo "=== route53_tests.rs: show surrounding blocks for each operation ==="
for pat in "ListHostedZonesByName" "GetHostedZoneCount" "CreateHealthCheck" "GetHealthCheck" "DeleteHealthCheck" "ListHealthChecks" "ChangeTagsForResource" "ListTagsForResource"; do
  echo
  echo "--- block around $pat ---"
  # show ~35 lines around first match for each op
  line=$(rg -n "$pat" "$UNIT" | head -n1 | awk -F: '{print $1}')
  if [[ -n "$line" ]]; then
    start=$((line-25)); end=$((line+80));
    if (( start < 1 )); then start=1; fi
    sed -n "${start},${end}p" "$UNIT"
  fi
done

echo
echo "=== parity core/extended: search for CLI-like route53 subcommands ==="
for f in "$CORE" "$EXT"; do
  echo "--- $f ---"
  rg -n -i "route53.*(list-hosted-zones-by-name|get-hosted-zone-count|create-health-check|get-health-check|delete-health-check|list-health-checks|change-tags-for-resource|list-tags-for-resource)" "$f" || true
done

echo
echo "=== parity core/extended: search for API operation-like strings ==="
for f in "$CORE" "$EXT"; do
  echo "--- $f ---"
  rg -n "(ListHostedZonesByName|GetHostedZoneCount|CreateHealthCheck|GetHealthCheck|DeleteHealthCheck|ListHealthChecks|ChangeTagsForResource|ListTagsForResource)" "$f" || true
done

Repository: JesseKoldewijn/openstack

Length of output: 34079


🏁 Script executed:

#!/bin/bash
set -euo pipefail

UNIT="crates/services/route53/tests/route53_tests.rs"
PERF="crates/services/route53/tests/route53_perf_tests.rs"

echo "=== Confirm unit tests cover each operation name ==="
for op in ListHostedZonesByName GetHostedZoneCount CreateHealthCheck GetHealthCheck DeleteHealthCheck ListHealthChecks ChangeTagsForResource ListTagsForResource; do
  echo "--- $op ---"
  rg -n "\"$op\"|fn test_.*$op|$op" "$UNIT" | head -n 50 || true
done

echo
echo "=== Look for error-path tests (4xx/Code/NoSuch/Error) near each operation ==="
for op in ListHostedZonesByName GetHostedZoneCount CreateHealthCheck GetHealthCheck DeleteHealthCheck ListHealthChecks ChangeTagsForResource ListTagsForResource; do
  echo
  echo "--- $op (near keywords: NoSuch|not found|404|error) ---"
  # Show up to 120 lines around the first occurrence for each op
  line=$(rg -n "\"$op\"" "$UNIT" | head -n1 | awk -F: '{print $1}') || true
  if [[ -n "${line:-}" ]]; then
    start=$((line-60)); end=$((line+180))
    if (( start < 1 )); then start=1; fi
    sed -n "${start},${end}p" "$UNIT" | rg -n "NoSuch|not found|404|error|Code" || true
  fi
done

echo
echo "=== Confirm perf tests do NOT mention the 8 operations ==="
for op in ListHostedZonesByName GetHostedZoneCount CreateHealthCheck GetHealthCheck DeleteHealthCheck ListHealthChecks ChangeTagsForResource ListTagsForResource; do
  echo "--- $op ---"
  rg -n "$op" "$PERF" || true
done

echo
echo "=== Quick perf file scan for key structs/dispatch ops ==="
rg -n "dispatch\\(&make_ctx\\(|Instant::now\\(" "$PERF" | head -n 50

Repository: JesseKoldewijn/openstack

Length of output: 4140


Route53: missing required parity/perf coverage and incomplete error-path unit tests for 7/8 listed operations

  • tests/parity/scenarios/core.json and tests/parity/scenarios/extended.json contain no parity scenarios for: ListHostedZonesByName, GetHostedZoneCount, CreateHealthCheck, GetHealthCheck, DeleteHealthCheck, ListHealthChecks, ChangeTagsForResource, ListTagsForResource.
  • crates/services/route53/tests/route53_perf_tests.rs has Instant-based performance assertions, but none for these 8 operations (only hosted-zone/rrset/control-plane ops).
  • crates/services/route53/tests/route53_tests.rs includes happy-path tests for all 8 operations, but only GetHealthCheck has an explicit error-path test (test_get_health_check_not_found); the other 7 operations lack an error-path unit test.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/services/route53/src/provider.rs` around lines 469 - 774, The PR is
missing parity scenario entries, performance assertions, and error-path unit
tests for the eight Route53 operations; add parity scenarios for
"ListHostedZonesByName", "GetHostedZoneCount", "CreateHealthCheck",
"GetHealthCheck", "DeleteHealthCheck", "ListHealthChecks",
"ChangeTagsForResource", and "ListTagsForResource" to
tests/parity/scenarios/core.json and extended.json; add Instant-based perf
assertions in crates/services/route53/tests/route53_perf_tests.rs exercising
those handlers (same pattern used for hosted-zone/rrset control-plane ops); and
add unit tests in crates/services/route53/tests/route53_tests.rs that exercise
the error paths for each handler (e.g., missing store or missing resource)
verifying the XML error responses produced by the handler branches (look at
handlers matching "ListHostedZonesByName", "GetHostedZoneCount",
"CreateHealthCheck", "DeleteHealthCheck", "ListHealthChecks",
"ChangeTagsForResource", "ListTagsForResource" and reuse the existing test
pattern from test_get_health_check_not_found to assert xml_error outputs and
status codes).

Source: Coding guidelines

Comment on lines +522 to +568
#[tokio::test]
async fn test_create_and_get_health_check() {
let p = Route53Provider::new();
let hc_xml = r#"<CreateHealthCheckRequest>
<CallerReference>hc-ref-1</CallerReference>
<HealthCheckConfig>
<IPAddress>1.2.3.4</IPAddress>
<Port>80</Port>
<Type>HTTP</Type>
<ResourcePath>/health</ResourcePath>
<RequestInterval>30</RequestInterval>
<FailureThreshold>3</FailureThreshold>
</HealthCheckConfig>
</CreateHealthCheckRequest>"#;

let resp = p
.dispatch(&make_ctx(
"CreateHealthCheck",
hc_xml,
"/2013-04-01/healthcheck",
"POST",
))
.await
.unwrap();
assert_eq!(resp.status_code, 201);
let body = body_str(&resp);
assert!(body.contains("CreateHealthCheckResponse"));
assert!(body.contains("<Type>HTTP</Type>"));
assert!(body.contains("1.2.3.4"));

let hc_id = xml_text(&body, "Id").unwrap();

let get_resp = p
.dispatch(&make_ctx(
"GetHealthCheck",
"",
&format!("/2013-04-01/healthcheck/{hc_id}"),
"GET",
))
.await
.unwrap();
assert_eq!(get_resp.status_code, 200);
let get_body = body_str(&get_resp);
assert!(get_body.contains("GetHealthCheckResponse"));
assert!(get_body.contains("1.2.3.4"));
assert!(get_body.contains("<Port>80</Port>"));
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Missing error-path coverage for CreateHealthCheck and DeleteHealthCheck.

The test suite covers happy-path flows for both operations but lacks error-path tests:

  1. CreateHealthCheck (lines 522–568): No test for missing required fields (e.g., CallerReference, Port, Type). The provider should validate these and return a 400 error; add a test to verify that behavior.
  2. DeleteHealthCheck (lines 623–662): No test for attempting to delete a nonexistent health check. Add a test that calls DeleteHealthCheck on a health check ID that was never created and assert the expected error response (likely 404 with NoSuchHealthCheck).

As per coding guidelines, every operation must have at least one test covering the primary error path.

Also applies to: 623-662

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/services/route53/tests/route53_tests.rs` around lines 522 - 568, Add
error-path tests alongside test_create_and_get_health_check: (1) Add a test
(e.g., test_create_health_check_missing_fields) that uses
Route53Provider::dispatch with make_ctx("CreateHealthCheck", <invalid XML
missing CallerReference/Port/Type>, "/2013-04-01/healthcheck", "POST") and
assert resp.status_code == 400 and body contains an error element/validation
message; (2) Add a test (e.g., test_delete_nonexistent_health_check) that calls
dispatch with make_ctx("DeleteHealthCheck", "",
&format!("/2013-04-01/healthcheck/nonexistent-id"), "DELETE") and assert
resp.status_code == 404 and body contains "NoSuchHealthCheck"; follow the same
pattern as test_create_and_get_health_check for using body_str/xml_text to
inspect response contents.

Source: Coding guidelines

Comment on lines +288 to +578
// ----------------------------------------------------------------
// GetSendQuota
// ----------------------------------------------------------------
"GetSendQuota" => {
let sent_count = self
.store
.get(account_id, region)
.map(|s| s.emails.len() as f64)
.unwrap_or(0.0);
let inner = format!(
"<Max24HourSend>50000.0</Max24HourSend>\
<MaxSendRate>14.0</MaxSendRate>\
<SentLast24Hours>{sent_count}</SentLast24Hours>"
);
Ok(xml_resp("GetSendQuota", &rid, &inner))
}

// ----------------------------------------------------------------
// GetSendStatistics
// ----------------------------------------------------------------
"GetSendStatistics" => {
let data_points = self
.store
.get(account_id, region)
.map(|store| {
// Group emails by hour bucket for realistic stats
let total = store.emails.len() as u64;
if total == 0 {
return String::new();
}
format!(
"<member>\
<Timestamp>{}</Timestamp>\
<DeliveryAttempts>{total}</DeliveryAttempts>\
<Bounces>0</Bounces>\
<Complaints>0</Complaints>\
<Rejects>0</Rejects>\
</member>",
Utc::now().format("%Y-%m-%dT%H:00:00Z")
)
})
.unwrap_or_default();
let inner = format!("<SendDataPoints>{data_points}</SendDataPoints>");
Ok(xml_resp("GetSendStatistics", &rid, &inner))
}

// ----------------------------------------------------------------
// CreateTemplate
// ----------------------------------------------------------------
"CreateTemplate" => {
let template_name = match str_param(ctx, "Template.TemplateName") {
Some(n) => n.to_string(),
None => {
return Ok(xml_error(
"MissingParameter",
"Template.TemplateName required",
400,
));
}
};
let subject_part = str_param(ctx, "Template.SubjectPart")
.unwrap_or("")
.to_string();
let html_part = str_param(ctx, "Template.HtmlPart")
.unwrap_or("")
.to_string();
let text_part = str_param(ctx, "Template.TextPart")
.unwrap_or("")
.to_string();

let mut store = self.store.get_or_create(account_id, region);
if store.templates.contains_key(&template_name) {
return Ok(xml_error(
"AlreadyExists",
&format!("Template {template_name} already exists"),
400,
));
}
store.templates.insert(
template_name.clone(),
EmailTemplate {
template_name,
subject_part,
html_part,
text_part,
created_at: Utc::now(),
},
);
Ok(xml_no_result("CreateTemplate", &rid))
}

// ----------------------------------------------------------------
// DeleteTemplate
// ----------------------------------------------------------------
"DeleteTemplate" => {
let template_name = match str_param(ctx, "TemplateName") {
Some(n) => n.to_string(),
None => {
return Ok(xml_error("MissingParameter", "TemplateName required", 400));
}
};
let mut store = self.store.get_or_create(account_id, region);
store.templates.remove(&template_name);
Ok(xml_no_result("DeleteTemplate", &rid))
}

// ----------------------------------------------------------------
// GetTemplate
// ----------------------------------------------------------------
"GetTemplate" => {
let template_name = match str_param(ctx, "TemplateName") {
Some(n) => n.to_string(),
None => {
return Ok(xml_error("MissingParameter", "TemplateName required", 400));
}
};
let Some(store) = self.store.get(account_id, region) else {
return Ok(xml_error(
"TemplateDoesNotExist",
&format!("Template {template_name} does not exist"),
400,
));
};
match store.templates.get(&template_name) {
Some(t) => {
let inner = format!(
"<Template>\
<TemplateName>{}</TemplateName>\
<SubjectPart>{}</SubjectPart>\
<HtmlPart>{}</HtmlPart>\
<TextPart>{}</TextPart>\
</Template>",
t.template_name, t.subject_part, t.html_part, t.text_part
);
Ok(xml_resp("GetTemplate", &rid, &inner))
}
None => Ok(xml_error(
"TemplateDoesNotExist",
&format!("Template {template_name} does not exist"),
400,
)),
}
}

// ----------------------------------------------------------------
// ListTemplates
// ----------------------------------------------------------------
"ListTemplates" => {
let Some(store) = self.store.get(account_id, region) else {
return Ok(xml_resp("ListTemplates", &rid, "<TemplatesMetadata />"));
};
let members: String = store
.templates
.values()
.map(|t| {
format!(
"<member><Name>{}</Name><CreatedTimestamp>{}</CreatedTimestamp></member>",
t.template_name,
t.created_at.format("%Y-%m-%dT%H:%M:%SZ")
)
})
.collect();
let inner = if members.is_empty() {
"<TemplatesMetadata />".to_string()
} else {
format!("<TemplatesMetadata>{members}</TemplatesMetadata>")
};
Ok(xml_resp("ListTemplates", &rid, &inner))
}

// ----------------------------------------------------------------
// SendTemplatedEmail
// ----------------------------------------------------------------
"SendTemplatedEmail" => {
let source = match str_param(ctx, "Source") {
Some(s) => s.to_string(),
None => return Ok(xml_error("MissingParameter", "Source required", 400)),
};
let template_name = match str_param(ctx, "Template") {
Some(t) => t.to_string(),
None => return Ok(xml_error("MissingParameter", "Template required", 400)),
};
let to = addresses_from_params(ctx, "Destination.ToAddresses.member");
let cc = addresses_from_params(ctx, "Destination.CcAddresses.member");
let bcc = addresses_from_params(ctx, "Destination.BccAddresses.member");

let (subject, html_part, text_part) = {
let Some(store) = self.store.get(account_id, region) else {
return Ok(xml_error(
"TemplateDoesNotExist",
&format!("Template {template_name} does not exist"),
400,
));
};
match store.templates.get(&template_name) {
Some(t) => (
t.subject_part.clone(),
t.html_part.clone(),
t.text_part.clone(),
),
None => {
return Ok(xml_error(
"TemplateDoesNotExist",
&format!("Template {template_name} does not exist"),
400,
));
}
}
};

let message_id = Uuid::new_v4().to_string();
let email = StoredEmail {
message_id: message_id.clone(),
source,
destination_to: to,
destination_cc: cc,
destination_bcc: bcc,
subject,
body_text: text_part,
body_html: html_part,
sent_at: Utc::now(),
};
let mut store = self.store.get_or_create(account_id, region);
store.emails.insert(message_id.clone(), email);
let inner = format!("<MessageId>{message_id}</MessageId>");
Ok(xml_resp("SendTemplatedEmail", &rid, &inner))
}

// ----------------------------------------------------------------
// SetIdentityFeedbackForwardingEnabled
// ----------------------------------------------------------------
"SetIdentityFeedbackForwardingEnabled" => {
let identity = match str_param(ctx, "Identity") {
Some(i) => i.to_string(),
None => return Ok(xml_error("MissingParameter", "Identity required", 400)),
};
let enabled = str_param(ctx, "ForwardingEnabled")
.map(|v| v.eq_ignore_ascii_case("true"))
.unwrap_or(true);
let mut store = self.store.get_or_create(account_id, region);
let attrs = store.notification_attrs.entry(identity).or_default();
attrs.forwarding_enabled = enabled;
Ok(xml_no_result("SetIdentityFeedbackForwardingEnabled", &rid))
}

// ----------------------------------------------------------------
// GetIdentityNotificationAttributes
// ----------------------------------------------------------------
"GetIdentityNotificationAttributes" => {
let identities = addresses_from_params(ctx, "Identities.member");
let entries = self
.store
.get(account_id, region)
.map(|store| {
identities
.iter()
.map(|id| {
let attrs = store
.notification_attrs
.get(id)
.cloned()
.unwrap_or_default();
let bounce = attrs
.bounce_topic
.as_deref()
.map(|t| format!("<BounceTopic>{t}</BounceTopic>"))
.unwrap_or_default();
let complaint = attrs
.complaint_topic
.as_deref()
.map(|t| format!("<ComplaintTopic>{t}</ComplaintTopic>"))
.unwrap_or_default();
let delivery = attrs
.delivery_topic
.as_deref()
.map(|t| format!("<DeliveryTopic>{t}</DeliveryTopic>"))
.unwrap_or_default();
format!(
"<entry><key>{id}</key><value>\
{bounce}{complaint}{delivery}\
<ForwardingEnabled>{}</ForwardingEnabled>\
</value></entry>",
attrs.forwarding_enabled
)
})
.collect::<String>()
})
.unwrap_or_default();
let inner = format!("<NotificationAttributes>{entries}</NotificationAttributes>");
Ok(xml_resp("GetIdentityNotificationAttributes", &rid, &inner))
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Description: Verify SES test coverage for new operations.

echo "=== Checking SES unit tests ==="
if [[ -f crates/services/ses/tests/ses_tests.rs ]]; then
  rg -n 'test_.*template|test_.*quota|test_.*notification' crates/services/ses/tests/ses_tests.rs | head -15
else
  echo "SES unit test file may be incomplete"
fi

echo "=== Checking SES parity scenarios ==="
fd -e json . tests/parity/scenarios/ -x cat {} \; | grep -i 'ses.*template\|GetSendQuota\|GetSendStatistics' || echo "No parity scenarios found for new operations"

Repository: JesseKoldewijn/openstack

Length of output: 657


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "=== Locate SES perf tests file ==="
fd -g'*ses*_perf_tests.rs' crates/services/ses/tests -a 2>/dev/null || true
fd -e rs ses_perf_tests.rs crates/services/ses/tests -a 2>/dev/null || true
fd -e rs ses_perf_tests crates/services/ses/tests -a 2>/dev/null || true
# If file exists, search for operation identifiers
if ls crates/services/ses/tests/*perf*tests*.rs >/dev/null 2>&1; then
  echo "=== Searching perf tests for operation names ==="
  rg -n --no-heading "GetSendQuota|GetSendStatistics|CreateTemplate|DeleteTemplate|GetTemplate|ListTemplates|SendTemplatedEmail|SetIdentityFeedbackForwardingEnabled|GetIdentityNotificationAttributes" crates/services/ses/tests/*perf*tests*.rs || true
fi

echo "=== Search SES unit tests for error-path coverage per operation ==="
if [[ -f crates/services/ses/tests/ses_tests.rs ]]; then
  for op in "GetSendQuota" "GetSendStatistics" "CreateTemplate" "DeleteTemplate" "GetTemplate" "ListTemplates" "SendTemplatedEmail" "SetIdentityFeedbackForwardingEnabled" "GetIdentityNotificationAttributes"; do
    echo "--- unit tests: $op ---"
    rg -n --no-heading "$op|test_.*$(echo "$op" | tr 'A-Z' 'a-z' | sed 's/[^a-z0-9]//g')" crates/services/ses/tests/ses_tests.rs || true
  done
fi

echo "=== Search parity scenarios for SES operation names ==="
if [[ -d tests/parity/scenarios ]]; then
  rg -n --no-heading "GetSendQuota|GetSendStatistics|CreateTemplate|DeleteTemplate|GetTemplate|ListTemplates|SendTemplatedEmail|SetIdentityFeedbackForwardingEnabled|GetIdentityNotificationAttributes" tests/parity/scenarios || true
fi

echo "=== Enumerate any ses-related parity scenario files ==="
if [[ -d tests/parity/scenarios ]]; then
  fd -e json tests/parity/scenarios -t f | rg -i "ses|amazon.*ses" || true
  # show filenames for SES
  fd -e json tests/parity/scenarios -t f -i | rg -i "ses" | head -50 || true
fi

Repository: JesseKoldewijn/openstack

Length of output: 3381


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "=== Locate SES perf tests file ==="
fd -e rs ses_perf_tests.rs crates/services/ses/tests -a 2>/dev/null || true
if ls crates/services/ses/tests/*perf*tests*.rs >/dev/null 2>&1; then
  echo "Perf test files:"
  ls -1 crates/services/ses/tests/*perf*tests*.rs
fi

echo "=== Searching perf tests for operation names ==="
if ls crates/services/ses/tests/*perf*tests*.rs >/dev/null 2>&1; then
  rg -n --no-heading "GetSendQuota|GetSendStatistics|CreateTemplate|DeleteTemplate|GetTemplate|ListTemplates|SendTemplatedEmail|SetIdentityFeedbackForwardingEnabled|GetIdentityNotificationAttributes" crates/services/ses/tests/*perf*tests*.rs || true
fi

echo "=== Search SES unit tests for coverage per operation ==="
if [[ -f crates/services/ses/tests/ses_tests.rs ]]; then
  for op in "GetSendQuota" "GetSendStatistics" "CreateTemplate" "DeleteTemplate" "GetTemplate" "ListTemplates" "SendTemplatedEmail" "SetIdentityFeedbackForwardingEnabled" "GetIdentityNotificationAttributes"; do
    echo "--- unit tests: $op ---"
    rg -n --no-heading "$op" crates/services/ses/tests/ses_tests.rs || true
  done

  echo "=== unit tests: function name patterns (best-effort) ==="
  rg -n --no-heading "test_.*quota|test_.*statistics|test_.*template|test_.*send_templated|test_.*identity_notification" crates/services/ses/tests/ses_tests.rs || true
fi

echo "=== Search parity scenarios for SES operation names ==="
if [[ -d tests/parity/scenarios ]]; then
  rg -n --no-heading "GetSendQuota|GetSendStatistics|CreateTemplate|DeleteTemplate|GetTemplate|ListTemplates|SendTemplatedEmail|SetIdentityFeedbackForwardingEnabled|GetIdentityNotificationAttributes" tests/parity/scenarios || true
fi

echo "=== List SES-related parity scenario files (filenames only) ==="
if [[ -d tests/parity/scenarios ]]; then
  fd -e json tests/parity/scenarios -t f | rg -i "ses" || true
fi

Repository: JesseKoldewijn/openstack

Length of output: 3615


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "=== Inspect ses_perf_tests.rs for Instant + exercised operations ==="
if [[ -f crates/services/ses/tests/ses_perf_tests.rs ]]; then
  wc -l crates/services/ses/tests/ses_perf_tests.rs
  rg -n "Instant::now|std::time::Instant" crates/services/ses/tests/ses_perf_tests.rs || true
  rg -n "make_ctx\\(\"|dispatch\\(&make_ctx\\(|GetSendQuota|GetSendStatistics|CreateTemplate|DeleteTemplate|GetTemplate|ListTemplates|SendTemplatedEmail|SetIdentityFeedbackForwardingEnabled|GetIdentityNotificationAttributes" crates/services/ses/tests/ses_perf_tests.rs || true
  echo "--- Context around any dispatch/make_ctx calls (first 120 matches) ---"
  rg -n "make_ctx\\(\"" crates/services/ses/tests/ses_perf_tests.rs | head -120 || true
else
  echo "Missing crates/services/ses/tests/ses_perf_tests.rs"
fi

echo
echo "=== Inspect ses_tests.rs test bodies for happy vs error-path per operation ==="
if [[ -f crates/services/ses/tests/ses_tests.rs ]]; then
  wc -l crates/services/ses/tests/ses_tests.rs

  # Print the full test functions for each relevant test name (best effort by range around signatures).
  # We'll extract by line numbers from rg and then show ~40 lines around each.
  for sig in \
    "test_get_send_quota_empty" "test_get_send_quota_after_sends" \
    "test_get_send_statistics_empty" "test_get_send_statistics_after_sends" \
    "test_create_and_get_template" "test_create_template_duplicate_fails" \
    "test_get_template_not_found" "test_list_templates" "test_delete_template" \
    "test_send_templated_email" "test_send_templated_email_missing_template" \
    "test_set_identity_notification_attributes" \
    "test_get_identity_notification_attributes"; do
    line=$(rg -n "async fn ${sig}" crates/services/ses/tests/ses_tests.rs | head -1 | cut -d: -f1 || true)
    if [[ -n "${line}" ]]; then
      start=$(( line>10 ? line-10 : 1 ))
      end=$(( line+70 ))
      echo "---- ${sig} (lines ${start}-${end}) ----"
      sed -n "${start},${end}p" crates/services/ses/tests/ses_tests.rs
    fi
  done

  echo
  echo "=== Quick scan: any explicit error assertion helpers in ses_tests.rs ==="
  rg -n "assert_.*error|error|Fault|MissingParameter|TemplateDoesNotExist|AlreadyExists|does not exist|unwrap_err|is_err" crates/services/ses/tests/ses_tests.rs | head -200 || true
else
  echo "Missing crates/services/ses/tests/ses_tests.rs"
fi

echo
echo "=== Search parity scenarios (properly) for SES operations ==="
if [[ -d tests/parity/scenarios ]]; then
  echo "Parity scenario files (json):"
  fd -e json --full-path . tests/parity/scenarios | head -200 || true
  # Search for operation names in JSON scenario files
  rg -n --no-heading "GetSendQuota|GetSendStatistics|CreateTemplate|DeleteTemplate|GetTemplate|ListTemplates|SendTemplatedEmail|SetIdentityFeedbackForwardingEnabled|GetIdentityNotificationAttributes" tests/parity/scenarios -S --glob '*.json' || true
else
  echo "Missing tests/parity/scenarios directory"
fi

Repository: JesseKoldewijn/openstack

Length of output: 34926


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "=== Parity scenarios: search operation names in core.json/extended.json ==="
for f in tests/parity/scenarios/core.json tests/parity/scenarios/extended.json tests/parity/scenarios/all-services-smoke.json; do
  [[ -f "$f" ]] || continue
  echo "--- $f ---"
  rg -n --no-heading \
    "GetSendQuota|GetSendStatistics|CreateTemplate|DeleteTemplate|GetTemplate|ListTemplates|SendTemplatedEmail|SetIdentityFeedbackForwardingEnabled|GetIdentityNotificationAttributes" \
    "$f" -S || echo "No matches"
done

echo
echo "=== List ses_tests.rs functions per operation (quota/statistics/template/etc) ==="
ops_re='GetSendQuota|GetSendStatistics|CreateTemplate|DeleteTemplate|GetTemplate|ListTemplates|SendTemplatedEmail|SetIdentityFeedbackForwardingEnabled|GetIdentityNotificationAttributes'
# Find test fn names around dispatch contexts
rg -n --no-heading "async fn test_.*" crates/services/ses/tests/ses_tests.rs | head -50

for needle in \
  "test_.*quota" "test_.*statistics" \
  "test_create_template" "test_.*template" \
  "test_delete_template" "test_list_templates" \
  "test_send_templated_email" \
  "test_set_identity_feedback_forwarding_enabled" \
  "test_get_identity_notification_attributes"; do
  echo "--- matching: $needle ---"
  rg -n --no-heading "async fn ${needle}" crates/services/ses/tests/ses_tests.rs || true
done

echo
echo "=== Quick check: any error-path assertions tied to specific ops ==="
for op in \
  "GetSendQuota" "GetSendStatistics" "DeleteTemplate" "ListTemplates" \
  "SetIdentityFeedbackForwardingEnabled" "GetIdentityNotificationAttributes" ; do
  echo "--- assertions near $op ---"
  # show nearby lines where status_code/contains MissingParameter/TemplateDoesNotExist/AlreadyExists appear
  # (best-effort: exact matching of error-path per op may require manual inspection)
  rg -n --no-heading "$op|MissingParameter|TemplateDoesNotExist|AlreadyExists" crates/services/ses/tests/ses_tests.rs \
    | head -80 || true
done

Repository: JesseKoldewijn/openstack

Length of output: 6370


SES provider operations are missing required parity/perf coverage and most lack error-path unit tests

  • Parity scenarios missing: tests/parity/scenarios/{core.json,extended.json,all-services-smoke.json} contain no references to: GetSendQuota, GetSendStatistics, CreateTemplate, DeleteTemplate, GetTemplate, ListTemplates, SendTemplatedEmail, SetIdentityFeedbackForwardingEnabled, GetIdentityNotificationAttributes.
  • Performance regression coverage missing: crates/services/ses/tests/ses_perf_tests.rs uses std::time::Instant but does not dispatch/cover any of the above SES operations.
  • Unit tests missing required error paths (happy-only): crates/services/ses/tests/ses_tests.rs includes happy-path tests for GetSendQuota, GetSendStatistics, DeleteTemplate, ListTemplates, SetIdentityFeedbackForwardingEnabled, and GetIdentityNotificationAttributes, but no corresponding error-path tests for those operations.
  • Only some error paths exist: CreateTemplate (duplicate) + GetTemplate (not found) + SendTemplatedEmail (missing template) have error-path tests; the rest do not.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/services/ses/src/provider.rs` around lines 288 - 578, Add missing
parity, perf, and error-path tests for the SES operations: implement scenario
entries referencing GetSendQuota, GetSendStatistics, CreateTemplate,
DeleteTemplate, GetTemplate, ListTemplates, SendTemplatedEmail,
SetIdentityFeedbackForwardingEnabled, and GetIdentityNotificationAttributes in
the parity JSONs (core.json, extended.json, all-services-smoke.json); update
crates/services/ses/tests/ses_perf_tests.rs to dispatch/measure these operations
(use the existing dispatch helper used by other perf tests rather than adding
new timing primitives) so perf coverage includes them; and add unit tests in
crates/services/ses/tests/ses_tests.rs covering the missing error paths (e.g.,
GetSendQuota/GetSendStatistics when no store,
DeleteTemplate/GetTemplate/ListTemplates when template missing,
SendTemplatedEmail when template absent, and GetIdentityNotificationAttributes
for unknown identities) by asserting xml_error responses from the provider
handlers (targeting the same operation names used in provider.rs like
"GetTemplate", "CreateTemplate", "SendTemplatedEmail", etc.).

Source: Coding guidelines

Comment on lines +532 to +576
#[tokio::test]
async fn test_set_identity_feedback_forwarding_enabled() {
let p = SesProvider::new();
let mut verify = HashMap::new();
verify.insert("EmailAddress".to_string(), "notif@example.com".to_string());
p.dispatch(&make_ctx("VerifyEmailIdentity", verify))
.await
.unwrap();

let mut params = HashMap::new();
params.insert("Identity".to_string(), "notif@example.com".to_string());
params.insert("ForwardingEnabled".to_string(), "false".to_string());
let resp = p
.dispatch(&make_ctx("SetIdentityFeedbackForwardingEnabled", params))
.await
.unwrap();
assert_eq!(resp.status_code, 200);
let body = body_str(&resp);
assert!(body.contains("SetIdentityFeedbackForwardingEnabledResponse"));
}

#[tokio::test]
async fn test_get_identity_notification_attributes() {
let p = SesProvider::new();
let mut verify = HashMap::new();
verify.insert("EmailAddress".to_string(), "notif2@example.com".to_string());
p.dispatch(&make_ctx("VerifyEmailIdentity", verify))
.await
.unwrap();

let mut params = HashMap::new();
params.insert(
"Identities.member.1".to_string(),
"notif2@example.com".to_string(),
);
let resp = p
.dispatch(&make_ctx("GetIdentityNotificationAttributes", params))
.await
.unwrap();
assert_eq!(resp.status_code, 200);
let body = body_str(&resp);
assert!(body.contains("GetIdentityNotificationAttributesResponse"));
assert!(body.contains("notif2@example.com"));
assert!(body.contains("<ForwardingEnabled>true</ForwardingEnabled>"));
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick | 🔵 Trivial | ⚡ Quick win

Add error path tests for notification attribute operations.

Per coding guidelines, every operation must have at least one test covering the primary error path. The notification attribute operations (SetIdentityFeedbackForwardingEnabled, GetIdentityNotificationAttributes) only have happy-path tests. Consider adding error tests for scenarios like:

  • Setting feedback forwarding for a non-existent or unverified identity
  • Getting notification attributes for a non-existent identity
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/services/ses/tests/ses_tests.rs` around lines 532 - 576, Add
negative-path tests in the same test module using SesProvider, dispatch and
make_ctx: create one test (e.g.,
test_set_identity_feedback_forwarding_enabled_unverified) that calls
SetIdentityFeedbackForwardingEnabled with an identity that wasn't
VerifyEmailIdentity-verified and assert the response is an error (non-200) and
contains the expected error element/message; create another test (e.g.,
test_get_identity_notification_attributes_nonexistent) that calls
GetIdentityNotificationAttributes with an identity that doesn't exist and assert
it returns an error response and contains the appropriate error element/message;
reuse the existing test patterns (SesProvider::new(), make_ctx, body_str) and
mirror assertions style from test_set_identity_feedback_forwarding_enabled and
test_get_identity_notification_attributes to keep consistency.

Source: Coding guidelines

Comment on lines +151 to +214
"AssumeRoleWithWebIdentity" => {
let role_arn = param(ctx, "RoleArn")
.unwrap_or_else(|| format!("arn:aws:iam::{account_id}:role/default"));
let role_name = role_arn
.rsplit('/')
.next()
.filter(|s| !s.is_empty())
.unwrap_or("default")
.to_string();
let session_name =
param(ctx, "RoleSessionName").unwrap_or_else(|| "web-session".to_string());
let provider_id =
param(ctx, "ProviderId").unwrap_or_else(|| "www.amazon.com".to_string());
let subject = param(ctx, "WebIdentityToken")
.map(|t| t[..t.len().min(12)].to_string())
.unwrap_or_else(|| "sub-fake-0000".to_string());
let creds = temp_credentials_xml("wi//");
let role_id_suffix =
&Uuid::new_v4().to_string().replace('-', "")[..16].to_uppercase();
let inner = format!(
"{creds}\
<SubjectFromWebIdentityToken>{subject}</SubjectFromWebIdentityToken>\
<Provider>{}</Provider>\
<AssumedRoleUser>\
<AssumedRoleId>AROA{role_id_suffix}:{session_name}</AssumedRoleId>\
<Arn>arn:aws:sts::{account_id}:assumed-role/{role_name}/{session_name}</Arn>\
</AssumedRoleUser>",
xml_escape(&provider_id)
);
Ok(xml_resp("AssumeRoleWithWebIdentity", &rid, &inner))
}

"AssumeRoleWithSAML" => {
let role_arn = param(ctx, "RoleArn")
.unwrap_or_else(|| format!("arn:aws:iam::{account_id}:role/default"));
let role_name = role_arn
.rsplit('/')
.next()
.filter(|s| !s.is_empty())
.unwrap_or("default")
.to_string();
let principal_arn = param(ctx, "PrincipalArn").unwrap_or_else(|| {
format!("arn:aws:iam::{account_id}:saml-provider/MyProvider")
});
let session_name = "saml-session".to_string();
let creds = temp_credentials_xml("saml//");
let role_id_suffix =
&Uuid::new_v4().to_string().replace('-', "")[..16].to_uppercase();
let inner = format!(
"{creds}\
<Issuer>{}</Issuer>\
<AssumedRoleUser>\
<AssumedRoleId>AROA{role_id_suffix}:{session_name}</AssumedRoleId>\
<Arn>arn:aws:sts::{account_id}:assumed-role/{role_name}/{session_name}</Arn>\
</AssumedRoleUser>\
<NameQualifier>fake-nq</NameQualifier>\
<Subject>fake-subject</Subject>\
<SubjectType>persistent</SubjectType>\
<Audience>{}</Audience>",
xml_escape(&principal_arn),
xml_escape(&role_arn)
);
Ok(xml_resp("AssumeRoleWithSAML", &rid, &inner))
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Description: Verify STS test coverage for new operations.

echo "=== Checking STS unit tests ==="
if [[ -f crates/services/sts/tests/sts_tests.rs ]]; then
  rg -n 'AssumeRoleWithWebIdentity|AssumeRoleWithSAML' crates/services/sts/tests/sts_tests.rs | head -10
else
  echo "STS unit test file may be incomplete"
fi

echo "=== Checking STS parity scenarios ==="
fd -e json . tests/parity/scenarios/ -x cat {} \; | grep -i 'sts.*web.*identity\|sts.*saml' || echo "No parity scenarios found for new operations"

Repository: JesseKoldewijn/openstack

Length of output: 592


Fix missing required parity coverage for AssumeRoleWithWebIdentity and AssumeRoleWithSAML (plus ensure error/perf tests exist)

  • crates/services/sts/tests/sts_tests.rs already contains unit tests for both operations, covering the success path (asserts status_code == 200 and response XML contains the expected *Response).
  • No parity scenarios matching these operations were found under tests/parity/scenarios/; add core.json or extended.json scenarios that exercise them against both OpenStack and LocalStack.
  • Also ensure each operation has: (1) at least one error-path unit test (using the service’s existing error helper/formatting), and (2) an Instant-timed performance regression test in crates/services/sts/tests/sts_perf_tests.rs.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/services/sts/src/provider.rs` around lines 151 - 214, The PR is
missing parity scenarios and negative/perf tests for the
AssumeRoleWithWebIdentity and AssumeRoleWithSAML handlers: add parity scenario
JSON entries (core.json or extended.json) under tests/parity/scenarios that
invoke "AssumeRoleWithWebIdentity" and "AssumeRoleWithSAML" so they run against
OpenStack and LocalStack; add error-path unit tests in
crates/services/sts/tests/sts_tests.rs that exercise failure branches for those
handlers (use the repo's error helper/formatting) and assert error XML/status;
and add Instant-timed performance tests in
crates/services/sts/tests/sts_perf_tests.rs that call the handlers (e.g., using
the same request construction as in existing success tests) and assert they
complete within expected durations. Ensure test names reference
AssumeRoleWithWebIdentity and AssumeRoleWithSAML so they’re discoverable.

Source: Coding guidelines

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review continued from previous batch...

Comment on lines +366 to +391
#[tokio::test]
async fn test_delete_security_group() {
let p = Ec2Provider::new();
let mut sg_p = HashMap::new();
sg_p.insert("GroupName".to_string(), "del-sg".to_string());
sg_p.insert("Description".to_string(), "to delete".to_string());
let sg_resp = p
.dispatch(&make_ctx("CreateSecurityGroup", sg_p))
.await
.unwrap();
let group_id = extract_tag(&body_str(&sg_resp), "groupId");

let mut del = HashMap::new();
del.insert("GroupId".to_string(), group_id.clone());
let resp = p
.dispatch(&make_ctx("DeleteSecurityGroup", del))
.await
.unwrap();
assert_eq!(resp.status_code, 200);

let list = p
.dispatch(&make_ctx("DescribeSecurityGroups", HashMap::new()))
.await
.unwrap();
assert!(!body_str(&list).contains(&group_id));
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion | 🟠 Major | ⚡ Quick win

Missing error path test for DeleteSecurityGroup.

Per coding guidelines, every operation must cover the primary error path. Add a test for deleting a non-existent security group.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/services/ec2/tests/ec2_tests.rs` around lines 366 - 391, Add a new
test that exercises the error path for DeleteSecurityGroup by attempting to
delete a non-existent GroupId: create a new async test (e.g.,
test_delete_security_group_not_found) that instantiates Ec2Provider::new(),
calls p.dispatch(&make_ctx("DeleteSecurityGroup", params)) with a bogus GroupId
(not created earlier) and then asserts the provider returns the expected error
outcome (e.g., response status_code != 200 and body_str contains the AWS-style
error like "InvalidGroup.NotFound" or that dispatch returned an Err). Reuse
make_ctx, body_str, and Ec2Provider::dispatch as in test_delete_security_group
and keep assertions specific to DeleteSecurityGroup's error signal.

Source: Coding guidelines

Comment on lines +718 to +737
#[tokio::test]
async fn test_delete_volume() {
let p = Ec2Provider::new();
let vol_resp = p
.dispatch(&make_ctx("CreateVolume", HashMap::new()))
.await
.unwrap();
let volume_id = extract_tag(&body_str(&vol_resp), "volumeId");

let mut del = HashMap::new();
del.insert("VolumeId".to_string(), volume_id.clone());
let resp = p.dispatch(&make_ctx("DeleteVolume", del)).await.unwrap();
assert_eq!(resp.status_code, 200);

let desc = p
.dispatch(&make_ctx("DescribeVolumes", HashMap::new()))
.await
.unwrap();
assert!(!body_str(&desc).contains(&volume_id));
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion | 🟠 Major | ⚡ Quick win

Missing error path test for DeleteVolume.

Per coding guidelines, every operation must cover the primary error path. Add a test for deleting a non-existent volume.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/services/ec2/tests/ec2_tests.rs` around lines 718 - 737, Add a new
test function (e.g., test_delete_volume_not_found) alongside test_delete_volume
that constructs an Ec2Provider via Ec2Provider::new(), calls
p.dispatch(&make_ctx("DeleteVolume", del)) with a HashMap where "VolumeId" is a
non-existent id (e.g., "vol-00000000"), awaits the result, and asserts the error
path: check resp.status_code is not 200 and that body_str(&resp) contains the
expected error identifier (e.g., "InvalidVolume.NotFound" or other
service-specific not-found string); also keep the existing DescribeVolumes check
separate — reference functions/methods: test_delete_volume, Ec2Provider::new,
p.dispatch, make_ctx, "DeleteVolume", and body_str.

Source: Coding guidelines

Comment thread crates/services/ec2/tests/ec2_tests.rs
Comment on lines +235 to +265
#[tokio::test]
async fn test_create_and_describe_replication_group() {
let svc = ElastiCacheProvider::new();
let resp = svc
.dispatch(&make_ctx(
"CreateReplicationGroup",
[
p("ReplicationGroupId", "my-rg"),
p("ReplicationGroupDescription", "Test RG"),
p("NumCacheClusters", "2"),
p("AutomaticFailoverEnabled", "true"),
p("CacheNodeType", "cache.t3.micro"),
]
.into(),
))
.await
.unwrap();
assert_eq!(resp.status_code, 200);
let body = body_str(&resp);
assert!(body.contains("CreateReplicationGroupResponse"));
assert!(body.contains("<ReplicationGroupId>my-rg</ReplicationGroupId>"));
assert!(body.contains("<AutomaticFailover>enabled</AutomaticFailover>"));
assert!(body.contains("<NumCacheClusters>2</NumCacheClusters>"));

let desc = svc
.dispatch(&make_ctx("DescribeReplicationGroups", HashMap::new()))
.await
.unwrap();
assert_eq!(desc.status_code, 200);
assert!(body_str(&desc).contains("my-rg"));
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion | 🟠 Major | ⚡ Quick win

Missing error path test for CreateReplicationGroup.

Per coding guidelines, every operation must cover at least one error path. Consider adding a test for duplicate replication group creation (similar to the cache cluster duplicate test pattern).

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/services/elasticache/tests/elasticache_tests.rs` around lines 235 -
265, The test test_create_and_describe_replication_group only covers the success
path; add an error-path test that attempts to create a duplicate replication
group (call ElastiCacheProvider::new() and dispatch CreateReplicationGroup twice
using make_ctx with the same "ReplicationGroupId" like "my-rg"), then assert the
second response is an error (non-200 status_code) and its body contains the
expected duplicate error text (mirror the duplicate cache cluster test pattern:
check error code/message for an already-existing replication group). Ensure you
use the same helper functions (make_ctx, p, body_str, and
ElastiCacheProvider::dispatch) so the new assertions align with existing tests.

Source: Coding guidelines

Comment on lines +371 to +400
#[tokio::test]
async fn test_create_and_describe_subnet_group() {
let svc = ElastiCacheProvider::new();
let resp = svc
.dispatch(&make_ctx(
"CreateCacheSubnetGroup",
[
p("CacheSubnetGroupName", "my-sg"),
p("CacheSubnetGroupDescription", "test"),
p("VpcId", "vpc-12345"),
p("SubnetIds.SubnetIdentifier.1", "subnet-aaa"),
p("SubnetIds.SubnetIdentifier.2", "subnet-bbb"),
]
.into(),
))
.await
.unwrap();
assert_eq!(resp.status_code, 200);
let body = body_str(&resp);
assert!(body.contains("<CacheSubnetGroupName>my-sg</CacheSubnetGroupName>"));

let desc = svc
.dispatch(&make_ctx("DescribeCacheSubnetGroups", HashMap::new()))
.await
.unwrap();
let desc_body = body_str(&desc);
assert!(desc_body.contains("my-sg"));
assert!(desc_body.contains("subnet-aaa"));
assert!(desc_body.contains("subnet-bbb"));
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion | 🟠 Major | ⚡ Quick win

Missing error path test for CreateCacheSubnetGroup.

Per coding guidelines, every operation must cover at least one error path. Consider adding a test for duplicate subnet group creation.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/services/elasticache/tests/elasticache_tests.rs` around lines 371 -
400, Add an error-path test that attempts to create the same CacheSubnetGroup
twice using ElastiCacheProvider::dispatch with the CreateCacheSubnetGroup
parameters (reuse test_create_and_describe_subnet_group setup), then assert the
second response is an error: verify non-200 status or that the body contains the
ElastiCache error identifier (e.g., "CacheSubnetGroupAlreadyExistsFault" or an
appropriate error message) and a descriptive message; ensure the test name
reflects duplicate creation (e.g., test_create_subnet_group_duplicate_error) and
uses the same request shape (CacheSubnetGroupName, SubnetIds.*) so it exercises
the duplicate-path in CreateCacheSubnetGroup handling.

Source: Coding guidelines

Comment on lines +432 to +465
#[tokio::test]
async fn test_create_and_describe_subnet_group() {
let p = RedshiftProvider::new();
let mut params = HashMap::new();
params.insert("ClusterSubnetGroupName".to_string(), "my-sg".to_string());
params.insert("Description".to_string(), "test subnet group".to_string());
params.insert("VpcId".to_string(), "vpc-12345".to_string());
params.insert(
"SubnetIds.SubnetIdentifier.1".to_string(),
"subnet-aaa".to_string(),
);
params.insert(
"SubnetIds.SubnetIdentifier.2".to_string(),
"subnet-bbb".to_string(),
);
let resp = p
.dispatch(&make_ctx("CreateClusterSubnetGroup", params))
.await
.unwrap();
assert_eq!(resp.status_code, 200);
let body = body_str(&resp);
assert!(body.contains("CreateClusterSubnetGroupResponse"));
assert!(body.contains("<ClusterSubnetGroupName>my-sg</ClusterSubnetGroupName>"));

let resp = p
.dispatch(&make_ctx("DescribeClusterSubnetGroups", HashMap::new()))
.await
.unwrap();
assert_eq!(resp.status_code, 200);
let body = body_str(&resp);
assert!(body.contains("my-sg"));
assert!(body.contains("subnet-aaa"));
assert!(body.contains("subnet-bbb"));
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Missing error path test for CreateClusterSubnetGroup duplicate.

The provider implementation at provider.rs:511 returns a ClusterSubnetGroupAlreadyExists error when attempting to create a duplicate subnet group, but no unit test covers this error path. The coding guidelines require at least one test covering the primary error path for each operation.

Add a test similar to test_create_snapshot_duplicate_fails that attempts to create the same subnet group twice and asserts the duplicate error response.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/services/redshift/tests/redshift_tests.rs` around lines 432 - 465, Add
a new async test (modeled after test_create_snapshot_duplicate_fails and located
near test_create_and_describe_subnet_group) that uses RedshiftProvider::new(),
calls dispatch with make_ctx("CreateClusterSubnetGroup", params) to create
"my-sg" once, then calls dispatch a second time with the same params and asserts
the response is an error for the duplicate path: check resp.status_code is the
expected error code (e.g., 400) and that body_str(&resp) contains
"ClusterSubnetGroupAlreadyExists" (or the exact error string returned by the
provider implementation); reuse the same param keys ("ClusterSubnetGroupName",
"Description", "VpcId", "SubnetIds.SubnetIdentifier.1",
"SubnetIds.SubnetIdentifier.2") as in test_create_and_describe_subnet_group to
reproduce the duplicate creation scenario.

Source: Coding guidelines

Comment on lines +508 to +534
#[tokio::test]
async fn test_create_and_describe_parameter_group() {
let p = RedshiftProvider::new();
let mut params = HashMap::new();
params.insert("ParameterGroupName".to_string(), "my-pg".to_string());
params.insert(
"ParameterGroupFamily".to_string(),
"redshift-1.0".to_string(),
);
params.insert("Description".to_string(), "test pg".to_string());
let resp = p
.dispatch(&make_ctx("CreateClusterParameterGroup", params))
.await
.unwrap();
assert_eq!(resp.status_code, 200);
let body = body_str(&resp);
assert!(body.contains("CreateClusterParameterGroupResponse"));
assert!(body.contains("<ParameterGroupName>my-pg</ParameterGroupName>"));

let resp = p
.dispatch(&make_ctx("DescribeClusterParameterGroups", HashMap::new()))
.await
.unwrap();
let body = body_str(&resp);
assert!(body.contains("my-pg"));
assert!(body.contains("redshift-1.0"));
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Missing error path test for CreateClusterParameterGroup duplicate.

The provider implementation at provider.rs:598 returns a ClusterParameterGroupAlreadyExists error when attempting to create a duplicate parameter group, but no unit test covers this error path. The coding guidelines require at least one test covering the primary error path for each operation.

Add a test similar to test_create_snapshot_duplicate_fails that attempts to create the same parameter group twice and asserts the duplicate error response.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/services/redshift/tests/redshift_tests.rs` around lines 508 - 534, Add
a new test that asserts the duplicate-create error path for
CreateClusterParameterGroup: call RedshiftProvider::dispatch with
make_ctx("CreateClusterParameterGroup", params) twice (same params as in
test_create_and_describe_parameter_group) and verify the second response returns
the ClusterParameterGroupAlreadyExists error (similar to
test_create_snapshot_duplicate_fails). Ensure the test name indicates duplicate
failure (e.g., test_create_parameter_group_duplicate_fails), awaits the dispatch
calls, and asserts the error response status/body matches the provider's
ClusterParameterGroupAlreadyExists output.

Source: Coding guidelines

Comment on lines +577 to +605
#[tokio::test]
async fn test_enable_and_disable_logging() {
let p = RedshiftProvider::new();
let mut params = HashMap::new();
params.insert("ClusterIdentifier".to_string(), "log-cluster".to_string());
p.dispatch(&make_ctx("CreateCluster", params))
.await
.unwrap();

let mut enable = HashMap::new();
enable.insert("ClusterIdentifier".to_string(), "log-cluster".to_string());
let resp = p
.dispatch(&make_ctx("EnableLogging", enable.clone()))
.await
.unwrap();
assert_eq!(resp.status_code, 200);
let body = body_str(&resp);
assert!(body.contains("EnableLoggingResponse"));
assert!(body.contains("<LoggingEnabled>true</LoggingEnabled>"));

let resp = p
.dispatch(&make_ctx("DisableLogging", enable))
.await
.unwrap();
assert_eq!(resp.status_code, 200);
let body = body_str(&resp);
assert!(body.contains("DisableLoggingResponse"));
assert!(body.contains("<LoggingEnabled>false</LoggingEnabled>"));
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Missing error path tests for EnableLogging and DisableLogging cluster not found.

Both operations return a ClusterNotFound error when the cluster does not exist (provider.rs:684 and provider.rs:713), but no unit tests cover these error paths. The coding guidelines require at least one test covering the primary error path for each operation.

Add two tests similar to test_reboot_cluster_not_found that attempt to enable/disable logging on a non-existent cluster and assert the ClusterNotFound error response.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/services/redshift/tests/redshift_tests.rs` around lines 577 - 605, Add
two unit tests that cover the ClusterNotFound error path for EnableLogging and
DisableLogging: create tests named like test_enable_logging_not_found and
test_disable_logging_not_found that instantiate RedshiftProvider, call dispatch
with make_ctx("EnableLogging", params) and make_ctx("DisableLogging", params)
where params contains a non-existent ClusterIdentifier (e.g., "no-cluster"),
await the response, and assert the response is an error with the ClusterNotFound
payload similar to test_reboot_cluster_not_found (use body_str(&resp) and assert
it contains "ClusterNotFound" and the appropriate XML error structure); mirror
the structure and assertions used in test_reboot_cluster_not_found so the
provider.rs error branches for EnableLogging and DisableLogging are exercised.

Source: Coding guidelines

Comment on lines +325 to +769
"GetParameterHistory" => {
let name = match str_param(ctx, "Name") {
Some(n) => n,
None => return Ok(json_error("ValidationException", "Name is required", 400)),
};
let Some(store) = self.store.get(account_id, region) else {
return Ok(json_error(
"ParameterNotFound",
&format!("Parameter {name} not found"),
400,
));
};
if !store.parameters.contains_key(&name) {
return Ok(json_error(
"ParameterNotFound",
&format!("Parameter {name} not found"),
400,
));
}
let history: Vec<Value> = store
.parameter_history
.get(&name)
.map(|versions| versions.iter().map(param_to_json).collect())
.unwrap_or_default();
Ok(json_ok(json!({ "Parameters": history })))
}

// ----------------------------------------------------------------
// CreateDocument
// ----------------------------------------------------------------
"CreateDocument" => {
let name = match str_param(ctx, "Name") {
Some(n) => n,
None => return Ok(json_error("ValidationException", "Name is required", 400)),
};
let content = match str_param(ctx, "Content") {
Some(c) => c,
None => {
return Ok(json_error(
"ValidationException",
"Content is required",
400,
));
}
};
let document_type =
str_param(ctx, "DocumentType").unwrap_or_else(|| "Command".to_string());
let document_format =
str_param(ctx, "DocumentFormat").unwrap_or_else(|| "JSON".to_string());

let mut store = self.store.get_or_create(account_id, region);
if store.documents.contains_key(&name) {
return Ok(json_error(
"DocumentAlreadyExists",
&format!("Document {name} already exists"),
400,
));
}
let doc = Document {
name: name.clone(),
document_type: document_type.clone(),
document_format: document_format.clone(),
schema_version: "2.2".to_string(),
status: "Active".to_string(),
content,
owner: account_id.clone(),
created: Utc::now(),
tags: Default::default(),
};
store.documents.insert(name.clone(), doc);
Ok(json_ok(json!({
"DocumentDescription": {
"Name": name,
"DocumentType": document_type,
"DocumentFormat": document_format,
"Status": "Active",
"SchemaVersion": "2.2",
"Owner": account_id,
}
})))
}

// ----------------------------------------------------------------
// DeleteDocument
// ----------------------------------------------------------------
"DeleteDocument" => {
let name = match str_param(ctx, "Name") {
Some(n) => n,
None => return Ok(json_error("ValidationException", "Name is required", 400)),
};
let mut store = self.store.get_or_create(account_id, region);
if store.documents.remove(&name).is_none() {
return Ok(json_error(
"InvalidDocument",
&format!("Document {name} does not exist"),
400,
));
}
Ok(json_ok(json!({})))
}

// ----------------------------------------------------------------
// DescribeDocument
// ----------------------------------------------------------------
"DescribeDocument" => {
let name = match str_param(ctx, "Name") {
Some(n) => n,
None => return Ok(json_error("ValidationException", "Name is required", 400)),
};
let Some(store) = self.store.get(account_id, region) else {
return Ok(json_error(
"InvalidDocument",
&format!("Document {name} does not exist"),
400,
));
};
match store.documents.get(&name) {
Some(doc) => Ok(json_ok(json!({
"Document": {
"Name": doc.name,
"DocumentType": doc.document_type,
"DocumentFormat": doc.document_format,
"SchemaVersion": doc.schema_version,
"Status": doc.status,
"Owner": doc.owner,
}
}))),
None => Ok(json_error(
"InvalidDocument",
&format!("Document {name} does not exist"),
400,
)),
}
}

// ----------------------------------------------------------------
// GetDocument
// ----------------------------------------------------------------
"GetDocument" => {
let name = match str_param(ctx, "Name") {
Some(n) => n,
None => return Ok(json_error("ValidationException", "Name is required", 400)),
};
let Some(store) = self.store.get(account_id, region) else {
return Ok(json_error(
"InvalidDocument",
&format!("Document {name} does not exist"),
400,
));
};
match store.documents.get(&name) {
Some(doc) => Ok(json_ok(json!({
"Name": doc.name,
"DocumentType": doc.document_type,
"DocumentFormat": doc.document_format,
"Content": doc.content,
"Status": doc.status,
}))),
None => Ok(json_error(
"InvalidDocument",
&format!("Document {name} does not exist"),
400,
)),
}
}

// ----------------------------------------------------------------
// ListDocuments
// ----------------------------------------------------------------
"ListDocuments" => {
let Some(store) = self.store.get(account_id, region) else {
return Ok(json_ok(json!({ "DocumentIdentifiers": [] })));
};
let docs: Vec<Value> = store
.documents
.values()
.map(|d| {
json!({
"Name": d.name,
"DocumentType": d.document_type,
"DocumentFormat": d.document_format,
"SchemaVersion": d.schema_version,
"Owner": d.owner,
})
})
.collect();
Ok(json_ok(json!({ "DocumentIdentifiers": docs })))
}

// ----------------------------------------------------------------
// SendCommand
// ----------------------------------------------------------------
"SendCommand" => {
let document_name = match str_param(ctx, "DocumentName") {
Some(n) => n,
None => {
return Ok(json_error(
"ValidationException",
"DocumentName is required",
400,
));
}
};
let instance_ids: Vec<String> = ctx
.request_body
.get("InstanceIds")
.and_then(|v| v.as_array())
.map(|arr| {
arr.iter()
.filter_map(|v| v.as_str().map(String::from))
.collect()
})
.unwrap_or_default();

let command_id = req_id();
let now = Utc::now();

let invocations: Vec<CommandInvocation> = instance_ids
.iter()
.map(|iid| CommandInvocation {
command_id: command_id.clone(),
instance_id: iid.clone(),
document_name: document_name.clone(),
status: "Success".to_string(),
status_details: "Success".to_string(),
output: String::new(),
response_code: 0,
})
.collect();

let command = Command {
command_id: command_id.clone(),
document_name: document_name.clone(),
status: "Success".to_string(),
requested_date: now,
instance_ids: instance_ids.clone(),
invocations,
};

let mut store = self.store.get_or_create(account_id, region);
store.commands.insert(command_id.clone(), command);

Ok(json_ok(json!({
"Command": {
"CommandId": command_id,
"DocumentName": document_name,
"Status": "Success",
"RequestedDateTime": now.to_rfc3339(),
"InstanceIds": instance_ids,
}
})))
}

// ----------------------------------------------------------------
// ListCommands
// ----------------------------------------------------------------
"ListCommands" => {
let command_id_filter = str_param(ctx, "CommandId");
let instance_id_filter = str_param(ctx, "InstanceId");

let Some(store) = self.store.get(account_id, region) else {
return Ok(json_ok(json!({ "Commands": [] })));
};

let commands: Vec<Value> = store
.commands
.values()
.filter(|c| {
command_id_filter
.as_deref()
.map(|id| c.command_id == id)
.unwrap_or(true)
})
.filter(|c| {
instance_id_filter
.as_deref()
.map(|id| c.instance_ids.contains(&id.to_string()))
.unwrap_or(true)
})
.map(|c| {
json!({
"CommandId": c.command_id,
"DocumentName": c.document_name,
"Status": c.status,
"RequestedDateTime": c.requested_date.to_rfc3339(),
"InstanceIds": c.instance_ids,
})
})
.collect();

Ok(json_ok(json!({ "Commands": commands })))
}

// ----------------------------------------------------------------
// GetCommandInvocation
// ----------------------------------------------------------------
"GetCommandInvocation" => {
let command_id = match str_param(ctx, "CommandId") {
Some(id) => id,
None => {
return Ok(json_error(
"ValidationException",
"CommandId is required",
400,
));
}
};
let instance_id = match str_param(ctx, "InstanceId") {
Some(id) => id,
None => {
return Ok(json_error(
"ValidationException",
"InstanceId is required",
400,
));
}
};

let Some(store) = self.store.get(account_id, region) else {
return Ok(json_error(
"InvocationDoesNotExist",
"Command invocation not found",
400,
));
};

let invocation = store.commands.get(&command_id).and_then(|cmd| {
cmd.invocations
.iter()
.find(|inv| inv.instance_id == instance_id)
});

match invocation {
Some(inv) => Ok(json_ok(json!({
"CommandId": inv.command_id,
"InstanceId": inv.instance_id,
"DocumentName": inv.document_name,
"Status": inv.status,
"StatusDetails": inv.status_details,
"StandardOutputContent": inv.output,
"ResponseCode": inv.response_code,
}))),
None => Ok(json_error(
"InvocationDoesNotExist",
"Command invocation not found",
400,
)),
}
}

// ----------------------------------------------------------------
// AddTagsToResource / ListTagsForResource / RemoveTagsFromResource
// ----------------------------------------------------------------
"AddTagsToResource" => {
let resource_type = match str_param(ctx, "ResourceType") {
Some(t) => t,
None => {
return Ok(json_error(
"ValidationException",
"ResourceType is required",
400,
));
}
};
let resource_id = match str_param(ctx, "ResourceId") {
Some(id) => id,
None => {
return Ok(json_error(
"ValidationException",
"ResourceId is required",
400,
));
}
};
let tag_list = ctx
.request_body
.get("Tags")
.and_then(|v| v.as_array())
.cloned()
.unwrap_or_default();

let mut store = self.store.get_or_create(account_id, region);
if resource_type == "Document"
&& let Some(doc) = store.documents.get_mut(&resource_id)
{
for tag in &tag_list {
if let (Some(k), Some(v)) = (
tag.get("Key").and_then(|v| v.as_str()),
tag.get("Value").and_then(|v| v.as_str()),
) {
doc.tags.insert(k.to_string(), v.to_string());
}
}
}
Ok(json_ok(json!({})))
}

"ListTagsForResource" => {
let resource_type = str_param(ctx, "ResourceType").unwrap_or_default();
let resource_id = str_param(ctx, "ResourceId").unwrap_or_default();

let tags: Vec<Value> = self
.store
.get(account_id, region)
.and_then(|store| {
if resource_type == "Document" {
store.documents.get(&resource_id).map(|doc| {
doc.tags
.iter()
.map(|(k, v)| json!({"Key": k, "Value": v}))
.collect()
})
} else {
None
}
})
.unwrap_or_default();

Ok(json_ok(json!({ "TagList": tags })))
}

"RemoveTagsFromResource" => {
let resource_type = str_param(ctx, "ResourceType").unwrap_or_default();
let resource_id = str_param(ctx, "ResourceId").unwrap_or_default();
let tag_keys: Vec<String> = ctx
.request_body
.get("TagKeys")
.and_then(|v| v.as_array())
.map(|arr| {
arr.iter()
.filter_map(|v| v.as_str().map(String::from))
.collect()
})
.unwrap_or_default();

let mut store = self.store.get_or_create(account_id, region);
if resource_type == "Document"
&& let Some(doc) = store.documents.get_mut(&resource_id)
{
for k in &tag_keys {
doc.tags.remove(k);
}
}
Ok(json_ok(json!({})))
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
echo "=== New operations (should be 12) ==="
echo "GetParameterHistory CreateDocument DeleteDocument DescribeDocument GetDocument"
echo "ListDocuments SendCommand ListCommands GetCommandInvocation"
echo "AddTagsToResource ListTagsForResource RemoveTagsFromResource"

echo ""
echo "=== Unit tests in ssm_tests.rs ==="
rg -n 'async fn test_' crates/services/ssm/tests/ssm_tests.rs | rg -i 'document|command|invocation|history|tags'

echo ""
echo "=== Perf tests in ssm_perf_tests.rs ==="
rg -n 'async fn perf_' crates/services/ssm/tests/ssm_perf_tests.rs

echo ""
echo "=== Parity scenarios for SSM ==="
fd -e json . tests/parity/scenarios/ --exec rg -l 'ssm|SSM' {}

Repository: JesseKoldewijn/openstack

Length of output: 1197


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "=== List SSM test files ==="
ls -la crates/services/ssm/tests || true

echo
echo "=== Inspect ssm_tests.rs for coverage of required operations ==="
sed -n '1,220p' crates/services/ssm/tests/ssm_tests.rs | cat
echo
rg -n "async fn test_" crates/services/ssm/tests/ssm_tests.rs | cat
echo
rg -n "test_(add_tags|list_tags|remove_tags|.*document.*|.*command.*|.*invocation.*|.*history.*)" crates/services/ssm/tests/ssm_tests.rs || true

echo
echo "=== Inspect ssm_perf_tests.rs for coverage ==="
ls -la crates/services/ssm/tests/ssm_perf_tests.rs || true
rg -n "async fn perf_" crates/services/ssm/tests/ssm_perf_tests.rs | cat
echo
sed -n '1,240p' crates/services/ssm/tests/ssm_perf_tests.rs | cat

echo
echo "=== Search parity scenarios for SSM operations ==="
# Show any parity scenario JSON mentioning SSM operations / action names / service keyword
fd -e json . tests/parity/scenarios/ --exec rg -n -i "(ssm|GetParameterHistory|CreateDocument|DeleteDocument|DescribeDocument|GetDocument|ListDocuments|SendCommand|ListCommands|GetCommandInvocation|AddTagsToResource|ListTagsForResource|RemoveTagsFromResource)" {} \; || true

echo
echo "=== Count/locate parity scenarios that mention SSM ==="
fd -e json . tests/parity/scenarios/ --exec sh -c 'echo "$0"; rg -i -n "ssm" "$0" >/dev/null && echo "  matches ssm"' {} || true

Repository: JesseKoldewijn/openstack

Length of output: 13419


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "=== Check for tag-related unit tests ==="
rg -n -i "add_tags_to_resource|list_tags_for_resource|remove_tags_from_resource|tag(s)?_for_resource|tags" crates/services/ssm/tests/ssm_tests.rs || true
echo

echo "=== Check ssm_perf_tests.rs for doc/command/history/tag perf tests ==="
rg -n -i "document|command|invocation|history|tag(s)?" crates/services/ssm/tests/ssm_perf_tests.rs || true
echo

echo "=== Check parity core/extended for SSM operations ==="
for f in tests/parity/scenarios/core.json tests/parity/scenarios/extended.json; do
  echo "--- $f ---"
  rg -n -i '"ssm"|"get-parameter-history"|"create-document"|"delete-document"|"describe-document"|"get-document"|"list-documents"|"send-command"|"list-commands"|"get-command-invocation"|"add-tags-to-resource"|"list-tags-for-resource"|"remove-tags-from-resource"|' "$f" || true
done

echo
echo "=== Show any parity scenario lines for SSM in all-services-smoke.json ==="
rg -n -i "ssm|get-parameter" tests/parity/scenarios/all-services-smoke.json || true

Repository: JesseKoldewijn/openstack

Length of output: 24987


🏁 Script executed:

#!/bin/bash
set -euo pipefail

file="crates/services/ssm/tests/ssm_tests.rs"

echo "=== Extract relevant test bodies (line ranges around each test) ==="

# Helper: print from start line to just before next test_async
print_test() {
  local name="$1"
  local start
  start=$(rg -n "async fn ${name}\b" "$file" | head -n1 | cut -d: -f1)
  if [[ -z "${start:-}" ]]; then
    echo "MISSING: ${name}"
    return 0
  fi
  # Grab next ~120 lines; should include body for these tests
  echo
  echo "----- ${name} (around line ${start}) -----"
  sed -n "$start,$((start+140))p" "$file" | cat
}

print_test "test_get_parameter_history"
print_test "test_get_parameter_history_not_found"

print_test "test_create_and_describe_document"
print_test "test_create_document_duplicate_fails"
print_test "test_get_document"
print_test "test_list_documents"
print_test "test_delete_document"

print_test "test_send_command"
print_test "test_list_commands"
print_test "test_list_commands_filter_by_instance"

print_test "test_get_command_invocation"
print_test "test_get_command_invocation_not_found"

# Tags tests (if they exist)
rg -n "test_(add_tags|list_tags|remove_tags)" "$file" || true

Repository: JesseKoldewijn/openstack

Length of output: 38683


SSM provider: missing required parity/unit/perf coverage for new operations

  • crates/services/ssm/tests/ssm_tests.rs has unit tests for GetParameterHistory and most document/command operations (including an error case for CreateDocument duplicate and GetCommandInvocation not-found), but there’s no explicit error-path test for DeleteDocument, and there are no unit tests for AddTagsToResource / ListTagsForResource / RemoveTagsFromResource.
  • crates/services/ssm/tests/ssm_perf_tests.rs only benchmarks parameter-store operations; there are no perf tests (with std::time::Instant) for GetParameterHistory, any document/command operations, or any tag operations.
  • tests/parity/scenarios/ only contains an SSM probe for get-parameter in all-services-smoke.json; core.json and extended.json contain no parity scenarios for the required new SSM operations.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/services/ssm/src/provider.rs` around lines 325 - 769, The PR is
missing unit, perf, and parity coverage for several new SSM operations; add unit
tests in crates/services/ssm/tests/ssm_tests.rs that exercise the DeleteDocument
error path (DeleteDocument when document missing),
AddTagsToResource/ListTagsForResource/RemoveTagsFromResource (verify tags
added/listed/removed on Document), and ensure CreateDocument duplicate error and
GetCommandInvocation not-found remain covered; add performance benchmarks in
crates/services/ssm/tests/ssm_perf_tests.rs using std::time::Instant for
GetParameterHistory, a representative document flow (CreateDocument,
SendCommand, ListCommands, GetCommandInvocation), and tag operations
(Add/List/RemoveTagsFromResource); finally, add parity scenario entries under
tests/parity/scenarios (core/extended/all-services) to include probes for
get-parameter-history, create/delete/describe/get/list documents, send/list/get
commands, and tag operations so CI parity tests cover them. Ensure tests
reference the handler operation names exactly (GetParameterHistory,
CreateDocument, DeleteDocument, DescribeDocument, GetDocument, ListDocuments,
SendCommand, ListCommands, GetCommandInvocation, AddTagsToResource,
ListTagsForResource, RemoveTagsFromResource) so they exercise the code paths
shown.

Source: Coding guidelines

Comment on lines +174 to +243
#[tokio::test]
async fn test_assume_role_with_web_identity() {
let p = StsProvider::new();
let resp = p
.dispatch(&make_ctx(
"AssumeRoleWithWebIdentity",
&[
("RoleArn", "arn:aws:iam::000000000000:role/WebRole"),
("RoleSessionName", "web-session"),
("WebIdentityToken", "fake-web-token-12345"),
("ProviderId", "cognito-identity.amazonaws.com"),
],
))
.await
.unwrap();
assert_eq!(resp.status_code, 200);
let body = body_str(&resp);
assert!(body.contains("AssumeRoleWithWebIdentityResponse"));
assert!(body.contains("<AccessKeyId>"));
assert!(body.contains("<SessionToken>"));
assert!(body.contains("<Expiration>"));
assert!(body.contains("<SubjectFromWebIdentityToken>"));
assert!(body.contains("<Provider>cognito-identity.amazonaws.com</Provider>"));
assert!(body.contains("arn:aws:sts::000000000000:assumed-role/WebRole/web-session"));
}

#[tokio::test]
async fn test_assume_role_with_web_identity_defaults() {
let p = StsProvider::new();
let resp = p
.dispatch(&make_ctx("AssumeRoleWithWebIdentity", &[]))
.await
.unwrap();
assert_eq!(resp.status_code, 200);
let body = body_str(&resp);
assert!(body.contains("AssumeRoleWithWebIdentityResponse"));
assert!(body.contains("<AccessKeyId>"));
}

// ---------------------------------------------------------------------------
// AssumeRoleWithSAML
// ---------------------------------------------------------------------------

#[tokio::test]
async fn test_assume_role_with_saml() {
let p = StsProvider::new();
let resp = p
.dispatch(&make_ctx(
"AssumeRoleWithSAML",
&[
("RoleArn", "arn:aws:iam::000000000000:role/SamlRole"),
(
"PrincipalArn",
"arn:aws:iam::000000000000:saml-provider/MySAML",
),
("SAMLAssertion", "PHNhbWw+ZmFrZTwvc2FtbD4="),
],
))
.await
.unwrap();
assert_eq!(resp.status_code, 200);
let body = body_str(&resp);
assert!(body.contains("AssumeRoleWithSAMLResponse"));
assert!(body.contains("<AccessKeyId>"));
assert!(body.contains("<SessionToken>"));
assert!(body.contains("<Expiration>"));
assert!(body.contains("<Issuer>"));
assert!(body.contains("arn:aws:sts::000000000000:assumed-role/SamlRole/saml-session"));
assert!(body.contains("<SubjectType>persistent</SubjectType>"));
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Add error path tests for AssumeRoleWithWebIdentity and AssumeRoleWithSAML.

The coding guidelines mandate that every operation must have at least one test covering the happy path and one covering the primary error path. The three new tests added here (test_assume_role_with_web_identity, test_assume_role_with_web_identity_defaults, test_assume_role_with_saml) only cover happy paths. Please add error path tests for common failure scenarios such as:

  • Invalid or missing RoleArn
  • Invalid or missing WebIdentityToken / SAMLAssertion
  • Malformed parameters

These error tests should assert that the provider returns appropriate error responses (status 400 or similar) with meaningful error codes.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/services/sts/tests/sts_tests.rs` around lines 174 - 243, The tests add
happy-path coverage for AssumeRoleWithWebIdentity and AssumeRoleWithSAML but
lack error-path tests; add new tokio::test functions (e.g.,
test_assume_role_with_web_identity_error,
test_assume_role_with_web_identity_missing_token,
test_assume_role_with_saml_error) that use StsProvider::new() and
p.dispatch(&make_ctx(...)) with invalid/missing parameters (missing/invalid
RoleArn, missing/invalid WebIdentityToken or SAMLAssertion, and malformed
params) and assert the response.status_code is an error (400-range) and the body
contains the appropriate STS error code/message; reuse existing helpers body_str
and make_ctx to construct requests and assert specific error codes/messages
returned by the provider for the given failure scenarios.

Source: Coding guidelines

@JesseKoldewijn

Copy link
Copy Markdown
Owner Author

CodeRabbit blocker remediation

Addressed all blocker-scope inline findings in this push. Summary:

Fixed in this PR

  • Protocol/config: case-insensitive + underscore→hyphen service canonicalization (aws-protocol, config)
  • RDS: CreateDBSnapshot returns DBInstanceNotFound when source instance is missing (no synthetic defaults)
  • ElastiCache: delete/modify/reboot paths use get_mut instead of get_or_create to avoid empty store leaks
  • OpenSearch: DeleteDomain removes orphaned tag entries
  • Cognito: renamed callback_urls / logout_urls with AWS-compatible serde + response fields
  • EC2 tests: extract_tag fails fast when XML tags are missing
  • CI: pinned actions/checkout SHA + persist-credentials: false in benchmark.yml

Deferred (guideline debt — not CI blockers)

Tracked for follow-up PR test(coverage): add error-path tests for expanded service operations:

  • Missing error-path tests (~25 threads): EC2, RDS, ElastiCache, Route53, SES, STS, OpenSearch, Redshift subsets
  • Missing perf tests (2 threads): CloudTrail (10 ops), Redshift (11 new ops)
  • Refactors/nitpicks: CloudTrail/ECS inline json_error vs error module, unused uuid import, ARN helper extraction, unbounded store collections

Already addressed / false positives

  • ECS tests: ecs_tests.rs exists with 18 operation tests
  • Parity smoke: CloudTrail/Cognito/ECS/RDS/SSM probes are in all-services-smoke.json
  • SSM: 22 unit tests exist; only tag-resource ops lack dedicated coverage

Required CI was already green before these fixes; re-run expected to stay green.

JesseKoldewijn and others added 2 commits June 11, 2026 21:37
Normalize service name aliases for protocol routing and provider overrides,
correct RDS snapshot and ElastiCache store semantics, clean up OpenSearch
tags on domain delete, fix Cognito callback URL fields, harden EC2 test XML
parsing, and align benchmark workflow checkout hardening with CI.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@JesseKoldewijn

Copy link
Copy Markdown
Owner Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jun 11, 2026

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (11)
crates/services/cognito/src/provider.rs (3)

399-449: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Enforce UserPoolId on client-scoped lookups and deletes.

DeleteUserPoolClient and DescribeUserPoolClient only key by ClientId, so a client created in pool A can still be deleted or described while the caller passes pool B. Parse UserPoolId in both branches and reject when it does not match client.user_pool_id.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/services/cognito/src/provider.rs` around lines 399 - 449,
DeleteUserPoolClient and DescribeUserPoolClient currently identify clients only
by ClientId, allowing operations across user pools; parse the UserPoolId from
ctx in both handlers (same way str_param(ctx, "ClientId") is used), look up the
client via self.store.get/_or_create and then verify that the parsed UserPoolId
equals the client's user_pool_id before proceeding; if missing UserPoolId return
InvalidParameterException and if it doesn't match return
ResourceNotFoundException (or appropriate error) — update the
DeleteUserPoolClient and DescribeUserPoolClient branches to perform this
UserPoolId check against store.clients.get(&client_id) / removed client entry
and reject when client.user_pool_id != provided UserPoolId.

849-889: ⚠️ Potential issue | 🔴 Critical | ⚡ Quick win

Reject missing or empty passwords in AdminInitiateAuth.

Lines 855-863 coerce a missing PASSWORD to "", and Line 883 only rejects mismatches when the caller supplied a non-empty password. That means an enabled user can get tokens by omitting PASSWORD entirely. Fail the request before token issuance when PASSWORD is absent or empty.

Suggested fix
-                let password = auth_params
-                    .get("PASSWORD")
-                    .and_then(|v| v.as_str())
-                    .unwrap_or("")
-                    .to_string();
+                let Some(password) = auth_params
+                    .get("PASSWORD")
+                    .and_then(|v| v.as_str())
+                    .filter(|p| !p.is_empty())
+                else {
+                    return Ok(json_error(
+                        "NotAuthorizedException",
+                        "Incorrect username or password",
+                        400,
+                    ));
+                };
@@
-                        let stored_password = user.password.as_deref().unwrap_or("");
-                        if !password.is_empty() && stored_password != password {
+                        let stored_password = user.password.as_deref();
+                        if stored_password != Some(password) {
                             return Ok(json_error(
                                 "NotAuthorizedException",
                                 "Incorrect username or password",
                                 400,
                             ));
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/services/cognito/src/provider.rs` around lines 849 - 889, The
AdminInitiateAuth flow currently treats a missing or empty PASSWORD as "" and
only rejects when a non-empty password mismatches, allowing token issuance with
no password; update the auth logic in the AdminInitiateAuth handling (where
auth_params, username, password are extracted and where store.users.get(...) and
user.password are checked) to explicitly reject when password.is_empty() by
returning the same error used for bad credentials (e.g.,
NotAuthorizedException/400) before any stored_password comparison, while keeping
the existing enabled check and stored_password comparison for non-empty inputs.

207-218: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Avoid get_or_create on not-found error paths.

These branches create an account/region store before checking whether the pool, client, or user exists. A failing request against a fresh account/region therefore leaves behind an empty CognitoStore entry and can grow bundle state without bound. Use a read path first and only allocate store state in the actual create operations. This mirrors the empty-store leak pattern already called out elsewhere in this PR.

Also applies to: 278-292, 410-418, 591-600, 713-729, 757-770, 811-832

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/services/cognito/src/provider.rs` around lines 207 - 218, The code
currently calls self.store.get_or_create(account_id, region) in delete/read
paths (see usage around get_or_create, store.user_pools.remove,
store.clients.retain, store.users.retain), which allocates an empty CognitoStore
for missing account/region and leaks state; change these branches to first
perform a non-allocating lookup (e.g. use a read/get or get_mut variant that
does not create) to see if a store exists and only proceed to check/remove the
pool, clients, and users if the store is present; if the store or pool is
missing return the same ResourceNotFound json error, and only use get_or_create
in actual create operations.
crates/services/opensearch/src/provider.rs (2)

607-614: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Return the full ServiceSoftwareOptions shape on cancel as well.

StartServiceSoftwareUpdate includes NewVersion, but CancelServiceSoftwareUpdate drops it even though both responses are serializing the same persisted ServiceSoftwareOptions object. That inconsistent payload shape is easy to miss in tests and can break clients that deserialize one shared response model.

🛠️ Minimal fix
                         Ok(json_ok(json!({
                             "ServiceSoftwareOptions": {
                                 "CurrentVersion": options.current_version,
+                                "NewVersion": options.new_version,
                                 "UpdateAvailable": options.update_available,
                                 "Cancellable": options.cancellable,
                                 "UpdateStatus": options.update_status,
                                 "Description": options.description,
                             }
                         })))
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/services/opensearch/src/provider.rs` around lines 607 - 614, The
cancel path returns a truncated ServiceSoftwareOptions JSON; update the
CancelServiceSoftwareUpdate response to return the full ServiceSoftwareOptions
shape (matching StartServiceSoftwareUpdate) by including the NewVersion field
from the persisted options object (use options.new_version or the existing field
name) alongside CurrentVersion, UpdateAvailable, Cancellable, UpdateStatus, and
Description so both handlers serialize the same shape.

444-515: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Validate the ARN against an existing domain before serving tag ops.

AddTags currently does store.tags.entry(arn).or_default() without proving that ARN belongs to a stored domain, and RemoveTags / ListTags likewise succeed against unknown ARNs. That creates orphan tag state that DeleteDomain can never clean up and masks the primary ResourceNotFoundException path these operations are expected to expose.

🛠️ Minimal fix sketch
+fn domain_exists_by_arn(store: &OpenSearchStore, arn: &str) -> bool {
+    store.domains.values().any(|domain| domain.arn == arn)
+}
+
             "AddTags" => {
                 let arn = match ctx.request_body.get("ARN").and_then(|v| v.as_str()) {
                     Some(a) => a.to_string(),
                     None => {
                         return Ok(json_error("ValidationException", "ARN required", 400));
                     }
                 };
@@
                 let mut store = self.store.get_or_create(account_id, region);
+                if !domain_exists_by_arn(&store, &arn) {
+                    return Ok(json_error(
+                        "ResourceNotFoundException",
+                        &format!("Domain not found for ARN: {arn}"),
+                        404,
+                    ));
+                }
                 let tag_map = store.tags.entry(arn).or_default();
@@
             "RemoveTags" => {
@@
                 let mut store = self.store.get_or_create(account_id, region);
-                if let Some(tag_map) = store.tags.get_mut(&arn) {
-                    for key in &tag_keys {
-                        tag_map.remove(key);
-                    }
-                }
+                if !domain_exists_by_arn(&store, &arn) {
+                    return Ok(json_error(
+                        "ResourceNotFoundException",
+                        &format!("Domain not found for ARN: {arn}"),
+                        404,
+                    ));
+                }
+                if let Some(tag_map) = store.tags.get_mut(&arn) {
+                    for key in &tag_keys {
+                        tag_map.remove(key);
+                    }
+                }
                 Ok(json_ok(json!({})))
             }
@@
             "ListTags" => {
@@
-                let tags = self
-                    .store
-                    .get(account_id, region)
-                    .and_then(|store| store.tags.get(&arn).cloned())
-                    .unwrap_or_default();
+                let Some(store) = self.store.get(account_id, region) else {
+                    return Ok(json_error(
+                        "ResourceNotFoundException",
+                        &format!("Domain not found for ARN: {arn}"),
+                        404,
+                    ));
+                };
+                if !domain_exists_by_arn(&store, &arn) {
+                    return Ok(json_error(
+                        "ResourceNotFoundException",
+                        &format!("Domain not found for ARN: {arn}"),
+                        404,
+                    ));
+                }
+                let tags = store.tags.get(&arn).cloned().unwrap_or_default();
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/services/opensearch/src/provider.rs` around lines 444 - 515, Add
validation to ensure the ARN belongs to an existing domain before mutating or
returning tags: for AddTags, RemoveTags, and ListTags stop using get_or_create
for the store or unconditional entry creation and instead call
self.store.get(account_id, region) (or check the store.domains map) to verify
the domain exists; if the domain/ARN is not found return
Ok(json_error("ResourceNotFoundException", "Domain not found", 404)) rather than
creating or returning empty tag state. Specifically update the AddTags handler
(currently using store.tags.entry(arn).or_default()), the RemoveTags handler
(which checks store.tags.get_mut(&arn)), and the ListTags handler (which reads
store.tags.get(&arn)) to first lookup the domain existence (e.g.,
store.domains.contains_key(&arn) or store.get(...).and_then(|s|
s.domains.get(&arn))) and only proceed to mutate/read tags if present.

Source: Coding guidelines

crates/services/opensearch/tests/opensearch_tests.rs (1)

724-739: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Add the missing error-path test for CancelServiceSoftwareUpdate.

provider.rs has an explicit 404 ResourceNotFoundException branch for cancel, but this file only exercises the happy path for cancel. Please mirror the existing start/not-found case for CancelServiceSoftwareUpdate, and assert the error message in the existing start test as well so the body-content requirement is covered.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/services/opensearch/tests/opensearch_tests.rs` around lines 724 - 739,
Add a failing-path test for CancelServiceSoftwareUpdate and strengthen the
StartServiceSoftwareUpdate not-found test: create a new async test similar to
test_start_service_software_update_domain_not_found that calls
OpenSearchProvider::dispatch with make_ctx("CancelServiceSoftwareUpdate",
json!({ "DomainName": "nonexistent" }),
"/2021-01-01/opensearch/serviceSoftwareUpdate/cancel", "POST"), assert
resp.status_code == 404 and that body_json(&resp)["code"] ==
"ResourceNotFoundException" (and optionally matching ["message"]), and update
the existing test_start_service_software_update_domain_not_found to also assert
the error message in body_json(&resp)["message"] so both body code and message
are validated; use the same helpers OpenSearchProvider, make_ctx, and body_json
to locate code.

Source: Coding guidelines

crates/services/rds/src/provider.rs (1)

420-434: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Avoid creating an empty store on the missing-instance error path.

Line 420 still calls get_or_create before validating DBInstanceIdentifier, so a failed CreateDBSnapshot against a brand-new account/region leaves behind an empty RdsStore. Repeating that with arbitrary scopes can grow AccountRegionBundle indefinitely even though every request fails.

🔧 Minimal fix
-                let mut store = self.store.get_or_create(account_id, region);
+                if self.store.get(account_id, region).is_none() {
+                    return Ok(xml_error(
+                        "DBInstanceNotFound",
+                        &format!("DB instance {db_id} not found"),
+                        404,
+                    ));
+                }
+                let mut store = self.store.get_or_create(account_id, region);
                 if store.snapshots.contains_key(&snapshot_id) {
                     return Ok(xml_error(
                         "DBSnapshotAlreadyExists",
                         &format!("Snapshot {snapshot_id} already exists"),
                         400,

Based on learnings from the PR summary, this same get_or_create-on-error-path leak pattern was already fixed in the ElastiCache handlers.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/services/rds/src/provider.rs` around lines 420 - 434, The handler
currently calls self.store.get_or_create(account_id, region) before validating
the DBInstanceIdentifier, which leaks empty RdsStore entries on failed
CreateDBSnapshot requests; change the flow to check for the instance existence
first (e.g., query the existing store with a non-creating lookup or inspect
self.store.get/account-region accessor) and only call get_or_create(account_id,
region) when you are about to actually mutate the store (e.g., when adding to
store.snapshots). Update the logic around get_or_create, store.snapshots, and
store.instances in the CreateDBSnapshot handling code so that get_or_create is
invoked after confirming the DB instance exists.
crates/services/elasticache/src/provider.rs (4)

726-733: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Reject subnet-group deletes while clusters still reference the group.

This branch drops the subnet group without checking store.clusters, so any cluster created with CacheSubnetGroupName is left pointing at a deleted resource. That breaks the store’s internal contract and makes later describes return dangling references.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/services/elasticache/src/provider.rs` around lines 726 - 733, The
DeleteCacheSubnetGroup branch currently removes store.subnet_groups without
validating references; update the handler (the code around
store.subnet_groups.remove and the DeleteCacheSubnetGroup response) to first
scan store.clusters for any cluster whose CacheSubnetGroupName (or equivalent
field) equals the subnet group name and, if any are found, return an xml_error
indicating the group is in use (do not remove it); only call
store.subnet_groups.remove(&name) and return Ok(xml_resp(...)) when no clusters
reference that subnet group.

59-63: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Escape XML error fields before writing them into the response.

message is built from request parameters in the not-found/error branches, so IDs containing < or & will produce malformed XML and reflected markup in the fault payload.

🐛 Proposed fix
 fn xml_error(code: &str, message: &str, status: u16) -> DispatchResponse {
     let xml = format!(
         "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\
 <ErrorResponse xmlns=\"{EC_NS}\">\
-<Error><Code>{code}</Code><Message>{message}</Message></Error>\
-</ErrorResponse>"
+<Error><Code>{}</Code><Message>{}</Message></Error>\
+</ErrorResponse>",
+        xml_escape(code),
+        xml_escape(message),
     );
     DispatchResponse {
         status_code: status,
         body: ResponseBody::Buffered(Bytes::from(xml.into_bytes())),
         content_type: Cow::Borrowed("text/xml"),
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/services/elasticache/src/provider.rs` around lines 59 - 63, The
xml_error function writes unescaped code and message into the XML payload;
escape XML entities for the code and message before inserting them into the
format string to prevent malformed XML or injected markup. Update xml_error to
run code and message through an XML-escaping helper (e.g., an existing escape
utility or a small helper that replaces &, <, >, ", ') and then use those
escaped values in the format! call so the generated ErrorResponse is always
well-formed.

311-315: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Keep the replication group in sync when deleting a member cluster.

If this cluster belongs to a replication group, the delete path removes the cluster but leaves member_clusters and num_cache_clusters stale on the owning ReplicationGroup. After that, DescribeReplicationGroups can advertise members that no longer exist.

🛠️ Suggested direction
                 match store.clusters.remove(&cluster_id) {
                     Some(c) => {
+                        if let Some(rg_id) = &c.replication_group_id {
+                            if let Some(rg) = store.replication_groups.get_mut(rg_id) {
+                                rg.member_clusters.retain(|id| id != &cluster_id);
+                                rg.num_cache_clusters = rg.member_clusters.len() as u32;
+                            }
+                        }
                         let inner =
                             format!("<CacheCluster>{}</CacheCluster>", cluster_xml(&c, region));
                         Ok(xml_resp("DeleteCacheCluster", &rid, &inner))
                     }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/services/elasticache/src/provider.rs` around lines 311 - 315, When
removing a cluster via store.clusters.remove(&cluster_id) also locate its owning
replication group (use the cluster's replication_group_id / ReplicationGroup
id), remove the cluster_id from ReplicationGroup.member_clusters and adjust
ReplicationGroup.num_cache_clusters accordingly, updating the
store.replication_groups entry in place so subsequent DescribeReplicationGroups
reflects the change; ensure you handle the case where the replication group
becomes empty and persist the modified ReplicationGroup back into the store.

293-323: ⚠️ Potential issue | 🟠 Major | 🏗️ Heavy lift

Add the required parity, unit, and perf coverage for these modified arms.

These operations were modified, and the supplied PR notes still call out remaining ElastiCache error-path test debt. Per the provider guideline, each modified arm needs a parity scenario, a happy-path + primary-error-path unit test, and a perf test entry with an Instant timing assertion before this is merge-ready.

As per coding guidelines, crates/services/*/src/provider.rs: “Every new or modified operation match arm in a service provider MUST have” parity coverage, a happy-path and error-path unit test, and a performance regression test; the PR summary says there are still “Missing error-path tests … across … ElastiCache … subsets.”

Also applies to: 350-388, 393-426, 541-577, 608-650, 708-734

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/services/elasticache/src/provider.rs` around lines 293 - 323, The
DeleteCacheCluster match arm in provider.rs (symbols: DeleteCacheCluster,
store.clusters.remove, cluster_xml, xml_resp, xml_error) was modified and needs
full test coverage: add a parity integration scenario, a unit test for the
happy-path (ensure removal returns the expected xml_resp with cluster_xml
content), a unit test for the primary error-path (missing CacheClusterId and
cluster-not-found cases asserting xml_error is returned), and a perf regression
test that invokes the DeleteCacheCluster code path and asserts an Instant timing
bound; apply the same pattern for the other modified ElastiCache arms in this
file (ensure each new/modified operation match arm has parity, happy-path +
error-path unit tests, and a perf test with an Instant assertion).

Source: Coding guidelines

♻️ Duplicate comments (1)
.github/workflows/benchmark.yml (1)

244-244: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Pin remaining GitHub Actions to commit SHAs.

actions/upload-artifact@v4 and actions/download-artifact@v4 are still tag-pinned instead of SHA-pinned, which leaves mutable supply-chain exposure in CI.

#!/bin/bash
# Read-only verification: list all non-SHA-pinned uses entries in this workflow
rg -n '^\s*uses:\s*[^#\s]+@v[0-9]+' .github/workflows/benchmark.yml

Also applies to: 270-270, 373-373

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/benchmark.yml at line 244, The workflow uses mutable tags
for GitHub Actions (e.g., actions/upload-artifact@v4 and
actions/download-artifact@v4); replace those tag-pinned references with the
exact commit SHAs for the corresponding action releases (e.g.,
actions/upload-artifact@<commit-sha>, actions/download-artifact@<commit-sha>) so
the workflow is SHA-pinned; update every occurrence of these symbols in the file
(all uses of actions/upload-artifact@v4 and actions/download-artifact@v4) and
verify the SHAs by fetching them from the official action repositories'
tags/releases.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@crates/services/cognito/src/provider.rs`:
- Around line 399-449: DeleteUserPoolClient and DescribeUserPoolClient currently
identify clients only by ClientId, allowing operations across user pools; parse
the UserPoolId from ctx in both handlers (same way str_param(ctx, "ClientId") is
used), look up the client via self.store.get/_or_create and then verify that the
parsed UserPoolId equals the client's user_pool_id before proceeding; if missing
UserPoolId return InvalidParameterException and if it doesn't match return
ResourceNotFoundException (or appropriate error) — update the
DeleteUserPoolClient and DescribeUserPoolClient branches to perform this
UserPoolId check against store.clients.get(&client_id) / removed client entry
and reject when client.user_pool_id != provided UserPoolId.
- Around line 849-889: The AdminInitiateAuth flow currently treats a missing or
empty PASSWORD as "" and only rejects when a non-empty password mismatches,
allowing token issuance with no password; update the auth logic in the
AdminInitiateAuth handling (where auth_params, username, password are extracted
and where store.users.get(...) and user.password are checked) to explicitly
reject when password.is_empty() by returning the same error used for bad
credentials (e.g., NotAuthorizedException/400) before any stored_password
comparison, while keeping the existing enabled check and stored_password
comparison for non-empty inputs.
- Around line 207-218: The code currently calls
self.store.get_or_create(account_id, region) in delete/read paths (see usage
around get_or_create, store.user_pools.remove, store.clients.retain,
store.users.retain), which allocates an empty CognitoStore for missing
account/region and leaks state; change these branches to first perform a
non-allocating lookup (e.g. use a read/get or get_mut variant that does not
create) to see if a store exists and only proceed to check/remove the pool,
clients, and users if the store is present; if the store or pool is missing
return the same ResourceNotFound json error, and only use get_or_create in
actual create operations.

In `@crates/services/elasticache/src/provider.rs`:
- Around line 726-733: The DeleteCacheSubnetGroup branch currently removes
store.subnet_groups without validating references; update the handler (the code
around store.subnet_groups.remove and the DeleteCacheSubnetGroup response) to
first scan store.clusters for any cluster whose CacheSubnetGroupName (or
equivalent field) equals the subnet group name and, if any are found, return an
xml_error indicating the group is in use (do not remove it); only call
store.subnet_groups.remove(&name) and return Ok(xml_resp(...)) when no clusters
reference that subnet group.
- Around line 59-63: The xml_error function writes unescaped code and message
into the XML payload; escape XML entities for the code and message before
inserting them into the format string to prevent malformed XML or injected
markup. Update xml_error to run code and message through an XML-escaping helper
(e.g., an existing escape utility or a small helper that replaces &, <, >, ", ')
and then use those escaped values in the format! call so the generated
ErrorResponse is always well-formed.
- Around line 311-315: When removing a cluster via
store.clusters.remove(&cluster_id) also locate its owning replication group (use
the cluster's replication_group_id / ReplicationGroup id), remove the cluster_id
from ReplicationGroup.member_clusters and adjust
ReplicationGroup.num_cache_clusters accordingly, updating the
store.replication_groups entry in place so subsequent DescribeReplicationGroups
reflects the change; ensure you handle the case where the replication group
becomes empty and persist the modified ReplicationGroup back into the store.
- Around line 293-323: The DeleteCacheCluster match arm in provider.rs (symbols:
DeleteCacheCluster, store.clusters.remove, cluster_xml, xml_resp, xml_error) was
modified and needs full test coverage: add a parity integration scenario, a unit
test for the happy-path (ensure removal returns the expected xml_resp with
cluster_xml content), a unit test for the primary error-path (missing
CacheClusterId and cluster-not-found cases asserting xml_error is returned), and
a perf regression test that invokes the DeleteCacheCluster code path and asserts
an Instant timing bound; apply the same pattern for the other modified
ElastiCache arms in this file (ensure each new/modified operation match arm has
parity, happy-path + error-path unit tests, and a perf test with an Instant
assertion).

In `@crates/services/opensearch/src/provider.rs`:
- Around line 607-614: The cancel path returns a truncated
ServiceSoftwareOptions JSON; update the CancelServiceSoftwareUpdate response to
return the full ServiceSoftwareOptions shape (matching
StartServiceSoftwareUpdate) by including the NewVersion field from the persisted
options object (use options.new_version or the existing field name) alongside
CurrentVersion, UpdateAvailable, Cancellable, UpdateStatus, and Description so
both handlers serialize the same shape.
- Around line 444-515: Add validation to ensure the ARN belongs to an existing
domain before mutating or returning tags: for AddTags, RemoveTags, and ListTags
stop using get_or_create for the store or unconditional entry creation and
instead call self.store.get(account_id, region) (or check the store.domains map)
to verify the domain exists; if the domain/ARN is not found return
Ok(json_error("ResourceNotFoundException", "Domain not found", 404)) rather than
creating or returning empty tag state. Specifically update the AddTags handler
(currently using store.tags.entry(arn).or_default()), the RemoveTags handler
(which checks store.tags.get_mut(&arn)), and the ListTags handler (which reads
store.tags.get(&arn)) to first lookup the domain existence (e.g.,
store.domains.contains_key(&arn) or store.get(...).and_then(|s|
s.domains.get(&arn))) and only proceed to mutate/read tags if present.

In `@crates/services/opensearch/tests/opensearch_tests.rs`:
- Around line 724-739: Add a failing-path test for CancelServiceSoftwareUpdate
and strengthen the StartServiceSoftwareUpdate not-found test: create a new async
test similar to test_start_service_software_update_domain_not_found that calls
OpenSearchProvider::dispatch with make_ctx("CancelServiceSoftwareUpdate",
json!({ "DomainName": "nonexistent" }),
"/2021-01-01/opensearch/serviceSoftwareUpdate/cancel", "POST"), assert
resp.status_code == 404 and that body_json(&resp)["code"] ==
"ResourceNotFoundException" (and optionally matching ["message"]), and update
the existing test_start_service_software_update_domain_not_found to also assert
the error message in body_json(&resp)["message"] so both body code and message
are validated; use the same helpers OpenSearchProvider, make_ctx, and body_json
to locate code.

In `@crates/services/rds/src/provider.rs`:
- Around line 420-434: The handler currently calls
self.store.get_or_create(account_id, region) before validating the
DBInstanceIdentifier, which leaks empty RdsStore entries on failed
CreateDBSnapshot requests; change the flow to check for the instance existence
first (e.g., query the existing store with a non-creating lookup or inspect
self.store.get/account-region accessor) and only call get_or_create(account_id,
region) when you are about to actually mutate the store (e.g., when adding to
store.snapshots). Update the logic around get_or_create, store.snapshots, and
store.instances in the CreateDBSnapshot handling code so that get_or_create is
invoked after confirming the DB instance exists.

---

Duplicate comments:
In @.github/workflows/benchmark.yml:
- Line 244: The workflow uses mutable tags for GitHub Actions (e.g.,
actions/upload-artifact@v4 and actions/download-artifact@v4); replace those
tag-pinned references with the exact commit SHAs for the corresponding action
releases (e.g., actions/upload-artifact@<commit-sha>,
actions/download-artifact@<commit-sha>) so the workflow is SHA-pinned; update
every occurrence of these symbols in the file (all uses of
actions/upload-artifact@v4 and actions/download-artifact@v4) and verify the SHAs
by fetching them from the official action repositories' tags/releases.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 689a6177-cbe0-4876-bb88-33396063f085

📥 Commits

Reviewing files that changed from the base of the PR and between 382659f and 377c853.

📒 Files selected for processing (13)
  • .github/workflows/benchmark.yml
  • crates/aws-protocol/src/protocol.rs
  • crates/aws-protocol/tests/protocol_integration.rs
  • crates/config/src/services.rs
  • crates/services/cognito/src/provider.rs
  • crates/services/cognito/src/store.rs
  • crates/services/ec2/tests/ec2_perf_tests.rs
  • crates/services/ec2/tests/ec2_tests.rs
  • crates/services/elasticache/src/provider.rs
  • crates/services/opensearch/src/provider.rs
  • crates/services/opensearch/tests/opensearch_tests.rs
  • crates/services/rds/src/provider.rs
  • crates/services/rds/tests/rds_tests.rs

JesseKoldewijn and others added 2 commits June 11, 2026 23:38
Harden Cognito auth and delete paths, validate OpenSearch tag ARNs,
fix ElastiCache delete side effects, and pin benchmark artifact actions.

Co-authored-by: Cursor <cursoragent@cursor.com>
Collapse nested if in DeleteCacheCluster and move s3 test module to end
of provider.rs to satisfy items-after-test-module lint.

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant