Skip to content

ci: enforce cargo check warning gate#69

Merged
JesseKoldewijn merged 11 commits into
developfrom
feat/ci-quality-gates
Jun 11, 2026
Merged

ci: enforce cargo check warning gate#69
JesseKoldewijn merged 11 commits into
developfrom
feat/ci-quality-gates

Conversation

@JesseKoldewijn

@JesseKoldewijn JesseKoldewijn commented Jun 9, 2026

Copy link
Copy Markdown
Owner

Summary

  • add a dedicated cargo check CI job with RUSTFLAGS=-D warnings so dead code and other rustc warnings fail required checks
  • include the new cargo check result in the required-check aggregators and PR results comment
  • fix the existing clippy findings in studio-ui and DynamoDB parsing so the stricter gate passes cleanly

Validation

  • cargo fmt --all
  • RUSTFLAGS=\"-D warnings\" cargo check --workspace --all-targets --all-features
  • cargo clippy --all-targets --all-features -- -D warnings

Summary by CodeRabbit

  • New Features

    • Added support for more AWS services (CloudTrail, Cognito/IdP, ECS, ElastiCache, RDS) across protocol routing, request translation, and service detection.
    • Improved object download streaming with size-aware read buffers for better large-object handling.
  • Bug Fixes

    • Overview tab error rate now computes from service-specific transactions, avoids overflow, and defaults safely; test added.
  • Tests

    • Expanded integration/unit tests and README service inventory checks for the new services.
  • Chores

    • CI: added a workspace Cargo Check job and surfaced its result in PR summaries.

Add a dedicated cargo check job with denied warnings so dead code and other rustc warnings fail required CI. Include the new result in required-check aggregators and fix the current clippy violations needed for the gate to pass.
@JesseKoldewijn

Copy link
Copy Markdown
Owner Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jun 9, 2026

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai

coderabbitai Bot commented Jun 9, 2026

Copy link
Copy Markdown

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: e6e4c5e3-3cbe-480d-820e-571b5e1c2522

📥 Commits

Reviewing files that changed from the base of the PR and between 1dc5053 and 99fece4.

📒 Files selected for processing (2)
  • .github/workflows/ci.yml
  • crates/services/s3/src/provider.rs
💤 Files with no reviewable changes (1)
  • .github/workflows/ci.yml

📝 Walkthrough

Walkthrough

Adds a workspace cargo-check CI job and surfaces its result in PR markdown; refactors DynamoDB key-condition parsing; computes service-scoped, overflow-safe error-rate in the studio UI with a unit test; adds S3 GET streaming buffer selection; and extends protocol, gateway, native-HTTP translators, and tests to support CloudTrail, Cognito, ECS, ElastiCache, and RDS.

Changes

Maintenance and Safety Improvements

Layer / File(s) Summary
CI Cargo Check Job and Pipeline Integration
.github/workflows/ci.yml
New check job running cargo check --workspace --all-targets --all-features with RUSTFLAGS: -D warnings; job added to required-checks-main, required-checks-non-main, and pr-results-comment; PR markdown generator updated to include a “Cargo Check” row sourced from CHECK_RESULT; benchmark gate --memory-max increased to 125.
DynamoDB Key Condition Parser Refactoring
crates/services/dynamodb/src/provider.rs
parse_key_condition operator arms use pattern guards so RangeCondition::{Lt,Lte,Gt,Gte} is assigned only when lhs matches the range key, removing nested if checks.
Studio UI Error Rate Calculation Safety
crates/studio-ui/src/explorer.rs
ServiceExplorerViewModel::build computes overview.error_rate_pct from log.for_service(service) only, uses checked_mul(100)/checked_div(...) with a 0 fallback and caps at 100; adds a unit test ensuring service-scoped transaction/error counts.
S3 GetObject streaming buffer selection
crates/services/s3/src/provider.rs
Introduce small/large read-buffer constants and a helper to select buffer size by object length; GetObject streaming uses the helper; add unit tests for cutoff behavior.

AWS Service Additions

Layer / File(s) Summary
AwsProtocol mapping and tests
crates/aws-protocol/src/protocol.rs, crates/aws-protocol/tests/protocol_integration.rs
Adds elasticache and rds to the Query arm; adds ecs, cloudtrail, and cognito-idp to the Json arm; updates test assertions to expect these mappings.
Gateway service detection and tests
crates/gateway/src/server.rs
Canonicalizes cognitocognito-idp, extends known-service lists to include ecs/cloudtrail/cognito-idp, adds CloudTrail-specific target parsing and version-suffix stripping, and extends prefix-to-service mappings; adds unit tests for host and target recognition.
Native HTTP translators and probes
crates/tests/integration/src/native_http.rs
Wires translate_command dispatch for cloudtrail, cognito-idp, ecs, elasticache, and rds; adds translate_* implementations producing JSON or query plans for supported operations; adds translation tests and probe helpers.
Service inventory / parity tests
crates/tests/integration/src/parity.rs
Expands all_service_names() and README-to-service-ID mapping test to include CloudTrail, Cognito, ECS, ElastiCache, RDS, and other name variants so tests reflect the updated service list.

Sequence Diagram(s)

sequenceDiagram
  participant NativeCLI
  participant translate_command
  participant translate_service
  participant json_target_plan
  participant query_plan
  participant Gateway
  NativeCLI->>translate_command: invoke translate for service:operation
  translate_command->>translate_service: delegate to service-specific translator (e.g., cloudtrail, cognito-idp)
  translate_service->>json_target_plan: build JSON target plan (x-amz-target)
  translate_service->>query_plan: build Query plan for Query-protocol ops
  query_plan->>Gateway: Gateway inspects x-amz-target/prefix to infer service
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

"A rabbit runs the cargo check with cheer,
Match guards keep the keys sincere,
Studio math hops safe and sound,
New services find their ground,
Little paws applaud the merge. 🐇"

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and concisely summarizes the main objective: enforcing a cargo check gate that turns warnings into failures in CI.
Description check ✅ Passed The PR description provides a clear summary, lists key changes, and includes validation steps; however, it does not follow the repository's required template structure with all expected sections.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/ci-quality-gates

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/ci.yml:
- Around line 143-145: In the check job update the three actions
(actions/checkout@v4, dtolnay/rust-toolchain@stable, Swatinem/rust-cache@v2) to
be pinned to their full commit SHAs, set the actions/checkout step to include
with: persist-credentials: false, and add a job-level permissions: override for
the check job to the least-privilege required (do not leave pull-requests:
write) so the job does not inherit write permissions from the workflow-level
setting.

In `@crates/studio-ui/src/explorer.rs`:
- Around line 365-375: The error_rate_pct is computed from the global tx_summary
but should use service-scoped counts; update the computation in
ServiceExplorerViewModel::build to call log.for_service(service).summary() (or
otherwise aggregate client_error, server_error, and total from the
service-scoped iterator) and compute errors = client_error + server_error, then
do the checked_mul/checked_div with the service total and fallback to 0/min 100
as before so division-by-zero is handled; reference tx_summary ->
service_tx_summary, log.for_service(service), service_tx_count, and
error_rate_pct when making the change.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 4840b441-efc3-46de-9796-d8f8fcbaab7f

📥 Commits

Reviewing files that changed from the base of the PR and between 8441ac1 and f0f756e.

📒 Files selected for processing (3)
  • .github/workflows/ci.yml
  • crates/services/dynamodb/src/provider.rs
  • crates/studio-ui/src/explorer.rs

Comment thread .github/workflows/ci.yml Outdated
Comment thread crates/studio-ui/src/explorer.rs Outdated
Scope Studio explorer error rates to the selected service and add a regression test for mixed-service traffic. Also harden the new cargo check workflow job with pinned actions, disabled checkout credential persistence, and least-privilege permissions.
@JesseKoldewijn

Copy link
Copy Markdown
Owner Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jun 9, 2026

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

Use the same 125 MB default benchmark memory ceiling in PR CI as the benchmark workflow so the gate stays consistent across both paths.
@JesseKoldewijn

Copy link
Copy Markdown
Owner Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jun 9, 2026

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@JesseKoldewijn

Copy link
Copy Markdown
Owner Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jun 9, 2026

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
crates/gateway/src/server.rs (1)

2357-2361: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Allow hyphens in host-based service validation.

At Line 2359, the character gate rejects cognito-idp, so host-derived Cognito requests can’t resolve via this branch even after normalization.

Suggested fix
-                .all(|c| c.is_ascii_lowercase() || c.is_ascii_digit())
+                .all(|c| c.is_ascii_lowercase() || c.is_ascii_digit() || c == '-')
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/gateway/src/server.rs` around lines 2357 - 2361, The host-derived
service name validation in the potential_service branch rejects hyphens (e.g.,
"cognito-idp"), preventing known services from matching; update the character
check inside the if in server.rs so the predicate allows '-' as well (i.e.,
change the closure used with potential_service.chars().all(...) to accept
c.is_ascii_lowercase() || c.is_ascii_digit() || c == '-'), keeping the
is_known_service(&potential_service) call unchanged so normalized host names
like "cognito-idp" resolve correctly.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/gateway/src/server.rs`:
- Around line 2599-2606: The X-Amz-Target parsing is too permissive: update the
CloudTrail prefix check and make ECS version stripping generic. For CloudTrail,
instead of is_some_and(|prefix|
prefix.eq_ignore_ascii_case("com.amazonaws.cloudtrail")), check that the target
starts with "com.amazonaws.cloudtrail" case-insensitively and that the next
character (if any) is a delimiter ('.', '_', or end-of-string) to avoid
over-matching. For ECS, replace the hardcoded two-version strip logic with a
generic version-trimming step that removes a trailing 'V' followed by a numeric
date/version token (e.g., regex or manual scan for 'V' + digits) before matching
"AmazonEC2ContainerService", ensuring variants like
"AmazonEC2ContainerServiceV20141113.*" match the ECS branch; apply these changes
to all occurrences that use target.split/next_back and the cloudtrail prefix
check (the branches handling ECS and CloudTrail in the X-Amz-Target parsing).

---

Outside diff comments:
In `@crates/gateway/src/server.rs`:
- Around line 2357-2361: The host-derived service name validation in the
potential_service branch rejects hyphens (e.g., "cognito-idp"), preventing known
services from matching; update the character check inside the if in server.rs so
the predicate allows '-' as well (i.e., change the closure used with
potential_service.chars().all(...) to accept c.is_ascii_lowercase() ||
c.is_ascii_digit() || c == '-'), keeping the
is_known_service(&potential_service) call unchanged so normalized host names
like "cognito-idp" resolve correctly.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 6ea62675-ae3f-48d4-a6b7-219d12c9b669

📥 Commits

Reviewing files that changed from the base of the PR and between f629c26 and 3c4dabe.

⛔ Files ignored due to path filters (3)
  • README.md is excluded by none and included by none
  • tests/parity/known_differences.json is excluded by none and included by none
  • tests/parity/scenarios/all-services-smoke.json is excluded by none and included by none
📒 Files selected for processing (6)
  • .github/workflows/ci.yml
  • crates/aws-protocol/src/protocol.rs
  • crates/aws-protocol/tests/protocol_integration.rs
  • crates/gateway/src/server.rs
  • crates/tests/integration/src/native_http.rs
  • crates/tests/integration/src/parity.rs

Comment thread crates/gateway/src/server.rs Outdated
@JesseKoldewijn JesseKoldewijn marked this pull request as ready for review June 9, 2026 14:02
@JesseKoldewijn

Copy link
Copy Markdown
Owner Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jun 10, 2026

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@JesseKoldewijn

Copy link
Copy Markdown
Owner Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jun 11, 2026

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/ci.yml:
- Around line 409-418: The docker pull inside the retry loop can hang
indefinitely; update the pull command in the loop that uses "for attempt in 1 2
3; do" and the docker pull invocation of "$PARITY_LOCALSTACK_IMAGE" to run with
a per-attempt timeout (e.g., using the timeout utility) so each attempt fails
quickly on hang and the retry logic can proceed; apply the same change to the
other identical loops referenced (the ones around the other docker pull
invocations at the other locations) and ensure the loop still breaks on success
and exits with the existing error handling when the final attempt fails.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: caa729dd-cb70-4e2d-8c47-35f01df9b731

📥 Commits

Reviewing files that changed from the base of the PR and between 1dc5053 and 66b857f.

📒 Files selected for processing (2)
  • .github/workflows/ci.yml
  • crates/services/s3/src/provider.rs

Comment thread .github/workflows/ci.yml Outdated
@JesseKoldewijn

Copy link
Copy Markdown
Owner Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jun 11, 2026

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

Repository owner deleted a comment from github-actions Bot Jun 11, 2026
Repository owner deleted a comment from github-actions Bot Jun 11, 2026
@github-actions

github-actions Bot commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

Benchmark Gate: ✅ PASS

Profile: default | Mode: docker | Requests: 100 | Concurrency: 6
Timestamp: 2026-06-11T09:53:41Z | Targets: openstack, LocalStack, moto
Thresholds: openstack p95 ≤ 10ms | openstack memory ≤ 125MB | errors = 0

Memory Footprint

Target Idle RSS (MB) Loaded RSS (MB)
openstack 20.5 102.0
LocalStack 381.9 863.2
moto 43.2 119.1

openstack uses 8x less memory than LocalStack under load

ACM

Operation Platform p50 (ms) p95 (ms) p99 (ms) RPS Errors
describe_certificate openstack 2.46 3.33 3.88 2316.5 0
LocalStack 8.53 (-3.5x) 14.36 (-4.3x) 16.96 (-4.4x) 620.8 (-3.7x) 0
moto 13.32 (-5.4x) 14.65 (-4.4x) 15.77 (-4.1x) 442.2 (-5.2x) 0
export_certificate openstack 1.77 2.71 3.23 3124.4 0
LocalStack 7.70 (-4.4x) 12.51 (-4.6x) 15.40 (-4.8x) 716.8 (-4.4x) 100
moto 13.34 (-7.5x) 14.19 (-5.2x) 17.44 (-5.4x) 442.4 (-7.1x) 0
list_certificates openstack 2.73 4.31 4.59 2051.1 0
LocalStack 37.18 (-13.6x) 54.59 (-12.7x) 67.18 (-14.6x) 160.2 (-12.8x) 0
moto 40.08 (-14.7x) 54.43 (-12.6x) 60.73 (-13.2x) 146.6 (-14.0x) 0
request_certificate openstack 0.63 2.47 3.63 312.5 0
LocalStack 476.51 (-756.4x) 589.00 (-238.5x) 607.25 (-167.3x) 11.3 (-27.7x) 0
moto 314.48 (-499.2x) 582.54 (-235.8x) 621.52 (-171.2x) 10.4 (-30.0x) 0

openstack vs LocalStack: 3.5×–756.4× faster on 4 ops (avg 194.5×)
openstack vs moto: 5.4×–499.2× faster on 4 ops (avg 131.7×)

APIGATEWAY

Operation Platform p50 (ms) p95 (ms) p99 (ms) RPS Errors
create_resource openstack 1.24 1.85 2.38 4334.5 0
LocalStack
moto 14.09 (-11.4x) 15.00 (-8.1x) 16.01 (-6.7x) 418.7 (-10.4x) 0
create_rest_api openstack 0.68 2.16 3.39 303.0 0
LocalStack 5.39 (-7.9x) 8.28 (-3.8x) 8.99 (-2.7x) 256.4 (-1.2x) 0
moto 10.60 (-15.6x) 858.10 (-397.3x) 870.10 (-256.7x) 37.5 (-8.1x) 0
get_rest_api openstack 1.12 1.82 2.00 4763.4 0
LocalStack 6.40 (-5.7x) 10.87 (-6.0x) 13.35 (-6.7x) 823.9 (-5.8x) 100
moto 13.28 (-11.9x) 13.94 (-7.7x) 19.11 (-9.6x) 439.8 (-10.8x) 0
get_rest_apis openstack 2.28 3.34 3.97 2437.3 0
LocalStack 6.33 (-2.8x) 9.88 (-3.0x) 10.97 (-2.8x) 867.6 (-2.8x) 100
moto 14.87 (-6.5x) 24.30 (-7.3x) 24.46 (-6.2x) 381.4 (-6.4x) 0

openstack vs LocalStack: 2.8×–7.9× faster on 3 ops (avg 5.5×)
openstack vs moto: 6.5×–15.6× faster on 4 ops (avg 11.3×)

CLOUDFORMATION

Operation Platform p50 (ms) p95 (ms) p99 (ms) RPS Errors
create_stack openstack 0.90 2.97 3.96 236.7 0
LocalStack 50.42 (-56.0x) 134.95 (-45.4x) 139.23 (-35.2x) 59.5 (-4.0x) 0
moto 10.01 (-11.1x) 15.49 (-5.2x) 16.00 (-4.0x) 186.9 (-1.3x) 0
describe_stacks openstack 2.09 3.27 3.51 2657.8 0
LocalStack 7.99 (-3.8x) 14.43 (-4.4x) 15.58 (-4.4x) 669.7 (-4.0x) 0
moto 13.35 (-6.4x) 14.29 (-4.4x) 15.69 (-4.5x) 442.7 (-6.0x) 0
get_template openstack 1.90 2.62 3.26 2943.4 0
LocalStack 6.89 (-3.6x) 10.44 (-4.0x) 12.00 (-3.7x) 809.9 (-3.6x) 0
moto 13.45 (-7.1x) 14.61 (-5.6x) 15.88 (-4.9x) 438.6 (-6.7x) 0
list_stacks openstack 2.53 3.83 4.18 2230.9 0
LocalStack 43.98 (-17.4x) 73.98 (-19.3x) 94.25 (-22.5x) 133.4 (-16.7x) 0
moto 13.38 (-5.3x) 14.31 (-3.7x) 16.46 (-3.9x) 440.4 (-5.1x) 0
update_stack openstack 2.04 3.06 3.17 2788.2 0
LocalStack 50.06 (-24.5x) 139.16 (-45.5x) 187.08 (-59.0x) 94.0 (-29.7x) 0
moto 18.75 (-9.2x) 20.52 (-6.7x) 22.71 (-7.2x) 314.8 (-8.9x) 0

openstack vs LocalStack: 3.6×–56.0× faster on 5 ops (avg 21.1×)
openstack vs moto: 5.3×–11.1× faster on 5 ops (avg 7.8×)

CLOUDWATCH

Operation Platform p50 (ms) p95 (ms) p99 (ms) RPS Errors
describe_alarms openstack 0.18 0.53 0.71 19556.3 0
LocalStack 6.70 (-37.2x) 40.28 (-76.0x) 45.29 (-63.8x) 607.4 (-32.2x) 0
moto 13.31 (-73.9x) 14.69 (-27.7x) 15.35 (-21.6x) 445.8 (-43.9x) 0
get_metric_statistics openstack 2.98 6.68 9.17 1757.3 0
LocalStack 53.37 (-17.9x) 67.24 (-10.1x) 83.25 (-9.1x) 108.7 (-16.2x) 0
moto 13.66 (-4.6x) 14.36 (-2.1x) 15.39 (-1.7x) 434.4 (-4.0x) 0
list_metrics openstack 2.77 4.46 4.81 2020.7 0
LocalStack 15.86 (-5.7x) 27.57 (-6.2x) 40.71 (-8.5x) 342.7 (-5.9x) 0
moto 21.00 (-7.6x) 26.27 (-5.9x) 31.20 (-6.5x) 273.8 (-7.4x) 0
put_metric_data openstack 2.16 3.34 4.41 2596.1 0
LocalStack 19.17 (-8.9x) 1276.09 (-382.1x) 2136.08 (-484.4x) 24.2 (-107.3x) 0
moto 19.75 (-9.1x) 22.49 (-6.7x) 25.03 (-5.7x) 297.8 (-8.7x) 0

openstack vs LocalStack: 5.7×–37.2× faster on 4 ops (avg 17.4×)
openstack vs moto: 4.6×–73.9× faster on 4 ops (avg 23.8×)

DYNAMODB

Operation Platform p50 (ms) p95 (ms) p99 (ms) RPS Errors
get_item openstack 0.98 1.44 1.64 5489.8 0
LocalStack 14.41 (-14.7x) 31.87 (-22.1x) 40.72 (-24.8x) 362.0 (-15.2x) 0
moto 13.30 (-13.6x) 14.25 (-9.9x) 16.51 (-10.1x) 443.3 (-12.4x) 0
put_item openstack 0.18 0.67 0.84 19072.4 0
LocalStack 21.09 (-117.2x) 83.28 (-124.3x) 121.79 (-145.0x) 174.6 (-109.2x) 0
moto 13.28 (-73.8x) 14.04 (-21.0x) 15.69 (-18.7x) 445.8 (-42.8x) 0
query openstack 1.38 2.02 2.11 4050.1 0
LocalStack 17.51 (-12.7x) 96.22 (-47.6x) 101.98 (-48.3x) 231.6 (-17.5x) 0
moto 13.22 (-9.6x) 14.15 (-7.0x) 15.52 (-7.4x) 447.2 (-9.1x) 0
scan openstack 1.31 1.94 2.19 4138.5 0
LocalStack 16.75 (-12.8x) 35.35 (-18.2x) 41.87 (-19.1x) 323.5 (-12.8x) 0
moto 13.24 (-10.1x) 16.25 (-8.4x) 17.91 (-8.2x) 434.8 (-9.5x) 0

openstack vs LocalStack: 12.7×–117.2× faster on 4 ops (avg 39.3×)
openstack vs moto: 9.6×–73.8× faster on 4 ops (avg 26.8×)

EC2

Operation Platform p50 (ms) p95 (ms) p99 (ms) RPS Errors
create_security_group openstack 0.79 2.21 2.37 312.5 0
LocalStack 13.42 (-17.0x) 25.86 (-11.7x) 28.45 (-12.0x) 181.0 (-1.7x) 40
moto 10.49 (-13.3x) 14.19 (-6.4x) 14.75 (-6.2x) 201.0 (-1.6x) 0
create_vpc openstack 0.69 2.50 3.80 320.0 0
LocalStack 15.15 (-22.0x) 21.51 (-8.6x) 22.56 (-5.9x) 172.4 (-1.9x) 0
moto 9.28 (-13.4x) 12.77 (-5.1x) 14.83 (-3.9x) 203.0 (-1.6x) 0
describe_instances openstack 2.48 4.32 5.81 2162.4 0
LocalStack 192.58 (-77.7x) 657.60 (-152.2x) 752.38 (-129.5x) 22.9 (-94.4x) 0
moto 13.32 (-5.4x) 15.13 (-3.5x) 17.40 (-3.0x) 440.6 (-4.9x) 0
run_instances openstack 0.83 2.04 2.80 307.7 0
LocalStack 28.61 (-34.5x) 33.81 (-16.6x) 35.43 (-12.7x) 125.4 (-2.5x) 0
moto 9.22 (-11.1x) 13.17 (-6.5x) 13.83 (-4.9x) 206.2 (-1.5x) 0

openstack vs LocalStack: 17.0×–77.7× faster on 4 ops (avg 37.8×)
openstack vs moto: 5.4×–13.4× faster on 4 ops (avg 10.8×)

ECR

Operation Platform p50 (ms) p95 (ms) p99 (ms) RPS Errors
batch_get_image openstack 2.23 3.08 4.13 2528.8 0
moto 13.22 (-5.9x) 13.95 (-4.5x) 15.65 (-3.8x) 446.9 (-5.7x) 0
create_repository openstack 0.64 2.04 3.43 320.0 0
moto 9.81 (-15.3x) 14.55 (-7.1x) 14.98 (-4.4x) 197.0 (-1.6x) 0
describe_repositories openstack 2.46 3.37 3.46 2279.2 0
moto 13.43 (-5.5x) 14.47 (-4.3x) 16.29 (-4.7x) 438.6 (-5.2x) 0
list_images openstack 1.36 1.99 2.23 4014.9 0
moto 13.32 (-9.8x) 14.24 (-7.2x) 15.79 (-7.1x) 444.4 (-9.0x) 0

⚠️ LocalStack: seed failed (seed request failed) — excluded from this service's benchmarks
⚠️ LocalStack: seed failed (seed request failed) — excluded from this service's benchmarks

openstack vs moto: 5.5×–15.3× faster on 4 ops (avg 9.1×)

EVENTBRIDGE

Operation Platform p50 (ms) p95 (ms) p99 (ms) RPS Errors
describe_event_bus openstack 1.21 1.81 2.16 4483.6 0
moto 13.35 (-11.0x) 24.50 (-13.5x) 31.62 (-14.6x) 414.6 (-10.8x) 0
describe_rule openstack 1.76 2.39 2.92 3154.8 0
moto 13.35 (-7.6x) 14.25 (-6.0x) 15.64 (-5.4x) 442.4 (-7.1x) 0
list_event_buses openstack 1.37 1.86 1.98 4008.9 0
moto 13.30 (-9.7x) 14.34 (-7.7x) 15.46 (-7.8x) 443.1 (-9.0x) 0
list_rules openstack 2.24 3.22 3.54 2497.5 0
moto 13.36 (-6.0x) 14.16 (-4.4x) 15.78 (-4.5x) 441.9 (-5.7x) 0
list_targets_by_rule openstack 1.10 1.58 1.89 4819.0 0
moto 13.46 (-12.2x) 14.57 (-9.2x) 15.96 (-8.4x) 439.6 (-11.0x) 0
put_rule openstack 0.53 1.70 2.69 317.5 0
moto 9.82 (-18.5x) 14.09 (-8.3x) 15.07 (-5.6x) 204.1 (-1.6x) 0

⚠️ LocalStack: seed failed (seed request failed) — excluded from this service's benchmarks
⚠️ LocalStack: seed failed (seed request failed) — excluded from this service's benchmarks

openstack vs moto: 6.0×–18.5× faster on 6 ops (avg 10.8×)

FIREHOSE

Operation Platform p50 (ms) p95 (ms) p99 (ms) RPS Errors
create_delivery_stream openstack 0.63 2.42 3.96 261.4 0
LocalStack 6.11 (-9.7x) 9.77 (-4.0x) 11.06 (-2.8x) 208.3 (-1.3x) 0
moto 9.02 (-14.3x) 13.44 (-5.6x) 14.31 (-3.6x) 186.0 (-1.4x) 0
describe_delivery_stream openstack 2.04 2.76 3.07 2770.6 0
LocalStack 5.86 (-2.9x) 10.28 (-3.7x) 13.74 (-4.5x) 894.0 (-3.1x) 0
moto 13.29 (-6.5x) 14.12 (-5.1x) 15.75 (-5.1x) 445.2 (-6.2x) 0
list_delivery_streams openstack 1.70 2.32 2.51 3271.8 0
LocalStack 6.07 (-3.6x) 9.91 (-4.3x) 12.91 (-5.1x) 888.4 (-3.7x) 0
moto 13.27 (-7.8x) 16.76 (-7.2x) 19.68 (-7.8x) 433.9 (-7.5x) 0
put_record openstack 1.09 1.85 2.28 4809.6 0
LocalStack 5.95 (-5.5x) 11.15 (-6.0x) 13.21 (-5.8x) 860.5 (-5.6x) 0
moto 13.23 (-12.1x) 13.87 (-7.5x) 15.66 (-6.9x) 447.9 (-10.7x) 0
put_record_batch openstack 1.72 2.35 2.91 3254.5 0
LocalStack 6.40 (-3.7x) 9.89 (-4.2x) 11.54 (-4.0x) 852.9 (-3.8x) 0
moto 13.30 (-7.7x) 14.19 (-6.0x) 15.18 (-5.2x) 444.7 (-7.3x) 0

openstack vs LocalStack: 2.9×–9.7× faster on 5 ops (avg 5.1×)
openstack vs moto: 6.5×–14.3× faster on 5 ops (avg 9.7×)

IAM

Operation Platform p50 (ms) p95 (ms) p99 (ms) RPS Errors
create_role openstack 0.66 1.77 2.59 310.1 0
LocalStack 30.08 (-45.6x) 35.08 (-19.8x) 38.05 (-14.7x) 120.8 (-2.6x) 0
moto 9.20 (-13.9x) 13.30 (-7.5x) 14.69 (-5.7x) 201.0 (-1.5x) 0
create_user openstack 0.66 2.68 2.92 325.2 0
LocalStack 11.44 (-17.3x) 14.03 (-5.2x) 14.24 (-4.9x) 203.0 (-1.6x) 0
moto 9.56 (-14.5x) 13.88 (-5.2x) 15.24 (-5.2x) 201.0 (-1.6x) 0
get_role openstack 2.07 3.04 3.28 2703.1 0
LocalStack 30.04 (-14.5x) 46.25 (-15.2x) 63.31 (-19.3x) 194.1 (-13.9x) 0
moto 13.46 (-6.5x) 14.92 (-4.9x) 15.59 (-4.8x) 438.0 (-6.2x) 0
get_user openstack 0.76 2.15 2.69 315.0 0
LocalStack 10.47 (-13.8x) 14.19 (-6.6x) 15.18 (-5.6x) 206.2 (-1.5x) 0
moto 10.33 (-13.6x) 16.03 (-7.5x) 16.72 (-6.2x) 188.7 (-1.7x) 0
list_users openstack 2.36 3.70 3.90 2376.4 0
LocalStack 31.98 (-13.6x) 53.15 (-14.4x) 62.92 (-16.1x) 181.7 (-13.1x) 0
moto 13.32 (-5.6x) 14.23 (-3.8x) 16.02 (-4.1x) 443.3 (-5.4x) 0

openstack vs LocalStack: 13.6×–45.6× faster on 5 ops (avg 20.9×)
openstack vs moto: 5.6×–14.5× faster on 5 ops (avg 10.8×)

KINESIS

Operation Platform p50 (ms) p95 (ms) p99 (ms) RPS Errors
describe_stream openstack 2.31 3.20 3.79 2448.4 0
LocalStack 50.21 (-21.7x) 70.37 (-22.0x) 79.38 (-20.9x) 113.7 (-21.5x) 0
moto 13.36 (-5.8x) 16.27 (-5.1x) 18.03 (-4.8x) 435.7 (-5.6x) 0
list_streams openstack 1.02 1.68 2.22 5006.7 0
LocalStack 48.62 (-47.7x) 55.31 (-32.9x) 73.50 (-33.1x) 123.8 (-40.4x) 0
moto 15.18 (-14.9x) 17.10 (-10.2x) 17.71 (-8.0x) 391.0 (-12.8x) 0
put_record openstack 0.96 1.49 1.67 5541.7 0
LocalStack 69.94 (-72.9x) 125.51 (-84.2x) 185.72 (-111.2x) 78.0 (-71.0x) 0
moto 13.31 (-13.9x) 14.31 (-9.6x) 16.52 (-9.9x) 442.7 (-12.5x) 0

openstack vs LocalStack: 21.7×–72.9× faster on 3 ops (avg 47.4×)
openstack vs moto: 5.8×–14.9× faster on 3 ops (avg 11.5×)

KMS

Operation Platform p50 (ms) p95 (ms) p99 (ms) RPS Errors
create_key openstack 0.67 2.82 3.19 268.5 0
LocalStack 6.62 (-9.9x) 55.19 (-19.6x) 55.88 (-17.5x) 189.6 (-1.4x) 0
moto 141.97 (-211.9x) 293.59 (-104.1x) 337.65 (-105.8x) 20.8 (-12.9x) 0
describe_key openstack 2.27 3.08 3.14 2490.3 0
LocalStack 6.05 (-2.7x) 10.55 (-3.4x) 12.94 (-4.1x) 896.1 (-2.8x) 0
moto 13.35 (-5.9x) 14.01 (-4.5x) 15.84 (-5.0x) 443.9 (-5.6x) 0
generate_data_key openstack 1.93 2.67 4.07 2881.7 0
LocalStack 7.71 (-4.0x) 12.71 (-4.8x) 13.35 (-3.3x) 738.9 (-3.9x) 0
moto 13.22 (-6.8x) 14.79 (-5.5x) 15.62 (-3.8x) 445.8 (-6.5x) 0
list_keys openstack 2.60 3.90 4.25 2187.3 0
LocalStack 6.78 (-2.6x) 11.09 (-2.8x) 13.66 (-3.2x) 810.3 (-2.7x) 0
moto 13.57 (-5.2x) 14.46 (-3.7x) 16.15 (-3.8x) 435.3 (-5.0x) 0

openstack vs LocalStack: 2.6×–9.9× faster on 4 ops (avg 4.8×)
openstack vs moto: 5.2×–211.9× faster on 4 ops (avg 57.5×)

LAMBDA

Operation Platform p50 (ms) p95 (ms) p99 (ms) RPS Errors
delete_function openstack 0.65 0.65 0.65 528.2 0
LocalStack 8.35 (-12.8x) 8.35 (-12.8x) 8.35 (-12.8x) 104.8 (-5.0x) 0
moto 4.71 (-7.2x) 4.71 (-7.2x) 4.71 (-7.2x) 166.7 (-3.2x) 0
get_function openstack 2.84 4.40 6.68 2073.1 0
LocalStack 8.69 (-3.1x) 12.21 (-2.8x) 15.60 (-2.3x) 695.6 (-3.0x) 0
moto 13.60 (-4.8x) 14.76 (-3.4x) 18.28 (-2.7x) 427.8 (-4.8x) 0
invoke openstack 0.25 1.21 1.35 12845.3 0
LocalStack 10.45 (-41.8x) 15.85 (-13.1x) 21.13 (-15.7x) 561.3 (-22.9x) 100
moto 31.71 (-126.8x) 84.05 (-69.5x) 105.76 (-78.3x) 163.6 (-78.5x) 0
list_functions openstack 2.30 3.42 3.83 2446.3 0
LocalStack 5.55 (-2.4x) 9.52 (-2.8x) 12.27 (-3.2x) 960.5 (-2.5x) 0
moto 13.42 (-5.8x) 17.02 (-5.0x) 18.04 (-4.7x) 418.8 (-5.8x) 0
update_function_code openstack 2.87 4.42 4.76 1962.6 0
LocalStack 49.79 (-17.3x) 77.73 (-17.6x) 89.03 (-18.7x) 114.9 (-17.1x) 100
moto 19.34 (-6.7x) 25.70 (-5.8x) 27.37 (-5.8x) 296.4 (-6.6x) 0
update_function_configuration openstack 3.06 5.06 6.66 1855.5 0
LocalStack 8.12 (-2.7x) 12.44 (-2.5x) 15.42 (-2.3x) 713.3 (-2.6x) 100
moto 17.52 (-5.7x) 20.96 (-4.1x) 24.10 (-3.6x) 331.4 (-5.6x) 100

openstack vs LocalStack: 2.4×–41.8× faster on 6 ops (avg 13.4×)
openstack vs moto: 4.8×–126.8× faster on 6 ops (avg 26.2×)

OPENSEARCH

Operation Platform p50 (ms) p95 (ms) p99 (ms) RPS Errors
create_domain openstack 0.80 2.92 3.20 210.5 0
LocalStack 5.59 (-7.0x) 9.41 (-3.2x) 10.28 (-3.2x) 173.2 (-1.2x) 0
moto 11.88 (-14.8x) 17.40 (-6.0x) 18.75 (-5.9x) 152.1 (-1.4x) 0
describe_domain openstack 2.76 6.85 9.72 1818.7 0
LocalStack 9.11 (-3.3x) 13.67 (-2.0x) 16.14 (-1.7x) 646.2 (-2.8x) 100
moto 17.50 (-6.3x) 20.55 (-3.0x) 24.51 (-2.5x) 334.4 (-5.4x) 0
describe_domain_config openstack 1.59 2.53 2.81 3362.1 0
LocalStack 10.35 (-6.5x) 16.95 (-6.7x) 22.62 (-8.0x) 562.2 (-6.0x) 100
moto 18.03 (-11.3x) 19.96 (-7.9x) 22.92 (-8.2x) 336.9 (-10.0x) 0
list_domain_names openstack 2.04 3.13 3.52 2755.7 0
LocalStack 9.07 (-4.4x) 14.37 (-4.6x) 18.89 (-5.4x) 621.0 (-4.4x) 100
moto 16.15 (-7.9x) 21.26 (-6.8x) 24.84 (-7.1x) 352.1 (-7.8x) 100
update_domain_config openstack 1.86 3.24 5.07 2896.5 0
LocalStack 7.64 (-4.1x) 13.06 (-4.0x) 14.77 (-2.9x) 748.9 (-3.9x) 0
moto 18.48 (-9.9x) 20.47 (-6.3x) 22.49 (-4.4x) 319.8 (-9.1x) 0

openstack vs LocalStack: 3.3×–7.0× faster on 5 ops (avg 5.1×)
openstack vs moto: 6.3×–14.8× faster on 5 ops (avg 10.1×)

REDSHIFT

Operation Platform p50 (ms) p95 (ms) p99 (ms) RPS Errors
create_cluster openstack 0.68 1.85 3.27 310.1 0
LocalStack 14.30 (-21.0x) 19.17 (-10.4x) 21.23 (-6.5x) 143.9 (-2.2x) 0
moto 10.24 (-15.1x) 15.23 (-8.2x) 15.53 (-4.7x) 184.3 (-1.7x) 0
describe_clusters openstack 2.64 3.58 4.50 2163.4 0
LocalStack 137.95 (-52.3x) 527.03 (-147.2x) 591.61 (-131.5x) 36.9 (-58.6x) 0
moto 13.24 (-5.0x) 14.77 (-4.1x) 18.90 (-4.2x) 440.5 (-4.9x) 0
modify_cluster openstack 2.14 3.32 4.32 2629.1 0
LocalStack 13.16 (-6.1x) 21.34 (-6.4x) 23.70 (-5.5x) 434.3 (-6.1x) 0
moto 13.46 (-6.3x) 14.28 (-4.3x) 16.58 (-3.8x) 438.1 (-6.0x) 0
reboot_cluster openstack 2.19 4.52 5.21 2380.5 0
LocalStack 8.34 (-3.8x) 14.11 (-3.1x) 20.45 (-3.9x) 664.1 (-3.6x) 100
moto 13.31 (-6.1x) 15.95 (-3.5x) 16.45 (-3.2x) 440.5 (-5.4x) 0

openstack vs LocalStack: 3.8×–52.3× faster on 4 ops (avg 20.8×)
openstack vs moto: 5.0×–15.1× faster on 4 ops (avg 8.1×)

ROUTE53

Operation Platform p50 (ms) p95 (ms) p99 (ms) RPS Errors
change_resource_record_sets openstack 2.04 3.23 3.51 2746.8 0
LocalStack 5.06 (-2.5x) 8.35 (-2.6x) 10.28 (-2.9x) 1064.9 (-2.6x) 0
moto 14.21 (-7.0x) 15.13 (-4.7x) 16.55 (-4.7x) 417.2 (-6.6x) 0
create_hosted_zone openstack 0.72 2.19 3.12 322.6 0
moto 21.73 (-30.2x) 36.35 (-16.6x) 38.08 (-12.2x) 114.6 (-2.8x) 0
get_hosted_zone openstack 1.96 3.02 3.70 2811.4 0
moto 37.58 (-19.2x) 41.47 (-13.7x) 43.14 (-11.7x) 157.9 (-17.8x) 0
list_hosted_zones openstack 2.32 3.35 4.58 2440.2 0
moto 36.13 (-15.6x) 38.29 (-11.4x) 40.12 (-8.8x) 164.4 (-14.8x) 0
list_resource_record_sets openstack 2.05 2.80 3.39 2742.0 0
moto 60.64 (-29.6x) 62.98 (-22.5x) 63.68 (-18.8x) 98.6 (-27.8x) 0

⚠️ LocalStack: seed failed (seed request failed) — excluded from this service's benchmarks

openstack vs LocalStack: 2.5×–2.5× faster on 1 op (avg 2.5×)
openstack vs moto: 7.0×–30.2× faster on 5 ops (avg 20.3×)

S3

S3 — 1MB

Operation Platform p50 (ms) p95 (ms) p99 (ms) RPS Errors
get_object_1mb openstack 1.47 2.97 3.81 3858.4 0
LocalStack 25.30 (-17.2x) 28.17 (-9.5x) 30.60 (-8.0x) 235.6 (-16.4x) 0
moto 17.21 (-11.7x) 19.87 (-6.7x) 20.79 (-5.5x) 341.4 (-11.3x) 0
head_object_1mb openstack 0.17 0.44 0.71 20442.2 0
LocalStack 5.91 (-34.8x) 10.07 (-22.9x) 12.20 (-17.2x) 918.2 (-22.3x) 0
moto 12.82 (-75.4x) 17.57 (-39.9x) 18.02 (-25.4x) 440.8 (-46.4x) 0
list_objects_v2_1mb openstack 2.18 3.11 3.59 2597.4 0
LocalStack 5.78 (-2.7x) 9.78 (-3.1x) 11.29 (-3.1x) 929.1 (-2.8x) 0
moto 13.19 (-6.1x) 14.82 (-4.8x) 21.54 (-6.0x) 437.0 (-5.9x) 0
put_object_1mb openstack 6.48 13.48 17.24 809.7 0
LocalStack 19.52 (-3.0x) 21.33 (-1.6x) 32.80 (-1.9x) 296.3 (-2.7x) 0
moto 36.76 (-5.7x) 38.56 (-2.9x) 40.47 (-2.3x) 162.6 (-5.0x) 0

openstack vs LocalStack: 2.7×–34.8× faster on 4 ops (avg 14.4×)
openstack vs moto: 5.7×–75.4× faster on 4 ops (avg 24.7×)

S3 — 10MB

Operation Platform p50 (ms) p95 (ms) p99 (ms) RPS Errors
get_object_10mb openstack 17.38 26.06 35.36 314.5 0
LocalStack 174.79 (-10.1x) 209.64 (-8.0x) 228.59 (-6.5x) 33.5 (-9.4x) 0
moto 32.80 (-1.9x) 41.02 (-1.6x) 42.26 (-1.2x) 177.9 (-1.8x) 0
head_object_10mb openstack 0.19 0.72 0.85 13892.3 0
LocalStack 6.11 (-32.2x) 11.12 (-15.4x) 11.44 (-13.5x) 880.7 (-15.8x) 0
moto 12.72 (-66.9x) 13.46 (-18.7x) 15.62 (-18.4x) 459.9 (-30.2x) 0
list_objects_v2_10mb openstack 2.22 3.37 3.51 2447.8 0
LocalStack 6.13 (-2.8x) 9.64 (-2.9x) 10.80 (-3.1x) 870.8 (-2.8x) 0
moto 13.31 (-6.0x) 14.84 (-4.4x) 16.45 (-4.7x) 435.6 (-5.6x) 0
put_object_10mb openstack 56.80 217.29 237.29 66.0 0
LocalStack 134.45 (-2.4x) 140.88 (+1.5x) 185.75 (+1.3x) 43.0 (-1.5x) 0
moto 202.71 (-3.6x) 221.95 252.97 (-1.1x) 29.2 (-2.3x) 1

openstack vs LocalStack: 2.4×–32.2× faster on 4 ops (avg 11.8×)
openstack vs moto: 1.9×–66.9× faster on 4 ops (avg 19.6×)

S3 — 50MB

Operation Platform p50 (ms) p95 (ms) p99 (ms) RPS Errors
get_object_50mb openstack 56.73 66.38 66.38 67.2 0
LocalStack 566.02 (-10.0x) 604.69 (-9.1x) 604.69 (-9.1x) 7.0 (-9.6x) 0
moto 74.91 (-1.3x) 88.96 (-1.3x) 88.96 (-1.3x) 51.8 (-1.3x) 0
head_object_50mb openstack 0.17 0.66 0.66 8016.3 0
LocalStack 4.19 (-24.6x) 6.99 (-10.6x) 6.99 (-10.6x) 813.3 (-9.9x) 0
moto 8.48 (-49.9x) 10.43 (-15.8x) 10.43 (-15.8x) 445.5 (-18.0x) 0
list_objects_v2_50mb openstack 1.56 2.42 2.42 2078.7 0
LocalStack 4.51 (-2.9x) 7.34 (-3.0x) 7.34 (-3.0x) 782.0 (-2.7x) 0
moto 9.03 (-5.8x) 10.86 (-4.5x) 10.86 (-4.5x) 409.6 (-5.1x) 0
put_object_50mb openstack 242.20 625.26 625.26 13.3 0
LocalStack 419.63 (-1.7x) 564.95 (+1.1x) 564.95 (+1.1x) 9.0 (-1.5x) 0
moto 603.56 (-2.5x) 668.80 (-1.1x) 668.80 (-1.1x) 6.6 (-2.0x) 13

openstack vs LocalStack: 1.7×–24.6× faster on 4 ops (avg 9.8×)
openstack vs moto: 1.3×–49.9× faster on 4 ops (avg 14.9×)

S3 — 100MB

Operation Platform p50 (ms) p95 (ms) p99 (ms) RPS Errors
get_object_100mb openstack 77.92 89.89 89.89 25.0 0
LocalStack 523.18 (-6.7x) 531.79 (-5.9x) 531.79 (-5.9x) 3.8 (-6.6x) 0
moto 76.50 88.20 88.20 25.6 0
head_object_100mb openstack 0.16 0.49 0.49 4316.9 0
LocalStack 2.33 (-14.6x) 3.55 (-7.2x) 3.55 (-7.2x) 714.4 (-6.0x) 0
moto 4.42 (-27.6x) 5.48 (-11.2x) 5.48 (-11.2x) 418.7 (-10.3x) 0
list_objects_v2_100mb openstack 1.03 1.58 1.58 1438.8 0
LocalStack 2.38 (-2.3x) 4.71 (-3.0x) 4.71 (-3.0x) 674.0 (-2.1x) 0
moto 4.62 (-4.5x) 5.64 (-3.6x) 5.64 (-3.6x) 394.2 (-3.6x) 0
put_object_100mb openstack 298.78 312.74 312.74 6.7 0
LocalStack 410.57 (-1.4x) 546.66 (-1.7x) 546.66 (-1.7x) 4.6 (-1.5x) 0
moto 551.34 (-1.8x) 682.87 (-2.2x) 682.87 (-2.2x) 3.5 (-1.9x) 2

openstack vs LocalStack: 1.4×–14.6× faster on 4 ops (avg 6.2×)
openstack vs moto: 1.8×–27.6× faster on 3 ops (avg 11.3×)
openstack vs moto: 1.0×–1.0× slower on 1 op (avg 1.0×)

SECRETSMANAGER

Operation Platform p50 (ms) p95 (ms) p99 (ms) RPS Errors
create_secret openstack 0.70 2.99 3.42 320.0 0
LocalStack 7.78 (-11.1x) 10.65 (-3.6x) 11.40 (-3.3x) 223.5 (-1.4x) 0
moto 9.44 (-13.5x) 13.03 (-4.4x) 13.40 (-3.9x) 209.4 (-1.5x) 0
get_secret_value openstack 1.93 3.02 3.72 2854.3 0
LocalStack 6.14 (-3.2x) 9.72 (-3.2x) 14.89 (-4.0x) 878.6 (-3.2x) 0
moto 13.35 (-6.9x) 14.50 (-4.8x) 16.19 (-4.4x) 443.5 (-6.4x) 0
list_secrets openstack 2.90 4.56 5.71 2064.8 0
LocalStack 25.49 (-8.8x) 41.54 (-9.1x) 48.96 (-8.6x) 228.7 (-9.0x) 0
moto 13.57 (-4.7x) 14.56 (-3.2x) 16.96 (-3.0x) 435.5 (-4.7x) 0
put_secret_value openstack 1.80 2.48 4.55 3103.5 0
LocalStack 6.06 (-3.4x) 10.67 (-4.3x) 11.45 (-2.5x) 901.1 (-3.4x) 100
moto 13.55 (-7.5x) 16.23 (-6.5x) 17.31 (-3.8x) 423.7 (-7.3x) 0
update_secret openstack 1.30 2.05 2.72 4158.7 0
LocalStack 6.18 (-4.8x) 11.24 (-5.5x) 13.87 (-5.1x) 874.2 (-4.8x) 100
moto 13.29 (-10.2x) 14.11 (-6.9x) 15.89 (-5.8x) 443.1 (-9.4x) 0

openstack vs LocalStack: 3.2×–11.1× faster on 5 ops (avg 6.2×)
openstack vs moto: 4.7×–13.5× faster on 5 ops (avg 8.6×)

SES

Operation Platform p50 (ms) p95 (ms) p99 (ms) RPS Errors
get_identity_verification_attributes openstack 1.87 2.78 4.03 3014.5 0
LocalStack 6.96 (-3.7x) 11.74 (-4.2x) 13.14 (-3.3x) 804.0 (-3.7x) 0
moto 13.55 (-7.2x) 15.58 (-5.6x) 18.00 (-4.5x) 431.7 (-7.0x) 0
list_identities openstack 2.01 3.09 3.62 2779.4 0
LocalStack 9.72 (-4.8x) 15.64 (-5.1x) 19.16 (-5.3x) 590.4 (-4.7x) 0
moto 13.51 (-6.7x) 16.52 (-5.3x) 17.39 (-4.8x) 427.6 (-6.5x) 0
send_email openstack 1.96 2.95 3.39 2863.0 0
LocalStack 11.56 (-5.9x) 19.29 (-6.5x) 22.48 (-6.6x) 497.2 (-5.8x) 0
moto 13.62 (-6.9x) 17.94 (-6.1x) 21.05 (-6.2x) 410.0 (-7.0x) 0
send_raw_email openstack 1.98 3.20 3.81 2825.1 0
LocalStack 8.25 (-4.2x) 12.23 (-3.8x) 13.00 (-3.4x) 699.7 (-4.0x) 100
moto 13.57 (-6.9x) 14.72 (-4.6x) 15.94 (-4.2x) 435.4 (-6.5x) 0
verify_email_identity openstack 0.86 2.39 2.88 315.0 0
LocalStack 9.67 (-11.2x) 12.76 (-5.3x) 14.10 (-4.9x) 211.6 (-1.5x) 0
moto 9.59 (-11.2x) 13.83 (-5.8x) 14.13 (-4.9x) 200.0 (-1.6x) 0

openstack vs LocalStack: 3.7×–11.2× faster on 5 ops (avg 6.0×)
openstack vs moto: 6.7×–11.2× faster on 5 ops (avg 7.8×)

SNS

Operation Platform p50 (ms) p95 (ms) p99 (ms) RPS Errors
get_topic_attributes openstack 2.68 4.15 4.42 2131.3 0
LocalStack 9.71 (-3.6x) 338.12 (-81.5x) 340.63 (-77.1x) 189.2 (-11.3x) 0
moto 14.92 (-5.6x) 16.02 (-3.9x) 18.13 (-4.1x) 396.8 (-5.4x) 0
list_subscriptions openstack 2.07 3.09 3.98 2682.5 0
LocalStack 6.54 (-3.2x) 10.59 (-3.4x) 14.92 (-3.7x) 838.6 (-3.2x) 0
moto 14.48 (-7.0x) 14.96 (-4.8x) 16.67 (-4.2x) 410.3 (-6.5x) 0
list_topics openstack 2.23 3.50 4.06 2473.0 0
LocalStack 8.03 (-3.6x) 15.04 (-4.3x) 19.87 (-4.9x) 641.9 (-3.9x) 0
moto 14.09 (-6.3x) 15.38 (-4.4x) 17.39 (-4.3x) 419.1 (-5.9x) 0
publish openstack 1.94 2.66 2.80 2885.3 0
LocalStack 6.72 (-3.5x) 11.07 (-4.2x) 12.37 (-4.4x) 807.8 (-3.6x) 0
moto 17.73 (-9.1x) 23.38 (-8.8x) 24.59 (-8.8x) 334.7 (-8.6x) 0

openstack vs LocalStack: 3.2×–3.6× faster on 4 ops (avg 3.5×)
openstack vs moto: 5.6×–9.1× faster on 4 ops (avg 7.0×)

SQS

Operation Platform p50 (ms) p95 (ms) p99 (ms) RPS Errors
delete_message openstack 1.64 2.61 3.27 3326.2 0
LocalStack 10.09 (-6.2x) 19.21 (-7.4x) 36.34 (-11.1x) 511.3 (-6.5x) 0
moto 17.41 (-10.6x) 20.43 (-7.8x) 21.09 (-6.4x) 346.3 (-9.6x) 0
get_queue_attributes openstack 2.08 3.32 4.22 2629.1 0
LocalStack 7.69 (-3.7x) 13.33 (-4.0x) 14.52 (-3.4x) 717.4 (-3.7x) 0
moto 16.03 (-7.7x) 93.82 (-28.3x) 94.15 (-22.3x) 286.3 (-9.2x) 0
list_queues openstack 1.87 3.03 4.01 2923.5 0
LocalStack 6.67 (-3.6x) 11.00 (-3.6x) 11.57 (-2.9x) 809.1 (-3.6x) 0
moto 14.43 (-7.7x) 16.19 (-5.3x) 17.99 (-4.5x) 406.7 (-7.2x) 0
receive_message openstack 2.66 4.13 4.57 2173.2 0
LocalStack 13.59 (-5.1x) 23.36 (-5.7x) 40.22 (-8.8x) 397.7 (-5.5x) 0
moto 38.68 (-14.5x) 64.79 (-15.7x) 67.95 (-14.9x) 135.0 (-16.1x) 0
send_message openstack 2.01 3.15 3.83 2769.9 0
LocalStack 11.23 (-5.6x) 18.55 (-5.9x) 24.99 (-6.5x) 491.4 (-5.6x) 0
moto 24.11 (-12.0x) 28.50 (-9.0x) 30.48 (-8.0x) 244.6 (-11.3x) 0

openstack vs LocalStack: 3.6×–6.2× faster on 5 ops (avg 4.8×)
openstack vs moto: 7.7×–14.5× faster on 5 ops (avg 10.5×)

SSM

Operation Platform p50 (ms) p95 (ms) p99 (ms) RPS Errors
describe_parameters openstack 1.79 2.92 3.48 3065.0 0
LocalStack 8.16 (-4.6x) 12.88 (-4.4x) 14.64 (-4.2x) 678.3 (-4.5x) 0
moto 13.34 (-7.5x) 14.94 (-5.1x) 15.82 (-4.5x) 440.2 (-7.0x) 0
get_parameter openstack 2.01 3.03 3.92 2831.4 0
LocalStack 12.04 (-6.0x) 31.47 (-10.4x) 55.82 (-14.2x) 410.3 (-6.9x) 0
moto 13.47 (-6.7x) 14.86 (-4.9x) 17.51 (-4.5x) 434.6 (-6.5x) 0
get_parameters openstack 2.05 2.78 3.33 2716.2 0
LocalStack 12.22 (-6.0x) 19.82 (-7.1x) 21.77 (-6.5x) 466.2 (-5.8x) 0
moto 13.45 (-6.6x) 14.57 (-5.2x) 15.76 (-4.7x) 439.1 (-6.2x) 0
get_parameters_by_path openstack 2.00 2.93 3.21 2772.5 0
LocalStack 7.98 (-4.0x) 13.15 (-4.5x) 18.14 (-5.7x) 670.2 (-4.1x) 0
moto 13.32 (-6.7x) 15.33 (-5.2x) 16.48 (-5.1x) 437.1 (-6.3x) 0
put_parameter openstack 0.72 1.18 1.45 7105.5 0
LocalStack 24.30 (-33.8x) 32.65 (-27.7x) 36.48 (-25.2x) 240.7 (-29.5x) 0
moto 13.49 (-18.7x) 14.64 (-12.4x) 17.02 (-11.7x) 435.1 (-16.3x) 0

openstack vs LocalStack: 4.0×–33.8× faster on 5 ops (avg 10.8×)
openstack vs moto: 6.6×–18.7× faster on 5 ops (avg 9.2×)

STEPFUNCTIONS

Operation Platform p50 (ms) p95 (ms) p99 (ms) RPS Errors
create_state_machine openstack 0.87 3.05 3.79 294.1 0
LocalStack 11.69 (-13.4x) 17.33 (-5.7x) 18.07 (-4.8x) 187.8 (-1.6x) 0
moto 10.92 (-12.6x) 15.45 (-5.1x) 16.53 (-4.4x) 191.4 (-1.5x) 0
describe_state_machine openstack 2.21 2.98 3.02 2567.9 0
LocalStack 6.14 (-2.8x) 10.58 (-3.6x) 11.59 (-3.8x) 873.7 (-2.9x) 0
moto 13.29 (-6.0x) 14.45 (-4.8x) 16.02 (-5.3x) 443.1 (-5.8x) 0
list_state_machines openstack 2.31 3.17 3.89 2469.0 0
LocalStack 8.30 (-3.6x) 13.56 (-4.3x) 16.07 (-4.1x) 668.3 (-3.7x) 0
moto 13.33 (-5.8x) 14.18 (-4.5x) 16.06 (-4.1x) 444.2 (-5.6x) 0
start_execution openstack 0.83 2.73 3.48 296.3 0
LocalStack 100.26 (-120.8x) 597.69 (-218.9x) 650.22 (-186.8x) 27.5 (-10.8x) 0
moto 10.22 (-12.3x) 15.38 (-5.6x) 15.50 (-4.5x) 186.0 (-1.6x) 0

openstack vs LocalStack: 2.8×–120.8× faster on 4 ops (avg 35.2×)
openstack vs moto: 5.8×–12.6× faster on 4 ops (avg 9.2×)

STS

Operation Platform p50 (ms) p95 (ms) p99 (ms) RPS Errors
assume_role openstack 2.26 3.54 4.06 2501.8 0
LocalStack 10.54 (-4.7x) 17.37 (-4.9x) 23.24 (-5.7x) 523.6 (-4.8x) 0
moto 13.44 (-5.9x) 14.40 (-4.1x) 15.48 (-3.8x) 440.4 (-5.7x) 0
get_access_key_info openstack 1.88 2.85 3.21 2981.1 0
LocalStack 8.25 (-4.4x) 12.76 (-4.5x) 13.37 (-4.2x) 683.1 (-4.4x) 100
moto 13.37 (-7.1x) 14.36 (-5.0x) 15.19 (-4.7x) 442.7 (-6.7x) 0
get_caller_identity openstack 1.91 3.04 4.16 2887.3 0
LocalStack 8.41 (-4.4x) 16.44 (-5.4x) 22.40 (-5.4x) 638.0 (-4.5x) 0
moto 13.69 (-7.2x) 14.29 (-4.7x) 16.18 (-3.9x) 434.0 (-6.7x) 0
get_session_token openstack 2.24 3.19 3.36 2553.6 0
LocalStack 8.98 (-4.0x) 14.35 (-4.5x) 18.71 (-5.6x) 604.7 (-4.2x) 0
moto 13.35 (-6.0x) 14.78 (-4.6x) 15.51 (-4.6x) 444.8 (-5.7x) 0

openstack vs LocalStack: 4.0×–4.7× faster on 4 ops (avg 4.4×)
openstack vs moto: 5.9×–7.2× faster on 4 ops (avg 6.5×)

Overall Performance

openstack vs LocalStack: 1.4×–756.4× faster on 105 ops (avg 21.3×)
openstack vs moto: 1.3×–499.2× faster on 119 ops (avg 18.6×)
openstack vs moto: 1.0×–1.0× slower on 1 op (avg 1.0×)

@github-actions

Copy link
Copy Markdown
Contributor

CI Results

Checks

Check Result
Rustfmt success
Cargo Check success
Clippy success
Tests success
Build Artifact success
Coverage Validation success

Parity

Profile Scenarios Passed Failed Accepted
core 5 5 0 0
all-services-smoke 34 34 0 95

@JesseKoldewijn

Copy link
Copy Markdown
Owner Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jun 11, 2026

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@JesseKoldewijn JesseKoldewijn merged commit 6f660c3 into develop Jun 11, 2026
55 checks passed
@JesseKoldewijn JesseKoldewijn deleted the feat/ci-quality-gates branch June 11, 2026 09:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant