Add manually triggered AWS performance tests for Sendium #352

Summary

Implement a basic performance-testing harness and a manually triggered GitHub Actions workflow to measure Sendium's sending and delivery-report performance.

Maintainers select the branch using Run workflow -> Use workflow from. The workflow tests that branch's exact commit and publishes results with the workflow run.

V1 tests one version per run. Automated comparisons with main, historical analysis, and performance regression thresholds are deferred.

Goals

• Make performance tests convenient and repeatable.
• Exercise HTTP and SMPP admission through outgoing SMPP submission.
• Include database-backed delivery-report processing by default.
• Measure useful sending throughput, latency, resource consumption, and unfinished work.
• Keep infrastructure sizes, traffic rate, and test durations configurable.
• Preserve machine-readable results for future comparisons.

Architecture

GitHub-hosted Actions runner
    |
    +-- Build application and harness artifacts
    +-- Provision and control the test environment
    +-- Retrieve results
    +-- Clean up temporary resources

AWS private test environment
    |
    +-- Harness EC2
    |     +-- HTTP traffic generator
    |     +-- SMPP load client
    |     +-- SMPP provider simulator
    |     +-- HTTP callback / SMPP DLR receivers
    |     +-- Measurement and reporting
    |
    +-- Sendium EC2
    |     +-- HTTP / SMPP admission
    |     +-- Routing and outgoing SMPP worker
    |     +-- DLR tracking and forwarding
    |
    +-- Private RDS PostgreSQL

Run the harness separately from Sendium so traffic generation and simulation do not consume Sendium's resource budget.

Use private networking between the instances and RDS. EC2 instances do not require public IPs or publicly exposed benchmark ports. Provide connectivity to AWS management services and deployment artifacts through the appropriate VPC endpoints or outbound networking.

Use AWS Systems Manager for instance control. Prefer placing both EC2 instances in the same Availability Zone and record the database's placement.

Initial test scope

Support two selectable scenarios:

1. HTTP -> Sendium -> simulated SMPP provider
2. SMPP client -> Sendium -> simulated SMPP provider

Both scenarios use:

• Single-part messages.
• One outgoing SMPP destination.
• Successful provider responses with a configurable fixed response delay.
• DLR enabled by default, with an option to disable it.
• RDS-backed DLR tracking when DLR is enabled.

For DLR-enabled runs, the simulator returns a provider message ID and generates a successful delivery receipt after a fixed, configurable delay. The harness receives and measures Sendium's forwarded HTTP callbacks or SMPP receipts.

Workflow inputs

Expose the essential inputs:

Keep region, networking, RDS sizing, connection/window settings, and JVM configuration in a documented infrastructure/test preset. Record the effective settings in every result.

Start with packaged JVM execution. Build artifacts compatible with the selected instance architecture.

Test lifecycle

1. Record the selected branch and resolve its exact commit SHA.
2. Build application and harness artifacts outside the measurement period.
3. Prepare the EC2 instances and isolated RDS test database.
4. Verify Sendium readiness, database availability, and SMPP connections.
5. Warm up for 2 minutes, exercising sending and DLR processing.
6. Measure for 10 minutes at the configured offered rate.
7. Stop new submissions and drain for up to 2 minutes, continuing provider responses and outstanding DLRs.
8. Finalize message accounting and reports.
9. Retrieve artifacts and clean up resources owned by the run.

Durations must be configurable. Distinguish warm-up messages from measurement messages. Track measurement messages through completion even when their outcomes arrive during drain.

Load generation and accounting

The generator should schedule a fixed rate of new submissions rather than reducing input automatically when responses become slow.

• Bound outstanding requests.
• Report requested versus achieved input rate.
• Report missed generation targets and harness saturation.
• Correlate submissions with provider requests and delivery receipts.
• Keep measurement bookkeeping bounded.
• Identify unexpected duplicates and accepted messages left unfinished after drain.

A growing outstanding backlog must be visible; fast admission alone must not be presented as sustainable sending throughput.

Required measurements

Report overall results and one-minute interval results.

Define each latency boundary explicitly. For DLR forwarding, distinguish the configured simulated delivery delay from the time taken to forward the generated receipt.

Record instance CPU-credit state and Standard/Unlimited mode when using burstable instances. A ten-minute run does not establish long-term CPU-credit sustainability.

Flag measurements as inconclusive if the harness cannot supply the intended load or is otherwise the bottleneck.

Result storage

GitHub Actions run summary

Publish a concise, human-readable report containing:

• Sending and DLR throughput.
• Latency percentiles.
• Admission errors and rejections.
• CPU/memory measurements.
• Outstanding and unfinished work.
• Validity status and relevant limitations.

Downloadable workflow artifact

Include:

• results.json with a versioned schema.
• Overall and interval measurements.
• Effective application, workload, and infrastructure configuration.
• Branch, commit SHA, workflow run URL, harness revision, and artifact identifiers.
• Diagnostic logs.

Set and document an explicit artifact retention period. Do not commit generated benchmark results into the repository.

Permanent result storage and automatic comparison are follow-up work.

Maintainer-controlled execution

• Trigger the workflow manually.
• Restrict AWS access through a maintainer-controlled GitHub environment.
• Authenticate using GitHub OIDC.
• Keep application builds and execution separate from AWS provisioning privileges.
• Use isolated test credentials and database state.
• Prevent concurrent runs from sharing mutable resources or data.
• Tag resources with the workflow run identity.
• Clean up after success, failure, and cancellation.
• Provide an independent expiry mechanism for abandoned temporary resources.

Acceptance criteria

• A maintainer can test a selected branch without editing workflow code.
• The report identifies the exact application commit and effective test configuration.
• HTTP and SMPP single-part scenarios both work.
• DLR is enabled by default and can be disabled.
• DLR-enabled runs exercise private RDS tracking and receipt forwarding.
• Default phases are two-minute warm-up, ten-minute measurement, and two-minute bounded drain.
• Results include useful sending throughput, latency percentiles, resource usage, and message accounting.
• Overall and one-minute interval results appear in the summary and downloadable artifact.
• Generator saturation and unfinished accepted work are clearly reported.
• Setup failures and scenario-defined accounting violations fail the workflow; expected overload rejections are reported separately.
• No arbitrary performance pass/fail thresholds are introduced.
• Temporary AWS resources are cleaned up reliably.

Deferred

• Automated comparisons with main or other versions.
• Maximum sustainable-throughput search.
• Multipart and ingress-reassembly scenarios.
• Retries, rerouting, disconnections, and failure injection.
• Multiple outgoing destinations.
• Native-image benchmarks.
• Extended endurance tests.
• Historical dashboards and permanent result storage.
• Automatic execution on every PR.
• Performance regression gates.

This is a separate tooling effort from PR #351.