ci(perf): HTTPS/TLS performance gates for 3.9 - #309
Merged
Merged
Conversation
Extends the perf-gate infrastructure with TLS coverage, following the
runner-baseline convention: an absolute backstop per metric (must hold on
any CI hardware) plus a same-runner-CPU median comparison once >=5 samples
exist for that CPU, since absolute numbers move a lot between CI machines.
Gates (measured CWIST 3.9 baselines in comments):
1. tls_rtt_p50_ms sequential TLS client WITHOUT TCP_NODELAY,
first-response RTT p50 over 200 fresh connections; backstop <= 5ms.
This is the #307 regression test: pre-#307 measures ~43ms here.
2. https_churn_rps ab -n 6000 -c 32 new-conn-per-request; >= 1200
(~1650 healthy, ~600 pre-#307).
3. https_keepalive_rps wrk -t4 -c100 -d10s; >= 130k (~167-190k).
4. https_big_gbps wrk -t4 -c32 on a 1MiB body; >= 3GB/s (~4GB/s).
5. http_* plaintext controls for the same workloads (collateral damage).
The https/http keep-alive ratio (~0.65) is recorded as a diagnostic
signal, not gated. Bench server and cert are generated at runtime; the
gate job runs on PRs touching the TLS path and on dev pushes, and appends
to benchmarks/https_gates.json only after gates pass on a dev push.
Verified locally on a Ryzen 5600X: 7/7 gates pass on dev; with the
quickack re-arms disabled the run measures 43.2ms RTT p50 / 683 churn rps
and the evaluator exits 1 (fails gates 1 and 2).
CI run 37201724744 (PR #309, shared AMD EPYC 9V45 runner) failed only http_big_gbps: 7.8 measured vs the 8.0 backstop, while every other gate passed with the local-box numbers as reference. The absolute backstops were calibrated on a quiet 12-core desktop and are too tight for noisy shared runners. Recalibrate each backstop against the observed CI values with variance headroom: tls_rtt_p50_ms <=5ms (unchanged; observed 0.070) https_churn_rps >=1200 -> >=1000 (observed 2013) https_keepalive_rps >=130k -> >=110k (observed 135570) https_big_gbps >=3 -> >=2.5 (observed 4.03) http_churn_rps >=8000 -> >=6000 (observed 15793) http_keepalive_rps >=180k -> >=150k (observed 206650) http_big_gbps >=8 -> >=5.5 (observed 7.8) Verified: a row rebuilt from the failed run's observed values now passes 7/7, and the pre-#307 row (43.2ms RTT p50, 683/s churn) still fails gates 1 and 2. The https_keepalive_ratio diagnostic signal is unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
CWIST 3.9 had zero TLS coverage in perf gates:
benchmarks/webserver.json/scripts/ci/benchmark.pygate plaintext HTTP only. This adds a self-contained HTTPS gate suite, closing the loop on #306 (TLS handshake plateau) and locking in the #307 fix (quickack Nagle/delayed-ACK stall).New files (4, no changes to existing gates):
scripts/ci/https_perf_gates.shscripts/ci/https_gates_eval.pyrunner_baseline.py; history inbenchmarks/https_gates.json, seeded).github/workflows/perf-https-gates.ymlbenchmarks/https_gates.jsonGate table
tls_rtt_p50_mshttps_churn_rpsab -n 6000 -c 32, new connection per requesthttps_keepalive_rpswrk -t4 -c100 -d10shttps_big_gbpswrk -t4 -c32 -d10son a 1 MiB bodyhttp_*controlsRatio signal:
https_keepalive_ratio(HTTPS/HTTP keep-alive, ~0.65 expected) is recorded but not gated — a large drop hints at TLS-path damage even when absolute gates pass.Backstop calibration (updated after CI run 37201724744): absolute backstops were initially tuned on a quiet 12-core desktop and proved too tight for shared CI runners (
http_big_gbpsmeasured 7.8 vs the 8.0 backstop on a shared EPYC 9V45 while all other gates passed). Commit94eb670brecalibrated every backstop against the observed CI values with variance headroom; a row rebuilt from that failed run's numbers now passes 7/7, and the pre-#307 row still fails gates 1 and 2.Runner-relative gating: like the existing
runner_baseline.pygates, the absolute backstops are only the any-hardware backstop; once ≥5 history rows exist for a CI CPU model, each metric is also judged againstmedian / allowance(throughput) ormedian × allowance(RTT), because absolute numbers vary by CI CPU (documented inhttps_gates_eval.pyheader).Local verification (Ryzen 5 5600X, 12 vCPU)
Healthy code (
dev+ this branch), full end-to-end run ofscripts/ci/https_perf_gates.sh+ evaluator:Negative test (quickack re-arms disabled — simulates pre-#307):
Gate 1 and gate 2 reproduce the pre-#307 numbers (43ms / 683/s) and fail exactly as designed.
Also verified:
make libcwist.aclean,test_httpsandtest_https_parkpass on this branch.Notes
perf-https-handshake-shards.ymlworkflow.pushtodevafter gates pass; PR runs (including forks, which get read-only tokens) never modify the history file./tmpat runtime; no dependency onexample/certs.Refs #306, #307