perf(server): keep web responsive under provider load by seonghobae · Pull Request #849 · ContextualWisdomLab/contextual-orchestrator

seonghobae · 2026-08-25T12:02:25Z

Outcome

Slow provider calls no longer amplify the stdlib server's five-connection backlog or force a new TCP connection for every fixed-length response.

retains the existing daemon request-thread isolation for blocking provider I/O
uses the operating system's native SOMAXCONN accept backlog
uses HTTP/1.1 persistent connections for fixed-length JSON/text/SSE responses
preserves explicit nonblocking max_concurrent_runs admission (503 on saturation)
adds a real k6 E2E workload with 64 simultaneous delayed inference users and concurrent /healthz arrival traffic

Exact k6 evidence

Command:

uv run python tests/load/serve_synthetic_delay.py --delay-seconds 1 --max-concurrent-runs 64
BASE_URL=http://127.0.0.1:18089 k6 run tests/load/k6_web_concurrency.js --summary-mode=full

Base 838b3de1 -> tested candidate source 97ce7ed7:

authenticated inference checks: 128/128 -> 128/128
HTTP failures: 0/229 -> 0/229
inference scenario rate: 16.18 -> 25.02 req/s (+54.61%, measured only; not a routing weight)
inference connection-blocked average: 571.73 ms -> 1.79 ms
inference connection-blocked p95: 3.95 s -> 9.16 ms
liveness p99 during one-second provider waits: 18.73 ms -> 9.90 ms
candidate liveness checks: 101/101

The liveness threshold is an isolation invariant (p99 < synthetic provider delay), not a fabricated production SLO. Full details and APA 7 references: docs/benchmarks/2026-08-25-web-concurrency-k6.md.

Verification

python -m pytest -q tests/test_web_concurrency.py tests/test_healthz.py tests/test_security_hardening.py tests/test_api_contract.py tests/test_review_gateway.py tests/test_workflow_run_object_authorization.py — 52 passed
python -m pytest -q tests/test_request_framing.py tests/test_web_concurrency.py tests/test_healthz.py tests/test_security_hardening.py — 33 passed
full suite reached 1876 passed with one intentional HTTP/1.0 assertion exposed; that assertion was updated to the new HTTP/1.1 contract and the owning framing suite then passed
git diff --check origin/main...HEAD

Scope

No provider credentials or production data are used. TLS, real provider quotas, multi-process serving, and sustained soak capacity remain deployment-specific measurements rather than claims in this PR.

coderabbitai · 2026-08-25T12:02:35Z

Warning

Review limit reached

Next included review available in 20 minutes.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 9067879e-51e3-4fb5-a129-8bd2ac36c092

📥 Commits

Reviewing files that changed from the base of the PR and between 5a01759 and ede33b5.

📒 Files selected for processing (13)

CHANGELOG.md
contextual_orchestrator/__main__.py
contextual_orchestrator/server.py
docs/architecture.md
docs/benchmarks/2026-08-25-web-concurrency-k6.md
docs/rest_api_design.md
loadtests/k6_gateway_smoke.js
tests/load/k6_web_concurrency.js
tests/load/serve_synthetic_delay.py
tests/test_cli_auth.py
tests/test_http11_unread_body.py
tests/test_request_framing.py
tests/test_web_concurrency.py

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

seonghobae · 2026-08-26T01:37:08Z

Exact-head k6 E2E rerun on 6103806: 64 concurrent delayed inference users completed 128/128 inference checks while 101/101 liveness checks completed; 0/229 HTTP failures, 25.07 inference requests/s, and 6.2 ms liveness p99 against a deliberate 1 s provider delay. This is loopback synthetic-delay gateway evidence, not a production TLS/provider/soak SLO.

opencode-agent · 2026-08-26T08:02:56Z

Scheduled review-feedback autofix for this PR head.

Head SHA: d88e3218f093c7d4b77bd78a1d2b54445595ecb5

…zing k6 end-to-end evidence (loadtests/k6_gateway_smoke.js, mock agent pool so results measure the gateway layer, not provider latency): - Baseline (HTTP/1.0, default backlog 5): healthz error rate 7.8% at 200 req/s from connection-reset storms; client-side ephemeral-port exhaustion ('dial: i/o timeout') at burst; chat error rate 3.4% against the default 60 req/min per-IP budget. - After: checks 100%, http_req_failed 0% across 19k requests, healthz p95 34.5ms / chat p95 52ms under combined load. Thresholds in the k6 script encode this measured baseline as a regression gate. Changes: - Handler.protocol_version = 'HTTP/1.1' — responses already set Content-Length, so connections are reusable instead of paying a TCP handshake + TIME_WAIT socket per request. - GatewayHTTPServer.request_queue_size = 128 — accept queue no longer overflows under probe bursts. - Framing guard now emits 'Connection: close' on ambiguous-frame rejections so keep-alive clients can see the termination they always enforced server-side (test updated to assert both status and close). - --rate-limit-requests / --rate-limit-window-seconds CLI flags so operators and load tests can size the per-client budget explicitly.

…-fix

devin-ai-integration

Devin Review found 1 new potential issue.

seonghobae enabled auto-merge (squash) August 25, 2026 12:02

This comment was marked as resolved.

Sign in to view

opencode-agent Bot disabled auto-merge August 25, 2026 14:17

seonghobae enabled auto-merge (squash) August 25, 2026 14:39

opencode-agent Bot disabled auto-merge August 25, 2026 14:41

seonghobae enabled auto-merge (squash) August 25, 2026 14:56

This comment was marked as resolved.

Sign in to view

opencode-agent Bot disabled auto-merge August 26, 2026 00:48

seonghobae enabled auto-merge (squash) August 26, 2026 00:50

seonghobae added 11 commits August 26, 2026 17:37

perf(server): keep web responsive under provider load

cf3975e

docs(perf): record exact k6 concurrency evidence

6687181

test(server): assert HTTP 1.1 framing

5ab1190

fix(server): close rejected HTTP 1.1 request bodies

8003924

fix(server): bound inactive persistent request reads

ff60116

fix(server): preserve zero-length keep-alive requests

5846447

fix(server): treat missing body marker as consumed

7a37ab8

fix: close unsupported body-bearing requests

a8da189

fix(server): isolate persistent request context

a80a031

refactor(server): remove shadowed request handler

196ac2d

opencode-agent Bot disabled auto-merge August 26, 2026 10:11

fix(load): align smoke traffic with configured limits

4fa3983

seonghobae enabled auto-merge (squash) August 26, 2026 10:21

seonghobae force-pushed the feat/async-web-k6 branch from 92dec2b to 4fa3983 Compare August 26, 2026 10:26

This comment was marked as resolved.

Sign in to view

fix(server): emit one connection close header

566b74e

This comment was marked as resolved.

Sign in to view

seonghobae and others added 2 commits August 26, 2026 19:53

fix(cli): validate rate limits at parse boundary

e0847fe

Merge branch 'main' into feat/async-web-k6

7abf1b8

This comment was marked as resolved.

Sign in to view

opencode-agent Bot disabled auto-merge August 26, 2026 12:45

seonghobae enabled auto-merge (squash) August 26, 2026 12:53

seonghobae added 2 commits August 27, 2026 01:25

fix(load): exercise HTTP/1.1 connection reuse

649255b

Merge remote-tracking branch 'origin/main' into codex/pr849-keepalive…

ede33b5

…-fix

devin-ai-integration Bot reviewed Aug 26, 2026

View reviewed changes

Comment thread contextual_orchestrator/server.py

Uh oh!

Conversation

seonghobae commented Aug 25, 2026 • edited by devin-ai-integration Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Outcome

Exact k6 evidence

Verification

Scope

Uh oh!

coderabbitai Bot commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review limit reached

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

seonghobae commented Aug 26, 2026

Uh oh!

opencode-agent Bot commented Aug 26, 2026

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

seonghobae commented Aug 25, 2026 •

edited by devin-ai-integration Bot

Loading

coderabbitai Bot commented Aug 25, 2026 •

edited

Loading