[Bug]: Uploads are very slow through the agent reverse proxy (ModSecurity scanning request bodies)

[morepriyam]

Current URL / affected service

A container I created through the Manager, running a Fastify app that does resumable (TUS) uploads. It's reached through the agent's nginx like any other service. https://pulsevault.os.mieweb.org/docs

For comparison, the same app runs on a plain EC2 box with nothing in front of it (https://pulse.morepriyam.com), and uploads there are fast.

Username

pmore

Version

main

Steps to reproduce

1. Run an app that accepts large uploads (mine is Fastify + TUS) as a container, exposed as an HTTP service.
2. Upload a big file (tens/hundreds of MB) — it goes up as large PATCH requests.
3. Try the same upload straight to the app with no proxy in front, and compare the speed.

Expected behavior

Uploads through the platform proxy should be roughly as fast as hitting the app directly.

Actual behavior

They're much slower behind the proxy — bad enough to be unusable for bigger files. They don't fail, they just crawl.

I'm fairly sure it's the ModSecurity WAF on the agent. It's turned on globally in the nginx config and runs in DetectionOnly mode. Even in that mode it still reads the whole request body to scan it, so every upload chunk gets buffered and inspected before it reaches my container. That cancels out the proxy_request_buffering off that's already set, and since it's DetectionOnly it isn't even blocking anything, just logging — so I'm paying the scan cost for nothing.

A couple of other things in the generated config probably don't help big uploads either: the proxy timeouts are all hardcoded to 60s, and client_max_body_size is fixed at 2G.

Additional context

This has come up before — the WAF blocked PulseClip's TUS uploads in #219 (fixed by allowing the offset+octet-stream content type in #220), and it 403'd some HL7 traffic in #221, which is why it got moved to detection-only mode. So the pattern so far has been allowlisting specific cases rather than letting a service opt out.

Worth noting the error-pages server in the same config already uses modsecurity off, so turning it off per-server is possible. And #355 is currently reworking the same nginx template, so heads up on overlap.

Happy to help with a fix — I just wanted to raise it first and hear how you'd want to handle letting a service skip the WAF when it needs the upload speed.

[morepriyam]

@runleveldev can you please accept the latest requests for the resources bump(pulsevault), retire the remaning inactive nodes, and it is possible to turn off ModSecurity and other such services for the same container
this is what i'm testing with
https://github.com/mieweb/pulsevault/tree/main/examples/fastify-demo as a simple client server app

quick setup needs npm i and npm run build on root it uses PV as a file based module, and simply cd fastify demo, npm i and && start

and the client that uploads:
https://github.com/morepriyam/pulse ( Pulse-Camera on appstore connect, already updated on testflight (stateless))

Feedback on the setup and architecture also welcomed :)

[runleveldev]

@morepriyam I've approved the resource requests for the pulsevault container, the pulsevaultdemo requests are still pending due to a bug not allowing me to approve them without the container being created.

Before I make any changes to the ModSecurity config, we need to actually narrow down the cause of the upload speed. Below are some things you can try to help us narrow it down:

1. Setup a simple webserver (NGINX, Apache, etc) behind the proxy and test the upload there (ensure the upload request is essentially identical). If the speeds are faster, it's app-specific and not related to the proxy.
2. Try with smaller files, under the NGINX buffer size. If speeds are similar behind the proxy and not with smaller files, it could be a buffering issue.
3. If (2) is faster, try chunking uploads (if you aren't already) to keep them within the buffer limits. I won't be expanding the buffer as that can easily cause a denial-of-service for other users behind the proxy.
4. Setup a local NGINX with a similar proxying and ModSecurity setup to this repo. Compare upload speeds with the same file under different configurations. Determine the minimum-impact change which gives the most benefit. Let me know the findings. Similar to 3, there are some settings I'm not open to changing due to potential performance impact for other users.

[morepriyam]

@runleveldev
Thanks for approving the resources, and for the list. It gave me a clear way to work through this. Quick background for anyone reading from scratch (and so you can correct me if my reasoning is off): TUS uploads work by POST-ing to create an upload URL, then PATCH-ing the bytes to it. The server's Upload-Offset advances only as bytes actually land server-side; after a drop, the client HEADs the URL and resumes from that offset. So resume only works if bytes reach the app while the client sends. If a proxy collects the whole body first and forwards it after, a mid-upload drop means the app saw nothing and the offset is still 0.

Being honest up front: since the resource bump, throughput is a lot better, and speed alone probably wouldn't justify a config change anymore. But while testing I found the bigger issue: interrupted uploads always resume from offset 0 through the proxy (confirmed against my live container as well as the local setup below; they resume fine against the app directly). On flaky mobile networks that means big uploads basically never finish. My probe for everything below: kill an upload at ~40 MB, then HEAD it (~40 MB kept means streaming, 0 means buffered), plus a watcher recording when the first byte reaches the app's storage.

1) App-specific? I think I can rule it out. The exact proxy block from nginx-conf.ejs in front of the same app but without modsecurity on: first bytes hit the app ~800 ms in, interrupted upload keeps ~42 MB. Add modsecurity on back, the only change, and it's 0 bytes until completion, 0 kept.

2) Smaller files under the buffer size? I expected a threshold I could stay under, but there isn't one:

Size	Direct: first byte at app	Through proxy: first byte at app
1 MB	0.09 s	0.21 s, only at completion
8 MB	0.08 s	0.86 s, only at completion
32 MB	0.08 s	1.84 s, only at completion
100 MB	0.09 s	5.44 s, only at completion

Even 1 MB is held until the request completes (RAM up to SecRequestBodyInMemoryLimit, then a temp file under /var/lib/nginx/body; I watched one grow to ~40 MB and get deleted on disconnect). The cost just scales with size, which is why it felt like a big-file problem.

3) Chunking: tried it. Each PATCH is held the same way, so it only adds a round trip per chunk. Also worth flagging: crs-setup.conf ships SecRequestBodyLimit 13107200 + Reject. Dormant in DetectionOnly, but when I flipped the engine to On in my lab, chunks over 12.5 MB got 403'd (same failure mode as #219).

4) Local replica + minimum-impact change: I did this twice: first a faithful replica (same Debian 13 packages, your ModSec setup and rules, proxy block verbatim), one knob at a time; then, so you don't have to trust my replica, this repo itself: create-a-container + dev seeder, my container registered through the API, ghcr.io/mieweb/opensource-server/agent:latest as the edge, every variant delivered by editing views/nginx-conf.ejs and letting /etc/pull-config.d/nginx do fetch → test → reload. Both agree, on h2 and h1.1:

Config	Streams during transfer?	Kept after mid-upload drop
pristine template	no	0
+ SecRequestBodyAccess Off in the location	no	0
+ SecRuleEngine Off	no	0
+ modsecurity off; in the location	yes (~800 ms)	~42 MB

The first two surprised me, so I dug into the connector source. As far as I can tell from ngx_http_modsecurity_pre_access.c (v1.0.3): the pre-access handler unconditionally calls ngx_http_read_client_request_body() and pauses the request until the full body arrives, before proxy_pass runs and before any Sec-directive is consulted. Those control whether the body gets inspected, not whether it gets read, which I think is also what defeats the proxy_request_buffering off already in the template. The only path that skips the read is ModSecurity not being enabled for the location, matching the results.

So the smallest change I could find is modsecurity off; scoped to the proxied location, or a per-service opt-out flag gating that line in the template, if you'd rather not make it blanket. No buffer sizes, timeouts, or other vhosts touched, which I know were your constraints. My read is that this path is opaque video bytes under DetectionOnly anyway, but you know the security posture better than I do, so I'd defer to you on which shape is acceptable, and happy to run any other test you'd want to see.

Repro notes (if you want to verify)

• Manager: create-a-container, npm ci, migrate + dev seeder, server.js + job-runner.js; container registered via the API (http service on :3000).
• Edge: ghcr.io/mieweb/opensource-server/agent:latest on a 10.0.0.0/16 Docker network, SITE_ID/MANAGER_URL/API_KEY in /etc/environment, self-signed certs for the external domains, config pulled by /etc/pull-config.d/nginx.
• Probe: TUS create → PATCH 100 MB at 20 MB/s, kill at ~2 s (curl --limit-rate 20M --max-time 2 -T file) → HEAD for Upload-Offset; sample the app's partial file every 0.4 s during the PATCH.
• Gotcha: nginx -s reload is async, so wait for full worker turnover between config changes, or stale workers give misleading results (one of my early runs did exactly that).

[runleveldev]

@morepriyam thanks for the detailed follow up. I've done some research into ModSecurity's documentation on this issue and confirmed that it is as you described. We are blocked by #407, but I'll start a PR for after that's merged.

[morepriyam]

@runleveldev Resumability through the proxy works on my vhost now, so it looks like an override was applied to my container. Thank you! Verified with the same probe as before: killed a 100MB PATCH mid transfer, the server kept the ~20MB that was sent, and it resumed from that offset and completed. Previously this always came back Upload-Offset: 0.

To make sure this wasn't something specific to my container, I also reproduced the buffering locally in Docker using the repo's config (nginx + ModSecurity CRS in DetectionOnly, proxy location from nginx-conf.ejs). Killing a 100MB TUS PATCH after ~42MB: with the repo default the server kept 0 bytes, same with SecRuleEngine Off, and with modsecurity off; in the location it kept all 42MB. So it lines up with what you found in the docs.

One more thing I ran into while testing, : with ModSecurity out of the way, uploads are still pinned at ~1 MB/s at any file size, while the same request forced over HTTP/1.1 gets 6 to 17 MB/s. It turns out to be HTTP/2 flow control: the default stream window is 64KB (RFC 9113), and with proxy_request_buffering off nginx never grows it, so upload speed becomes 64KB per RTT (Cloudflare has a good write-up of the same problem on their edge). I benchmarked it in the same Docker lab with injected latency: defaults gave a flat 1.2 MB/s at 50ms RTT and 0.6 MB/s at 100ms, while either http2 off; on the vhost (clients fall back to HTTP/1.1 via ALPN) or raising http2_body_preread_size / http2_recv_buffer_size got it into the 5 to 15 MB/s range. Of the two, http2 off also kept resume offsets byte exact after a kill, so that's the one I'd lean toward for upload heavy services, but you know the infra tradeoffs.

Is there anything you'd like me to do to help move this along?

[runleveldev]

@morepriyam I haven't made any changes yet, it'll likely be early next week at the soonest. My plan for now is to disable ModSecurity entirely (I consider it a failed experiment). I will not, however, be disabling http2 (or http3 which we also have enabled). Since NGINX opensource has not implemented the autotuning solution in the Cloudflare write-up, I can include increased limits to the http2_*_size variables. If you'd still prefer to avoid http2 for your uploads it should be possible to avoid the upgrade from the client side.

[horner]

For testing the security block for nginx:

#!/usr/bin/env bash
# Reproduce mieweb/opensource-server#395: the agent proxy (ModSecurity) buffers
# TUS PATCH bodies, so an interrupted upload resumes from offset 0.
#
# Probe: start a large PATCH, kill it mid-flight, then HEAD the upload.
#   Upload-Offset: 0        -> proxy buffered the body (BUG reproduced)
#   Upload-Offset: ~bytes   -> body streamed through (fixed / no proxy)
#
# Usage:
#   ./pulse-tus-test.sh [BASE_URL]
#   ./pulse-tus-test.sh --http1.1 [BASE_URL]   # force HTTP/1.1 for the PATCH
#
# Default BASE_URL: https://pulsevault.os.mieweb.org
set -euo pipefail

HTTP_FLAG=""
if [[ "${1:-}" == "--http1.1" ]]; then
  HTTP_FLAG="--http1.1"
  shift
fi
BASE="${1:-https://pulsevault.os.mieweb.org}"
ENDPOINT="$BASE/pulsevault/upload"

SIZE_MB=100
SIZE_BYTES=$((SIZE_MB * 1024 * 1024))
RATE=5M        # upload rate cap
KILL_AFTER=5   # seconds before curl aborts the PATCH (~25 MB sent)
TESTFILE="$(mktemp /tmp/tus-test.XXXXXX.bin)"
trap 'rm -f "$TESTFILE"' EXIT

echo "==> Creating ${SIZE_MB} MB test file"
dd if=/dev/urandom of="$TESTFILE" bs=1048576 count="$SIZE_MB" 2>/dev/null

echo "==> Reserving artifactId: POST $BASE/reserve"
ARTIFACT_ID=$(curl -s -X POST "$BASE/reserve" | sed -n 's/.*"artifactId":"\([^"]*\)".*/\1/p')
[[ -n "$ARTIFACT_ID" ]] || { echo "ERROR: could not reserve an artifactId" >&2; exit 1; }
echo "    artifactId: $ARTIFACT_ID"
META="filename $(printf %s tus-repro.mp4 | base64),artifactId $(printf %s "$ARTIFACT_ID" | base64)"

echo "==> TUS create: POST $ENDPOINT"
LOC=$(curl -si -X POST "$ENDPOINT" \
  -H 'Tus-Resumable: 1.0.0' \
  -H "Upload-Length: $SIZE_BYTES" \
  -H "Upload-Metadata: $META" \
  | tr -d '\r' | awk -F': ' 'tolower($1)=="location"{print $2}')
[[ -n "$LOC" ]] || { echo "ERROR: no Location header from TUS create" >&2; exit 1; }
case "$LOC" in /*) LOC="$BASE$LOC";; esac
echo "    Upload URL: $LOC"

echo "==> PATCH: uploading at $RATE, killing after ${KILL_AFTER}s (~25 MB sent)"
set +e
curl -s -o /dev/null -X PATCH "$LOC" $HTTP_FLAG \
  -H 'Tus-Resumable: 1.0.0' \
  -H 'Upload-Offset: 0' \
  -H 'Content-Type: application/offset+octet-stream' \
  -T "$TESTFILE" \
  --limit-rate "$RATE" --max-time "$KILL_AFTER"
set -e

echo "==> HEAD probe: how many bytes did the app keep?"
OFFSET=$(curl -sI -X HEAD "$LOC" -H 'Tus-Resumable: 1.0.0' \
  | tr -d '\r' | awk -F': ' 'tolower($1)=="upload-offset"{print $2}')
echo "    Upload-Offset: ${OFFSET:-<missing>}"

if [[ "${OFFSET:-0}" -eq 0 ]]; then
  echo "RESULT: 0 bytes kept -> body was buffered by the proxy (issue #395 reproduced)"
  exit 2
else
  echo "RESULT: $OFFSET bytes kept -> body streamed to the app (resume works)"
fi

Ran terminal command: ./pulse-tus-test.sh

horner@hornerMacbookM5 opensource-server %  ./pulse-tus-test.sh
==> Creating 100 MB test file
==> Reserving artifactId: POST https://pulsevault.os.mieweb.org/reserve
    artifactId: d478ed65-f0cb-4898-be1f-5c4158b0935a
==> TUS create: POST https://pulsevault.os.mieweb.org/pulsevault/upload
    Upload URL: https://pulsevault.os.mieweb.org/pulsevault/upload/dmlkZW8vZDQ3OGVkNjUtZjBjYi00ODk4LWJlMWYtNWM0MTU4YjA5MzVhLm1wNA
==> PATCH: uploading at 5M, killing after 5s (~25 MB sent)
==> HEAD probe: how many bytes did the app keep?
    Upload-Offset: 0
RESULT: 0 bytes kept -> body was buffered by the proxy (issue #395 reproduced)

Issue #395 reproduced against the live host: curl sent ~25 MB of a 100 MB PATCH before aborting, but the HEAD probe returned Upload-Offset: 0 — the proxy spooled the entire body and the app never saw a byte, so resume restarts from zero.