fix(api): cap ingest request body and decompressed gzip size

[damilolaedwards]

What

The ingest endpoint read the full request body with io.ReadAll and decompressed gzip bodies with no size limit, so a small gzip payload could inflate to an arbitrary size in memory before any validation ran. Verified live: an 815KB request pushed the API process from a 40MB baseline to 1.67GB RSS in under a second.

How

• The raw request body is capped at 10MiB regardless of encoding.
• A gzip-encoded body's decompressed output is separately capped at 50MiB.
• Both use http.MaxBytesReader. Exceeding either limit now returns 413 instead of buffering the full payload first.

Test plan

• New tests: valid compressed/uncompressed reports still accepted, an oversized raw body rejected, a large-but-under-the-cap report still accepted (no off-by-one), and a gzip-bomb regression test at the real 50MiB limit.
• Rebuilt the binary and replayed the original live scenario against the fix: the same 815KB/800MB-decompressed payload now returns 413 in 25ms with peak RSS of 156MB, down from 1.67GB.
• go test -race ./pkg/api/... passes, go vet and gofmt clean.

[skylenet]

Reviewed this and pushed b0ab161. The core mechanism here is sound — I probed it directly rather than reasoning about it, and confirmed a gzip stream whose compressed size exceeds 10 MiB does surface as 413 (the MaxBytesError survives translation through compress/flate instead of being masked as a corruption error), a multi-member gzip bomb is capped, and a body of exactly maxIngestBodyBytes is accepted, so there's no off-by-one.

What I added is the endpoint the caps didn't reach.

/auth/login was the more exposed hole

handleLogin decodes with json.NewDecoder(r.Body).Decode(&req) and no limit, and it's reachable before any authentication runs. Rate limiting doesn't cover it either — rate_limit.enabled defaults to false, and the auth-tier limiter is only wired when it's on. Measured on this branch: a 64 MiB body shaped as one long username costs ~320 MiB of allocation. That's a bigger amplification factor than the ingest path this PR fixes, and it needs no credentials, where the ingest bomb at least needed the shared bearer token. Same shape at handleCreateAPIKey and the five decode sites in admin.go.

Fixed on this PR rather than a follow-up because it's the same defense and the same mechanism — splitting it would have shipped "we capped request bodies" with the unauthenticated endpoint still uncapped.

limitRequestBody(n) is mounted on the /auth and /admin groups at 1 MiB. A body advertising a Content-Length over the cap is refused on the headers without being read; a chunked body is bounded by the reader and reaches the handler as a decode error.

Two ordering decisions worth flagging for review:

• Per route group, not global. A global wrapper would become the underlying reader for gzipRequestBody's own MaxBytesReader, and the smaller of the two limits would silently win — capping ingest at 1 MiB.
• Ahead of requireAuth on /admin. An unauthenticated flood should be refused on size, not read in full and then rejected as unauthorized. Consequence: an oversized unauthenticated POST to /admin/* returns 413 rather than 401.

Tests cover the middleware directly and the router wiring, the latter driven through the real buildRouter with no store configured — so if the limiter is ever dropped from a route group, the request reaches a handler that panics on the nil store instead of returning 413. I verified that test fails when the middleware is removed.

Left alone deliberately

• The seven decode sites still return 400 rather than 413 when a chunked over-cap body trips the reader mid-decode. The Content-Length pre-check covers every ordinary client; making the rest exact means threading errors.As through seven handlers, which felt like the wrong trade on this PR.
• The limits are hardcoded, while every comparable bound in this server (rate_limit.*, session_ttl, stale_threshold, indexing interval) is configurable.

Follow-ups I'd file separately — not regressions from this PR

1. The runner never handles 413. livereport/reporter.go:165 treats any status ≥ 300 as a warn and re-posts the identical snapshot next tick, forever. A large EEST suite's Tests map plus the verbatim Config blob can genuinely cross 50 MiB decompressed, at which point every report fails, the stalewatcher deletes the live row, and the run disappears from the UI with only a runner-side WARN to explain it. There's no size guard on the producing side and no distinction between a permanent 4xx and a transient failure.

2. A report that passes the caps is stored whole and re-served on every poll. No field is length-bounded — TestIngest_LargeButUnderLimitGzipReportAccepted posts a ~50 MiB instance_id and asserts 204. That row lands in live_runs, and handleListLiveRuns re-marshals every live row on each request with no cache (unlike handleIndex). One such report makes every subsequent /index/live_runs poll a 50 MiB response until the row expires. Bounding the transport without bounding what gets persisted moves the cost from ingest to read.

3. io.ReadAll + json.Unmarshal double-buffers. Peak stays a multiple of the 50 MiB cap per concurrent request (matching the 156 MB you measured for a single one), and nothing bounds ingest concurrency since the limiter is again gated on rate_limit.enabled. json.NewDecoder(r.Body).Decode(&report) streams instead, and the payload_bytes log field the ReadAll exists for can come from a counting reader on the decompressed side.

Minor: TestIngest_LargeButUnderLimitGzipReportAccepted says it guards an off-by-one but sits 4 KiB under the cap, so it can't detect one — a body of exactly maxIngestDecompressedBytes is the case that distinguishes the implementations. And nothing pins the gzip-over-raw-cap path I probed above.

go test -race ./pkg/api/... passes, go vet and gofmt clean, golangci-lint run --new-from-rev=origin/master reports 0 issues.