Skip to content

Add guardrails against the aggregate endpoint's memory usage - #100

Merged
iandees merged 1 commit into
masterfrom
aggregate-endpoint-guardrails
Aug 26, 2026
Merged

iandees merged 1 commit into
masterfrom
aggregate-endpoint-guardrails

Conversation

@iandees

@iandees iandees commented Aug 26, 2026

Copy link
Copy Markdown
Member

Summary

  • Add an in-flight lock (Redis SETNX via the existing Flask-Caching cache) so only one worker builds a given (hash_digest, release, table_code) aggregate download at a time; concurrent requests/retries for the same download get a 409 instead of piling on redundant work.
  • Add a size cap in aggregate_decennial(): count the census blocks a geography joins to before running the expensive block-level query, and reject with 413 above MAX_SYNC_AGGREGATION_BLOCKS = 150,000.

Why

On 2026-08-26 a request for /1.0/aggregate/523ee393241af8bb996db584a704d913/dec_pl94_compare_2020_2010/P3 (an uploaded "2024 Census Counties" geography spanning 336,949 + 421,507 census blocks across both decennial vintages) was retried 3 times by the client in ~40 seconds while the backend was still building it. aggregate_decennial_comparison() holds two full block-level dataframes (with geometry) in memory at once with no streaming, so the concurrent duplicate attempts drove the box into heavy swapping, and gunicorn killed and respawned 3 workers over the next several minutes. MAX_SYNC_AGGREGATION_BLOCKS is chosen from the real distribution of existing uploads (p50 4,632 / p90 11,574 / p99 163,592 blocks) — it passes everything except this hash and one other similarly oversized upload that's never actually been exercised through this endpoint.

Test plan

  • Validated the new block-count SQL directly against production data (336,949 for the incident hash, 38,786 for a normal upload) via dokku postgres:connect.
  • Confirmed CACHE_TYPE=redis is set in production so the lock isn't a silent no-op, and that cachelib's RedisCache.add() uses an atomic SETNX.
  • Manual smoke test against a real request post-deploy (small geography still downloads normally; oversized geography returns 413; concurrent requests for the same hash return 409).

@iandees
iandees merged commit 222716f into master Aug 26, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant