Production fixes from legacy migration - #105
Merged
Merged
Conversation
werkzeug 3 removed the legacy salted-sha256 hash method, so check_password_hash raises ValueError on any account whose stored hash predates the move to pbkdf2:sha256 — every login attempt on such an account returned a raw 500, locking it out entirely. - utils/passwords.py (new): verify_password() checks current pbkdf2 hashes via werkzeug and legacy sha256$salt$digest hashes via the stdlib HMAC primitive old werkzeug used (validated against werkzeug 2.2.3, the version that wrote them). Malformed/None hashes read as invalid credentials instead of raising. - Route all three login handlers (/auth/login, /api/login, /admin/login) through it; on a successful legacy-hash login the stored hash is transparently rewritten to pbkdf2:sha256, so users keep their existing passwords with no forced reset. - Tests: legacy round-trip vector, migration on all three handlers, wrong password leaves the hash untouched, malformed/None hashes can never 500 a login.
Users routinely open verify/reset emails well after they arrive; a 15-minute window made valid links read as broken. An hour matches common practice for reset links. Named constant + a test pinning the lifetime. Note: tokens are stateless and not single-use, so this is also the replay window for an already-used link — fine at an hour, revisit if we ever make links single-use.
Email tokens (verify / reset / join-org) were stateless JWTs, so a leaked or already-used link kept working until it expired — a window the previous commit widened to an hour. - create_email_token stamps a jti; consume_email_token records it in the new used_email_token table on first successful use (the unique constraint is the gate, so concurrent uses can't both win; expired rows are pruned opportunistically) - all four consuming handlers (/api/reset_password, /api/verify_token, /auth/verify/<token>, /auth/reset/<token> POST) refuse replays; consumption happens after domain checks pass and before any mutation - replayed verify links render as verified-successfully when the account is already verified (re-clicks, mail-scanner prefetch); reset links only burn on the form POST, so scanner GETs are harmless - tokens minted before jti stamping are refused; they age out within the one-hour lifetime of a deploy - alembic a9e5d21c7b48 adds used_email_token; migration parity and downgrade covered by the existing migration smoke test
tippecanoe opens temporary files scaled to the host core count plus
per-tile geometry files. On a many-core data host a large folder's tile
build exceeds Docker's default soft nofile limit of 1024 and tippecanoe
aborts with EMFILE ("Too many open files") — deterministically, so
Celery's retries fail identically and the filing is left with imported
fabric but no tiles. Small builds stay under the limit, which is why
the failure looked intermittent.
Verified on the data host: the exact failing tippecanoe command
succeeds with the soft limit at 65536 (well under the 524288 hard cap).
Recreating the worker container applies it; failed folders then just
need a map regeneration.
run_tippecanoe captured stderr but only printed it after a successful run: with check=True the CalledProcessError raised before the print, so on the one path where stderr matters it was discarded. A prod tile failure stored only "returned non-zero exit status 1" as the task result and took shell access on the worker to diagnose (the real error was EMFILE — see the deploy ulimit commit). Failures now raise RuntimeError carrying the tail of tippecanoe's actual stderr, which is what lands in celerytaskinfo.result, and log it; successful runs log stderr (progress/counts) instead of print.
The fabric debt in submission_debts cleared as soon as a fabric file row existed — but that row is committed synchronously at upload, before (and regardless of whether) the worker import succeeds. A failed import left the badge saying "Ready to submit" on a filing whose fabric never loaded, i.e. a submittable filing with no fabric-derived served locations. The gate now requires fabric_data rows to exist for the folder's fabric files — the imported locations are the source of truth. When the file row exists but no rows have landed, the latest Fabric task's status shapes the message: PENDING/STARTED/RETRY reads "import in progress", anything else "import failed — retry or re-upload". Either way the filing is not ready.
shaddi
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A collection of fixes from prod that we hit when a legacy user went to do a new filing: