Skip to content

fix(paperless-ngx): widen email cron to */10 to reduce double-ingest#39

Open
ClaydeCode wants to merge 1 commit into
mainfrom
fix/clayde/paperless-email-cron-race
Open

fix(paperless-ngx): widen email cron to */10 to reduce double-ingest#39
ClaydeCode wants to merge 1 commit into
mainfrom
fix/clayde/paperless-email-cron-race

Conversation

@ClaydeCode

Copy link
Copy Markdown
Contributor

Problem

Fixes #30.

On Paperless NGX, an email attachment gets ingested twice. The first ingest succeeds; the second always fails and raises a failure alert in the Paperless UI — the document actually imported fine, so users are confused by an error that means nothing.

Root cause

apps/paperless-ngx/docker-compose.yml.template overrode the mail-check cron to every minute:

- PAPERLESS_EMAIL_TASK_CRON=* * * * *

Paperless's mail-check task has no overlap guard, and it writes the message to its ProcessedMail dedup table only after the attachment is consumed (post-OCR). So a mail-check fired by the next cron tick can pass the dedup .exists() check and consume the same attachment again before the first pass recorded it. First import succeeds; the second is rejected by document-hash duplicate detection → the failure alert.

Change

One line: widen the interval to */10 * * * * — Paperless upstream's default.

Because the dedup record lands only after OCR completes, the race window can span minutes; 10 minutes gives a comfortable margin over a single fetch+OCR pass while email ingestion stays acceptably responsive. The prior every-minute value was chosen for responsiveness (per git history) but is the aggressive knob that triggers the race.

Scope note: this mitigates, not eliminates. A genuine fix needs an upstream overlap lock; and the user-configured multi-rule case (two mail rules matching one message, which upstream treats as not-a-bug) is out of the template's control.

Verification

  • python3 build_store_data.py exits 0 (generated zip/metadata are untracked; only the template changed).
  • No other file or doc references the cron value.
  • Behavioural repro (send one attachment, watch for two consume tasks) requires a live shard and is not runnable in CI.

Review panel

Adversarial reviewer (always-on) — no blocking findings. Advisories, all addressed:

  • "*/5 reduces but doesn't eliminate the race; dedup is written post-OCR so the window is minutes-long — prefer upstream default */10 for 2x margin."Fixed: adopted */10 instead of the originally-committed */5.
  • "Commit says 'stop double-ingest' — overclaim, cron tuning only mitigates."Fixed: reworded commit + PR to "reduce/mitigate" and documented the residual race and the multi-rule case.
  • "Issue says second ingest 'always' fails; a pure overlap race is intermittent — could be a second mechanism (multi-rule match / post-OCR dedup write)."Noted: the post-OCR dedup write makes the window minutes-long, which explains near-deterministic failure under an every-minute cron; the multi-rule case is user-config and out of template scope. Called out in the scope note above.

No other specialist (test / DevEx / security / DB / API / UX) was dispatched — a one-line config value change touches none of those lenses.


🤖 Generated with Claude Code

The every-minute PAPERLESS_EMAIL_TASK_CRON (`* * * * *`) races with
itself: Paperless's mail-check task has no overlap guard, and it writes
the message to the ProcessedMail dedup table only after the attachment
is consumed (post-OCR). So a second mail-check fired by the next cron
tick can pass the dedup `.exists()` check and consume the same
attachment again before the first pass has recorded it. The first
import succeeds; the second is rejected by Paperless's document-hash
duplicate detection and surfaces as a failure alert users find
confusing.

Widen the interval to `*/10` — Paperless upstream's default. Because
the dedup record is written only after OCR completes, the race window
can span minutes; 10 minutes gives a comfortable margin over a single
mail-fetch+OCR pass while email ingestion stays acceptably responsive.
This mitigates, not eliminates, the race: a genuine fix would need an
upstream overlap lock, and the user-configured multi-rule case (two
mail rules matching one message) is out of the template's control.

Fixes #30

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Paperless: email attachments ingested twice; second ingest always fails and alerts, confusing users

1 participant