Skip to content

Add configurable --delay throttle for API requests - #48

Open
Chisomnwa wants to merge 1 commit into
Open-Book-Genome-Project:mainfrom
Chisomnwa:fix/throttle-configurable-delay
Open

Add configurable --delay throttle for API requests#48
Chisomnwa wants to merge 1 commit into
Open-Book-Genome-Project:mainfrom
Chisomnwa:fix/throttle-configurable-delay

Conversation

@Chisomnwa

@Chisomnwa Chisomnwa commented Jul 30, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds a configurable --delay parameter to backfill_tags.py to throttle API requests during batch backfill, preventing rate limiting from Open Library.

What Changed

backfill_tags.py: Added --delay argument (float, default 0.5s) that controls the pause between API fetch requests during backfill. This lets callers adjust throttle rate per-run instead of editing source code.

Motivation

During production backfill planning, @jimchamp warned that unthrottled requests may be blocked by OL's rate limiting. This change adds a safe default delay (0.5s) while keeping it configurable per-run — use --delay 0.1 for faster runs or --delay 1.5 when being extra cautious.

Related

  • Prerequisite for Phase 2c production backfill issue #46
  • @mekarpeles previously suggested configurable delays over hardcoded values PR #40 review

Stakeholders

@mekarpeles @jimchamp

@jimchamp

Copy link
Copy Markdown

Noting that, as per our API usage guidelines, only 1 request per second can be made with no User-Agent. I don't believe that User-Agent can be set for requests made with the client (as per this open issue), so it may be good to set the default delay to one second.

@Chisomnwa

Copy link
Copy Markdown
Collaborator Author

Thanks @jimchamp, that makes sense. I'll update the defaults to 1 second delay and batch size 100.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants