From d963c7c80b48a9d34b54812c8570b00a365cc353 Mon Sep 17 00:00:00 2001 From: Seth Wheeler <23089578+Megapixel99@users.noreply.github.com> Date: Mon, 17 Aug 2026 00:48:35 -0500 Subject: [PATCH] Link the write-up from the README The published article links here through its repo field; nothing here pointed back. Anyone arriving at the repo had no way to find the account of why the politeness limiter took three attempts. Also removes crawler-politeness-draft.md, the draft that became that article, which has nothing the article does not. Its only unique lines are the original title, the two Hacker News paragraphs cut before publishing, and a closing pointer to this repo that the post now handles through its own metadata. Keeping a stale copy of a live article in the repo it describes is worse than not having it. Co-Authored-By: Claude Opus 5 --- README.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/README.md b/README.md index caa722d..a72d55a 100644 --- a/README.md +++ b/README.md @@ -12,6 +12,11 @@ library. > engine by implementing each one myself, not to compete with a production > engine. See [Limitations](#limitations). +**Write-up:** [Rate Limiting a Crawler Across Node Cluster +Workers](https://sethwheeler.dev/blog/crawler-rate-limiter/) — the politeness +limiter took three attempts, and the first two enforced a perfectly correct +delay while the crawled host received a hundred simultaneous requests. + --- ## How it works