Go crawler daemon that scrapes supported book sources, publishes crawl events to Redis, and downloads cover images to shared local storage.
- Redis-driven crawl orchestration
- Concurrent scraping, Redis publishing, and image downloading
- Structured JSON logging with
slog - Redis-backed dedupe and distributed crawl locking
- Event-driven architecture using Redis queues
- Shared filesystem storage for downloaded covers
kniga.lvazon.marketmnogoknig.comchitai-gorod.ru
- Go 1.27+
- A running Redis instance
- An images directory shared with
libri-api
Set these environment variables before running:
REDIS_URL=redis://localhost:6379
IMAGES_DIR=/path/to/images
LOG_LEVEL=info # optional: debug | info | warn | errorRun the crawler daemon:
make runThe crawler listens for commands from Redis queue:
crawler:commands
and publishes crawl events to:
crawl:events
Push a crawl command into Redis:
{
"crawlId": 1,
"source": "kniga.lv"
}Example using redis-cli:
redis-cli LPUSH crawler:commands \
'{"crawlId":1,"source":"kniga.lv"}'To crawl multiple sources, enqueue multiple commands.
The crawler publishes all events via Redis LPUSH into the crawler:events
queue.
Fired when a new book is successfully scraped.
{
"type": "book",
"crawlId": 1,
"book": {
// ...
}
}Fired periodically to broadcast the current count of books found during the crawl.
{
"type": "progress",
"crawlId": 1,
"booksFound": 42
}Fired when the crawling process finishes successfully.
{
"type": "completed",
"crawlId": 1,
"booksFound": 120
}Fired if the crawl job is explicitly aborted before finishing.
{
"type": "cancelled",
"crawlId": 1,
"booksFound": 15
}Fired when a global or system-level error stops the crawl entirely (e.g., source locking issues).
{
"type": "error",
"crawlId": 1,
"error": "source mnogoknig.com is already running"
}Fired when a non-fatal or page-specific error occurs while processing a specific URL.
{
"type": "crawl_error",
"crawlId": 1,
"error": "failed to parse page body: timeout",
"url": "https://example.com/book/123"
}Fired periodically to signal that the crawler instance is alive and actively working on the task.
{
"type": "heartbeat",
"crawlId": 1
}Only one crawl per source can run at a time. Duplicate crawl requests are rejected and emitted as error events.
Build the binary:
make buildClean build output:
make cleanSee CONTRIBUTING.md.