WIP: scrapy + multiprocessing#7

Open

lhaarman wants to merge 26 commits into

mainfrom

experimental/scrapy_multiprocessing

Commits on May 20, 2026

Add configuration for skippable domains such as social media
lhaarman
committed

Commits on May 21, 2026

Basic setup scrapy with multiprocessing; TODO testing
lhaarman
committed

Commits on May 28, 2026

temp commit; saving result
lhaarman
committed

Commits on Jun 3, 2026

Refactoring, timeout, language/country filtering, schema.org in hesitant spider
lhaarman
committed

Commits on Jun 4, 2026

add logs dir
lhaarman
committed

Commits on Jun 9, 2026

Commits on Jun 24, 2026

Split up targeting between netloc/path
lhaarman
committed

Commits on Jun 29, 2026

add missing config keys

dominikblatt
authored and
lhaarman
committed
adjust testing code of hesitant spider

dominikblatt
authored and
lhaarman
committed
remove unused code

dominikblatt
authored and
lhaarman
committed
fix imports for -m flag running

dominikblatt
authored and
lhaarman
committed
move scrapy modules into scrape

dominikblatt
authored and
lhaarman
committed
move schema parser to parse module

dominikblatt
authored and
lhaarman
committed
update normalize_url documentation

dominikblatt
authored and
lhaarman
committed
remove unused lines

dominikblatt
authored and
lhaarman
committed
rename main scripts

dominikblatt
authored and
lhaarman
committed
update requirements.txt

dominikblatt
authored and
lhaarman
committed

Commits on Jul 1, 2026