Web scraping API benchmark

This repository contains benchmark results comparing major web scraping APIs across protected targets under standardized test conditions.

Companion code for the Zenrows article Best Web Scraping APIs in 2026 (Benchmarked).

Providers tested

• Zenrows
• Zyte
• ScraperAPI
• Scrapfly

Targets tested

• Zim
• Glassdoor
• Idealista
• Indeed

Benchmark methodology

Each provider executed:

• 100 requests per target
• Standard protected-target request configuration
• No custom header tuning
• No manual proxy rotation adjustments
• Standard extraction workflow

The benchmark focused on two primary metrics:

• Success rate
• Average response time

Raw results

Provider	Target	Success Rate	Avg Response Time (ms)	Total Requests
Zenrows	Zim	99%	10013.71	100
Zenrows	Glassdoor	100%	8849.47	100
Zenrows	Idealista	98%	10530.13	100
Zenrows	Indeed	99%	7054.59	100
Zyte	Zim	99%	15463.92	100
Zyte	Glassdoor	99%	6929.69	100
Zyte	Idealista	92%	14632.84	100
Zyte	Indeed	8%	20144.59	100
ScraperAPI	Zim	88%	45897.43	100
ScraperAPI	Glassdoor	25%	53518.78	100
ScraperAPI	Idealista	0%	10709.81	100
ScraperAPI	Indeed	83%	33676.04	100
Scrapfly	Zim	97%	11949.19	100
Scrapfly	Glassdoor	98%	8691.70	100
Scrapfly	Idealista	99%	11059.38	100
Scrapfly	Indeed	100%	11877.39	100

Multi-Target Consistency Summary

The following values represent the average success rate across all four protected targets under repeated-request execution.

Provider	Average Success Rate
Zenrows	99.00%
Scrapfly	98.50%
Zyte	74.75%
ScraperAPI	49.00%

This metric reflects repeated-request consistency across multiple protected domains rather than performance on any individual target.

Approximate protected-target resource consumption

The following values reflect approximate dashboard telemetry observed during benchmark execution. Values were normalized against 1,000 protected-target requests.

Provider	Benchmark Consumption Calculation	Approximate Protected-Target Cost
Zenrows	~$2.74 across 400 requests → (2.74 ÷ 400) × 1000	~$6.85 / 1K requests
Zyte	~$1.59 across 400 requests → (1.59 ÷ 400) × 1000	~$3.97 / 1K requests
ScraperAPI	~3055 credits across 400 requests → (3055 ÷ 400) × 1000	~7,600 credits / 1K requests ($3.72 / 1K requests)
Scrapfly	~16,160 credits across 400 requests → (16,160 ÷ 400) × 1000	~40,400 credits / 1K requests ($12.1 / 1K requests)

These values reflect protected-target configurations under repeated-request execution and may vary depending on retries, rendering behavior, proxy escalation, and browser-level anti-bot handling.

Summary

Zenrows

Zenrows delivered the strongest overall performance across all targets. It maintained consistently high success rates while also keeping response times relatively low.

Key results:

• 100% on Glassdoor
• 99% on Indeed
• 99% on Zim
• 98% on Idealista

Zenrows achieved the strongest overall balance between reliability and latency consistency. It also achieved the highest overall success rate in the benchmark.

Zyte

Zyte performed well on Zim and Glassdoor but struggled significantly on Indeed.

Key observations:

• 99% on Zim
• 99% on Glassdoor
• 92% on Idealista
• 8% on Indeed

Response times were also generally higher compared to other providers.

ScraperAPI

ScraperAPI produced mixed results depending on the target.

Key observations:

• 88% on Zim
• 83% on Indeed
• 25% on Glassdoor
• 0% on Idealista

The provider also experienced major latency increases under failure-heavy conditions.

Scrapfly

Scrapfly demonstrated strong consistency across all protected targets.

Key observations:

• 100% on Indeed
• 99% on Idealista
• 98% on Glassdoor
• 97% on Zim

Success-rate stability remained strong across all four targets, though response times were higher on average than Zenrows.

Overall observations

• Zenrows delivered the strongest overall success-rate and the fastest average response time in the benchmark.
• Scrapfly achieved the second strongest overall success-rate consistency.
• Zyte showed uneven performance depending on the target.
• ScraperAPI performed well on Zim but struggled heavily against stronger anti-bot protections.

Notes

These results reflect:

• Standard protected-target request configurations
• No manual optimization
• No custom fingerprint tuning or manual anti-bot optimization

Real-world production performance may improve with:

• Custom fingerprinting
• Session persistence
• Adaptive retry logic
• Proxy tuning
• Browser-level optimization

Running the benchmark

1. Clone the repository

git clone https://github.com/ZenRows/web-scraping-api-benchmark.git
cd web-scraping-api-benchmark

2. Install dependencies

pip install -r requirements.txt

3. Configure API keys

Copy .env.example to .env and add your provider credentials:

ZENROWS_API_KEY=your_zenrows_api_key
ZYTE_API_KEY=your_zyte_api_key
SCRAPERAPI_API_KEY=your_scraperapi_api_key
SCRAPFLY_API_KEY=your_scrapfly_api_key

4. Run the benchmark

python3 src/benchmark.py

For long-running benchmark sessions on macOS, you can prevent sleep interruptions with:

caffeinate -i python3 src/benchmark.py

Results files

Raw and summarised results are committed under results/:

results/
├── results-100req.csv   # one row per request
└── summary-100req.csv   # aggregated success rate and latency per provider/target

License

MIT