scrape-do-python

A Python SDK for the Scrape.do web-scraping proxy API.

Built on httpx and pydantic v2, with strict request validation, automatic retries on gateway errors, sticky-session validation, and SDK-native lifecycle hooks.

Status

Check the Changelog for the latest changes and project status

Installation

pip install scrape-do-python

Quickstart

from scrape_do import ScrapeDoClient

# API Token pulled from SCRAPE_DO_API_KEY env variable
# Can also be provided via 'api_token' argument

with ScrapeDoClient() as client:
    response = client.get(
        "https://example.com",
        super=True,
        render=True,
        return_json=True,
        show_frames=True,
        )
    
    print(response.is_proxy_error)

    print(response.frames[0].url)
    
    print(response.remaining_credits)

Features

Type-Checked Request Parameters

Request parameters are fully type-checked and automatically validated via the RequestParameters pydantic model

Smart Routing

ScrapeDoClient.request() accepts either **api_kwargs, a pre-built RequestParameters, or a raw api.scrape.do URL for request parameters

Automatic Retries

ScrapeDoClient can automatically retry requests on Scrape.do gateway errors (429 / 502 / 510) with customizable backoff (static or callable)

Sticky-Session Validation

Supply a session_validator callback to detect proxy node rotations and raise RotatedSessionError

SDK-Native Event Hooks

request / response / retry lifecycle hooks, distinct from httpx's transport-level hooks.

Strongly-Typed Responses

ScrapeDoResponse exposes the parsed JSON envelope, browser action results, screenshots, and network/websocket logs.

Browser Automation

Pydantic models for Browser Actions providing validation and type-hinting for the playWithBrowser API parameter

Documentation

Full API Reference

Roadmap

See ROADMAP for the upcoming Async Client, Proxy-Mode Clients, Async-API Support, and Plugin Support

Contributing

Pull Requests, Bug Reports, and Feature Requests are all welcome.

See CONTRIBUTING for local setup, test commands, and PR conventions

Community

Participation is governed by our Code of Conduct. To privately report a security issue, see the Security Policy.

License

scrape-do-python is released under the MIT License