Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 16 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -139,9 +139,22 @@ zenrows batch cancel <job-id> # stop an in-flight run
```

`jobs.jsonl` is one JSON object per line, each with a `url` (plus optional
per-line overrides like `js_render`, `premium_proxy`, `proxy_country`). Request
beta access from Zenrows to run in the cloud; until then, validate/estimate
locally or fan out with `zenrows fetch` per URL.
per-line overrides like `js_render`, `premium_proxy`, `proxy_country`,
`extract`). Request beta access from Zenrows to run in the cloud; until then,
validate/estimate locally or fan out with `zenrows fetch` per URL.

**Extract in Batch.** Set `"extract": "auto"` on a line to run that URL through
Extract — structured data instead of raw HTML:

```jsonl
{"url": "https://example.com/products", "extract": "auto", "external_id": "p1"}
{"url": "https://example.com/reviews", "extract": "auto", "external_id": "r1"}
```

An Extract task costs the same as a regular one (1 credit at base tier), so
`batch estimate` prices it correctly. Results for an Extract task carry two
keys: `html` (the raw page) and `parsed` (the structured data) — validate a
sample with `zenrows extract <url>` before running the full batch.

## 10. Browser Sessions

Expand Down
4 changes: 3 additions & 1 deletion agent-plugin/skills/batch-jobs/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,9 @@ Optional if exposed by the server: `batch_wait` to block until terminal status.
1. Validate the workflow on **one** URL with `scrape` or `extract`.
2. `batch_create` with the full task list (each task: `url` + optional
overrides like `js_render`, `premium_proxy`, `proxy_country`, `mode`,
`autoparse`, `external_id`, `metadata`).
`autoparse`, `extract`, `external_id`, `metadata`). `extract: "auto"`
returns structured data instead of raw HTML, at the same credit cost as a
regular task; the result carries `html` and `parsed` keys.
3. Poll `batch_status` until terminal (`completed` / `stopped` / …).
4. Collect with `batch_results`.
5. `batch_cancel` if the user aborts or the run should stop early.
Expand Down
16 changes: 15 additions & 1 deletion skills/batch-jobs/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,21 @@ zenrows batch estimate jobs.jsonl # validate the spec + estimate credits

`jobs.jsonl` is one JSON object per line, each with a `url` (plus optional
per-task keys: `external_id`, `metadata`, and scrape params like `js_render`,
`premium_proxy`, `proxy_country`, `mode`, `autoparse`).
`premium_proxy`, `proxy_country`, `mode`, `autoparse`, `extract`).

## Extract in Batch

`"extract": "auto"` on a line runs that URL through Extract, returning
structured data instead of raw HTML.

```
{"url": "https://example.com/products", "extract": "auto", "external_id": "p1"}
```

An Extract task costs the same as a regular one (1 credit at base tier), so
`batch estimate` prices it correctly. The result carries two keys: `html` (the
raw page) and `parsed` (the structured data). Validate on a single URL with
`zenrows extract <url>` before running the full batch.

## Cloud (needs a key + beta access)
```
Expand Down
7 changes: 6 additions & 1 deletion templates/batch-jsonl-pipeline/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,12 @@
Scaffold for a high-scale workload expressed as a JSONL job spec, run on the
Zenrows **Batch**. One JSON object per line, each with a `url` (plus
optional per-line overrides like `js_render`, `premium_proxy`, `proxy_country`,
`mode`, `autoparse`, and an `external_id` echoed back on each result).
`mode`, `autoparse`, `extract`, and an `external_id` echoed back on each
result).

Set `"extract": "auto"` on a line to run that URL through Extract — structured
data instead of raw HTML, at the same credit cost as a regular task. Extract
results carry `html` and `parsed` keys.

## Local (no key)

Expand Down
1 change: 1 addition & 0 deletions templates/batch-jsonl-pipeline/jobs.example.jsonl
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
{"url": "https://www.scrapingcourse.com/ecommerce/", "mode": "auto", "autoparse": true}
{"url": "https://httpbin.io/html"}
{"url": "https://example.com", "js_render": true, "premium_proxy": true}
{"url": "https://www.scrapingcourse.com/ecommerce/", "extract": "auto", "external_id": "extract-demo"}
Loading