diff --git a/README.md b/README.md index 4ce064e..4fd37d3 100644 --- a/README.md +++ b/README.md @@ -139,9 +139,22 @@ zenrows batch cancel # stop an in-flight run ``` `jobs.jsonl` is one JSON object per line, each with a `url` (plus optional -per-line overrides like `js_render`, `premium_proxy`, `proxy_country`). Request -beta access from Zenrows to run in the cloud; until then, validate/estimate -locally or fan out with `zenrows fetch` per URL. +per-line overrides like `js_render`, `premium_proxy`, `proxy_country`, +`extract`). Request beta access from Zenrows to run in the cloud; until then, +validate/estimate locally or fan out with `zenrows fetch` per URL. + +**Extract in Batch.** Set `"extract": "auto"` on a line to run that URL through +Extract — structured data instead of raw HTML: + +```jsonl +{"url": "https://example.com/products", "extract": "auto", "external_id": "p1"} +{"url": "https://example.com/reviews", "extract": "auto", "external_id": "r1"} +``` + +An Extract task costs the same as a regular one (1 credit at base tier), so +`batch estimate` prices it correctly. Results for an Extract task carry two +keys: `html` (the raw page) and `parsed` (the structured data) — validate a +sample with `zenrows extract ` before running the full batch. ## 10. Browser Sessions diff --git a/agent-plugin/skills/batch-jobs/SKILL.md b/agent-plugin/skills/batch-jobs/SKILL.md index daab22c..e626ff2 100644 --- a/agent-plugin/skills/batch-jobs/SKILL.md +++ b/agent-plugin/skills/batch-jobs/SKILL.md @@ -30,7 +30,9 @@ Optional if exposed by the server: `batch_wait` to block until terminal status. 1. Validate the workflow on **one** URL with `scrape` or `extract`. 2. `batch_create` with the full task list (each task: `url` + optional overrides like `js_render`, `premium_proxy`, `proxy_country`, `mode`, - `autoparse`, `external_id`, `metadata`). + `autoparse`, `extract`, `external_id`, `metadata`). `extract: "auto"` + returns structured data instead of raw HTML, at the same credit cost as a + regular task; the result carries `html` and `parsed` keys. 3. Poll `batch_status` until terminal (`completed` / `stopped` / …). 4. Collect with `batch_results`. 5. `batch_cancel` if the user aborts or the run should stop early. diff --git a/skills/batch-jobs/SKILL.md b/skills/batch-jobs/SKILL.md index e6bc028..df81fe5 100644 --- a/skills/batch-jobs/SKILL.md +++ b/skills/batch-jobs/SKILL.md @@ -24,7 +24,21 @@ zenrows batch estimate jobs.jsonl # validate the spec + estimate credits `jobs.jsonl` is one JSON object per line, each with a `url` (plus optional per-task keys: `external_id`, `metadata`, and scrape params like `js_render`, -`premium_proxy`, `proxy_country`, `mode`, `autoparse`). +`premium_proxy`, `proxy_country`, `mode`, `autoparse`, `extract`). + +## Extract in Batch + +`"extract": "auto"` on a line runs that URL through Extract, returning +structured data instead of raw HTML. + +``` +{"url": "https://example.com/products", "extract": "auto", "external_id": "p1"} +``` + +An Extract task costs the same as a regular one (1 credit at base tier), so +`batch estimate` prices it correctly. The result carries two keys: `html` (the +raw page) and `parsed` (the structured data). Validate on a single URL with +`zenrows extract ` before running the full batch. ## Cloud (needs a key + beta access) ``` diff --git a/templates/batch-jsonl-pipeline/README.md b/templates/batch-jsonl-pipeline/README.md index 15d65f9..15f2d0b 100644 --- a/templates/batch-jsonl-pipeline/README.md +++ b/templates/batch-jsonl-pipeline/README.md @@ -3,7 +3,12 @@ Scaffold for a high-scale workload expressed as a JSONL job spec, run on the Zenrows **Batch**. One JSON object per line, each with a `url` (plus optional per-line overrides like `js_render`, `premium_proxy`, `proxy_country`, -`mode`, `autoparse`, and an `external_id` echoed back on each result). +`mode`, `autoparse`, `extract`, and an `external_id` echoed back on each +result). + +Set `"extract": "auto"` on a line to run that URL through Extract — structured +data instead of raw HTML, at the same credit cost as a regular task. Extract +results carry `html` and `parsed` keys. ## Local (no key) diff --git a/templates/batch-jsonl-pipeline/jobs.example.jsonl b/templates/batch-jsonl-pipeline/jobs.example.jsonl index 109e879..fcf1d20 100644 --- a/templates/batch-jsonl-pipeline/jobs.example.jsonl +++ b/templates/batch-jsonl-pipeline/jobs.example.jsonl @@ -1,3 +1,4 @@ {"url": "https://www.scrapingcourse.com/ecommerce/", "mode": "auto", "autoparse": true} {"url": "https://httpbin.io/html"} {"url": "https://example.com", "js_render": true, "premium_proxy": true} +{"url": "https://www.scrapingcourse.com/ecommerce/", "extract": "auto", "external_id": "extract-demo"}