Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .github/plugin/marketplace.json
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,13 @@
},
"metadata": {
"description": "MrScraper plugin for GitHub Copilot CLI.",
"version": "0.1.0"
"version": "0.2.0"
},
"plugins": [
{
"name": "mrscraper",
"description": "Retrieve public web pages as HTML, Markdown, or screenshots and discover pages through Google search with MrScraper's hosted MCP server. Includes skills that keep the raw page as the source for local extraction and check account usage.",
"version": "0.1.0",
"description": "Retrieve public web pages as HTML, Markdown, or screenshots and discover pages through web search with MrScraper's hosted MCP server. Includes skills that keep the raw page as the source for local extraction and check account usage.",
"version": "0.2.0",
"source": "./plugins/mrscraper"
}
]
Expand Down
15 changes: 15 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,20 @@
# Changelog

## 0.2.0 - 2026-11-02

- Reworded search as web search in the plugin and marketplace descriptions,
README, and skills.
- Updated the `mrscraper-serp` skill for MrScraper MCP server 0.3.0, which
moved `webSearch` to MrScraper's new web search endpoint. Results are a
ranked list (rank, title, url, description, source); `region` sets the
results' country and the new `num_results` asks for 10 to 100 results. The
`raw`, `format`, `language`, `page`, and `render_js` options are gone, and
so is the search results URL example.
- Corrected the scopes in the `mrscraper` skill and README: `scraperAPI` needs
`scrape:write`, `webSearch` needs `scrape:read`, and `status` needs
`account:read`.
- The skills now match the MrScraper Claude plugin 0.3.0 apart from link style.

## 0.1.0 - 2026-10-16

- First release of the MrScraper plugin for GitHub Copilot CLI, in the Agent
Expand Down
14 changes: 7 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

MrScraper connects [GitHub Copilot CLI](https://docs.github.com/copilot/how-tos/copilot-cli)
to a hosted MCP server that retrieves public web pages as HTML, Markdown, or
screenshots, discovers pages through Google search, and reports account usage.
screenshots, discovers pages through web search, and reports account usage.
The plugin also includes three skills that tell the agent which tool to use and
to keep the raw page content as the source for summaries and local extraction.

Expand All @@ -25,8 +25,8 @@ public pages. This plugin helps when a task needs more control over retrieval:
to appear before the content is read.
- Content that differs by country, with retries and a per-request token cap.
- A screenshot of the page saved as an image file.
- Google results as structured JSON (organic results, related questions, and
pagination) for a chosen region, language, and results page.
- Web search results as a ranked list (title, link, description, and source)
for a chosen country, up to 100 at a time.

It retrieves pages; it does not drive a browser session. For clicking, form
filling, or signed-in flows, use Playwright or another browser automation tool.
Expand Down Expand Up @@ -75,11 +75,11 @@ keys into the chat.
| Tool | Purpose |
| --- | --- |
| `scraperAPI` | Retrieve a known page as HTML, Markdown, a screenshot, or a combination. |
| `webSearch` | Discover public pages through Google. |
| `webSearch` | Discover public pages through web search. |
| `status` | Inspect subscription status, token usage, and remaining tokens. |

The OAuth scopes are `scrape:read` (`scraperAPI` and `webSearch`) and
`account:read` (`status`).
The OAuth scopes are `scrape:write` (`scraperAPI`), `scrape:read`
(`webSearch`), and `account:read` (`status`).

## Try it

Expand All @@ -88,7 +88,7 @@ Fetch https://www.scrapethissite.com/pages/simple/ as Markdown and list the five
```

```text
Search Google for open source web scraping libraries and compare the top five results.
Search the web for open source web scraping libraries and compare the top five results.
```

```text
Expand Down
4 changes: 2 additions & 2 deletions plugins/mrscraper/plugin.json
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
{
"$schema": "https://agent-plugins.org/schemas/1.0.0/plugin.schema.json",
"name": "mrscraper",
"version": "0.1.0",
"description": "Retrieve public web pages as HTML, Markdown, or screenshots and discover pages through Google search with MrScraper's hosted MCP server. Includes skills that keep the raw page as the source for local extraction and check account usage.",
"version": "0.2.0",
"description": "Retrieve public web pages as HTML, Markdown, or screenshots and discover pages through web search with MrScraper's hosted MCP server. Includes skills that keep the raw page as the source for local extraction and check account usage.",
"author": {
"name": "MrScraper",
"email": "support@mrscraper.com",
Expand Down
79 changes: 25 additions & 54 deletions plugins/mrscraper/skills/mrscraper-serp/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
name: mrscraper-serp
description: |
Discover public pages through Google with MrScraper's webSearch tool using a search query or Google search URL. Use when no target URL is known or the user asks to inspect Google results. After discovery, retrieve selected pages with mrscraper-fetch.
Discover public pages through web search with MrScraper's webSearch tool. Use when no target URL is known or the user asks to inspect web search results. After discovery, retrieve selected pages with mrscraper-fetch.
---

# Discover Pages with MrScraper MCP Web Search
Expand All @@ -13,86 +13,58 @@ status, or broader routing.

## Step 1 — Build the Query

Include the important entity, attribute, location, or date in url, and set
raw=false to receive parsed results:
Pass the query in url. Include the important entity, attribute, location, or
date:

{
"url": "web scraping best practices",
"raw": false
"url": "web scraping best practices"
}

The input can also be a complete Google search URL:
The input can also be a search results URL; its q parameter becomes the query.

{
"url": "https://www.google.com/search?q=web+scraping&gl=us&hl=en&start=20",
"raw": false
}

For a Google URL, the tool reads q, gl, hl, and start. Explicit region,
language, and page inputs take precedence.

## Step 2 — Set Locale and Pagination
## Step 2 — Set the Country and Result Count

Set both country and language when results must match a locale:
Set region to a two-letter country code when results must come from one
country. It defaults to us:

{
"url": "running shoes",
"raw": false,
"region": "id",
"language": "id",
"page": 2
}

Start with the first page unless the user requests broader coverage. Inspect
the current page before requesting another.

## Step 3 — Choose the Output

raw defaults to true, which returns result-page HTML as the Python SDK does.
Set raw=false for parsed JSON; organic results are in data.data.organic, next
to sections such as people_also_ask and pagination.

Request result-page HTML when the page itself is needed:

{
"url": "iphone 17"
"region": "id"
}

Use JavaScript rendering for dynamic result features such as AI Overview:
The search returns 10 results by default. Ask for more, up to 100, only when
the user needs broader coverage:

{
"url": "what is web scraping",
"raw": false,
"render_js": true
"url": "running shoes",
"region": "id",
"num_results": 30
}

### Parameters

| MCP input | Default | Request mapping | Use |
| --- | --- | --- | --- |
| url | required | Body query | Search query or complete Google search URL. |
| raw | true | Body format=html | Return result-page HTML; set false to use format. |
| format | json | Body format | json or html when raw=false. |
| region | URL value or omitted | Body region | Google result country, such as us or id. |
| language | URL value or omitted | Body language | Google result language, such as en or id. |
| page | URL value or omitted | Body page | Positive, one-based results page. |
| render_js | false | Body renderJs | Render JavaScript-dependent result features. |
| url | required | Body query | Search query, or a search results URL whose q value is used. |
| region | us | Body country | Two-letter result country, such as us or id. |
| num_results | 10 | Body numResults | Number of results, from 10 to 100; fewer may come back. |
| timeout | 600 | MCP upstream deadline | Maximum wait in seconds; not sent in the search body. |

Authentication is handled by the MCP client's OAuth 2.1 connection and is
never a tool input.

## Step 4 — Inspect the Results
## Step 3 — Inspect the Results

The complete response envelope is available in MCP structuredContent and as
formatted JSON text. Prefer structuredContent. Check isError, error, and
status_code before using data.
status_code before using data. The results are a ranked list in data.data;
each result has rank, title, url, description, and source.

When the user requests a search artifact, save the successful data or full
envelope with local file-writing capability; webSearch has no output-path
input.

## Step 5 — Continue with Relevant URLs
## Step 4 — Continue with Relevant URLs

Select only URLs relevant to the user's goal, then load
`mrscraper-fetch` to retrieve and inspect each
Expand All @@ -102,15 +74,14 @@ Run independent follow-up URLs in parallel when the environment supports safe
parallel execution. Keep the number of pages proportional to the requested
coverage. If the user only requested URL discovery, stop after the search.

## Step 6 — Handle Weak or Missing Results
## Step 5 — Handle Weak or Missing Results

- Set both region and language when the locale is wrong.
- Set region when the results come from the wrong country.
- Rewrite noisy queries around the important entity, attribute, and location.
- Request another result page only after checking the current page.
- Retry once with render_js=true when a dynamic result feature is required.
- Raise num_results only after checking the first results.
- Increase timeout when the MCP request deadline is too short.
- Use `mrscraper` when the connection or authentication
fails.

webSearch is for public Google discovery, not for discovering URLs within one
webSearch is for public web search, not for discovering URLs within one
known site.
16 changes: 8 additions & 8 deletions plugins/mrscraper/skills/mrscraper/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
---
name: mrscraper
description: |
Connect, authenticate, route, and troubleshoot MrScraper MCP, and check account usage with the status tool. For known public URLs, retrieve and preserve page content with mrscraper-fetch first; use mrscraper-serp for query-first discovery through Google.
Connect, authenticate, route, and troubleshoot MrScraper MCP, and check account usage with the status tool. For known public URLs, retrieve and preserve page content with mrscraper-fetch first; use mrscraper-serp for query-first discovery through web search.
---

# MrScraper MCP

Use this skill for connection and authentication checks, tool selection, shared
response handling, account status, and troubleshooting. Load the focused fetch
or SERP skill for page retrieval and Google search.
or SERP skill for page retrieval and web search.

## Step 1 — Verify or Connect MrScraper MCP

Expand Down Expand Up @@ -47,9 +47,9 @@ Never place credentials in tool arguments, print them, write them into project
artifacts, or manually reproduce the OAuth exchange. Do not configure an
API-key or bearer-token fallback from these skills.

The connection uses two scopes: scrape:read for scraperAPI and webSearch, and
account:read for status. A tool called without its scope returns 403
insufficient_scope; reauthorize the connection so the client requests it.
The connection uses three scopes: scrape:write for scraperAPI, scrape:read for
webSearch, and account:read for status. A tool called without its scope returns
403 insufficient_scope; reauthorize the connection so the client requests it.

Verify end-to-end access with status, which reads the account without
retrieving a page:
Expand All @@ -65,7 +65,7 @@ complete.
| User outcome | MCP tool or skill |
| --- | --- |
| Retrieve and preserve a known public page as HTML, Markdown, or a screenshot | `mrscraper-fetch` with scraperAPI |
| Discover relevant pages from a Google query | `mrscraper-serp` with webSearch, then retrieve selected pages |
| Discover relevant pages from a web search query | `mrscraper-serp` with webSearch, then retrieve selected pages |
| Check the plan, token usage, or remaining tokens | status |

### Retrieve first, extract locally
Expand Down Expand Up @@ -158,8 +158,8 @@ account usage, not the progress of a request.
replaced by scraperAPI, webSearch, and status.
- Page incomplete or missing dynamic content: load mrscraper-fetch and revise
page-loading inputs deliberately.
- Search results weak: load mrscraper-serp and refine the query, locale, or
page.
- Search results weak: load mrscraper-serp and refine the query, country, or
number of results.

## Limits

Expand Down
Loading