Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 14 additions & 2 deletions .agents/plugins/marketplace.json
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,18 @@
"displayName": "Raintree Technology"
},
"plugins": [
{
"name": "docpull",
"source": {
"source": "local",
"path": "./plugins/docpull"
},
"policy": {
"installation": "AVAILABLE",
"authentication": "ON_INSTALL"
},
"category": "Productivity"
},
{
"name": "hig-doctor",
"source": {
Expand All @@ -17,10 +29,10 @@
"category": "Developer Tools"
},
{
"name": "trellis",
"name": "raintree-standards",
"source": {
"source": "local",
"path": "./plugins/trellis"
"path": "./plugins/raintree-standards"
},
"policy": {
"installation": "AVAILABLE",
Expand Down
71 changes: 65 additions & 6 deletions plugins.lock.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,14 @@
{
"schemaVersion": 1,
"plugins": [
{
"name": "docpull",
"repository": "https://github.com/raintree-technology/docpull.git",
"ref": "v6.5.1",
"commit": "ca55534347234990c361da83571472e00d6f33da",
"sourcePaths": ["plugin"],
"version": "6.5.1"
},
{
"name": "hig-doctor",
"repository": "https://github.com/raintree-technology/hig-doctor.git",
Expand All @@ -10,12 +18,63 @@
"version": "2.0.3"
},
{
"name": "trellis",
"repository": "https://github.com/raintree-technology/trellis.git",
"ref": "v0.3.1",
"commit": "fef84a32dff8694c6e6aa726eb7aef42b70997ab",
"sourcePaths": ["plugin"],
"version": "0.3.1"
"name": "raintree-standards",
"repository": "https://github.com/raintree-technology/raintree.standards.git",
"ref": "v1.1.1",
"commit": "5f68ad5da92cc902fe4eb56d23120d6dd9d2d9dc",
"sourcePaths": [
".ruby-version",
".tool-versions",
"AGENTS.md",
"CHANGELOG.md",
"CODE_OF_CONDUCT.md",
"CONTRIBUTING.md",
"LICENSE.md",
"README.md",
"SECURITY.md",
"THIRD_PARTY_NOTICES.md",
"agents",
"ai",
"analytics",
"api",
"catalog.yaml",
"content",
"coverage.md",
"data",
"design",
"discovery",
"engineering",
"error-messages.md",
"foundations",
"governance",
"growth",
"index.md",
"integrations",
"knowledge",
"legal",
"llms.txt",
"marketing",
"media",
"operations",
"patterns",
"playbooks",
"privacy",
"product",
"profiles",
"roadmap.md",
"sales",
"schema",
"scripts",
"security",
"seo",
"source-register.yaml",
"templates",
"testing",
"web",
"writing",
"plugin"
],
"version": "1.1.1"
}
]
}
26 changes: 26 additions & 0 deletions plugins/docpull/.claude-plugin/plugin.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
{
"name": "docpull",
"version": "6.5.1",
"description": "Pull public web sources into Claude Code. Indexes static and server-rendered sites as local Markdown with conditional-GET caching, then exposes them as MCP tools. Local, browser-free, no API keys.",
"author": {
"name": "Raintree Technology",
"email": "support@raintree.technology",
"url": "https://raintree.technology"
},
"homepage": "https://github.com/raintree-technology/docpull",
"repository": "https://github.com/raintree-technology/docpull",
"license": "MIT",
"keywords": [
"web",
"web-extraction",
"documentation",
"docs",
"fetch",
"markdown",
"rag",
"mcp",
"local-first",
"source-packs",
"context-packs"
]
}
49 changes: 49 additions & 0 deletions plugins/docpull/.codex-plugin/plugin.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
{
"id": "docpull",
"name": "docpull",
"version": "6.5.1",
"description": "Pull public web sources into Codex as local, searchable Markdown through docpull's MCP server.",
"skills": "./skills/",
"mcpServers": "./.mcp.json",
"author": {
"name": "Raintree Technology",
"email": "support@raintree.technology",
"url": "https://raintree.technology"
},
"homepage": "https://github.com/raintree-technology/docpull",
"repository": "https://github.com/raintree-technology/docpull",
"license": "MIT",
"keywords": [
"web",
"web-extraction",
"documentation",
"docs",
"fetch",
"markdown",
"rag",
"mcp",
"local-first",
"source-packs",
"context-packs"
],
"interface": {
"displayName": "docpull",
"shortDescription": "Fetch public web sources into local, searchable Markdown.",
"longDescription": "docpull gives Codex a local MCP server for fetching public static and server-rendered web sources, indexing them as Markdown, searching cached pages, reading cited passages, and inspecting local context packs without API keys or browser automation.",
"developerName": "Raintree Technology",
"category": "Productivity",
"capabilities": [
"Fetch public HTTPS pages as Markdown",
"Index web sources locally",
"Search cached Markdown with regex",
"Read cited source passages",
"Inspect local context packs",
"Manage user-defined source aliases"
],
"websiteURL": "https://github.com/raintree-technology/docpull",
"brandColor": "#0F6B5D",
"composerIcon": "./assets/logo.svg",
"logo": "./assets/logo.svg",
"defaultPrompt": "Use docpull when I ask about a specific library, framework, SDK, API, website, or public source URL. Prefer cached sources first, fetch only the requested source, and cite source paths when answering."
}
}
8 changes: 8 additions & 0 deletions plugins/docpull/.mcp.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
{
"mcpServers": {
"docpull": {
"command": "docpull",
"args": ["mcp"]
}
}
}
105 changes: 105 additions & 0 deletions plugins/docpull/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
<p align="center">
<img
src="https://raw.githubusercontent.com/raintree-technology/docpull/main/docs/launch-assets/logo-square-light-400.png"
alt="DocPull"
width="112"
/>
</p>

# DocPull agent plugin

**Active plugin for developers using Codex or Claude Code.** Pull static and
server-rendered public web sources into an agent's local context with citations.

DocPull aligns core workflows across CLI, Python SDK, and MCP, with each surface
optimized for its user. See the [Surface Contract](../docs/surface-contract.md)
for the boundary between the plugin's MCP tools and the broader CLI/SDK.

## Install

The plugin wraps the `docpull` CLI. Install the MCP extra first:

```bash
pip install 'docpull[mcp]' # or: pipx install 'docpull[mcp]'
docpull --version # should print 6.5.1 or newer
docpull mcp --help
```

The plain `pip install docpull` does not include the MCP dependency.

In Claude Code:

```text
/plugin marketplace add raintree-technology/docpull
/plugin install docpull@docpull
```

In Codex, install the plugin from the configured marketplace or a local plugin
source. The plugin starts the `docpull mcp` stdio server and exposes the
`docpull-research` skill.

## Try one source

```text
> /web-add fastapi
> How does FastAPI handle dependency injection scoping?
```

Expected result: the agent searches the cached source and answers with attribution to the local
Markdown. The first crawl populates the cache; later reads stay local.

## What you get

<!-- docpull:mcp-tools:start -->
- **MCP server** (37 tools):
- Read: `fetch_url`, `list_sources`, `list_indexed`, `grep_docs`, `read_doc`, `pack_score`, `pack_diff`, `pack_citations`, `pack_entities`, `pack_search`, `pack_brief`, `graph_status`, `graph_query`, `graph_neighbors`, `validate_policy`, `explain_routes`, `serve_pack_status`
- Write: `render_url`, `ensure_docs`, `workflow_run`, `website_pack`, `brand_pack`, `product_pack`, `styleguide_pack`, `image_pack`, `screenshot_pack`, `policy_pack`, `relationship_pack`, `intelligence_bundle`, `refresh_pack`, `audit_pack`, `pack_prepare`, `graph_build`, `graph_refresh`, `export_pack`, `add_source`, `remove_source`
- All read tools advertise `readOnlyHint` so hosts that auto-approve safe tools won't prompt for them.
<!-- docpull:mcp-tools:end -->
- **Claude Code slash commands**:
- `/web-add <alias-or-url>` — fetch a web source into the local index.
- `/web-search <pattern> [source]` — regex-search cached Markdown and pull surrounding context for the top hits.
- `/web-list` — show what's cached, with last-fetched age.
- `/web-refresh <source>` — bypass the 7-day cache and re-fetch.
- `/web-remove <source> [--keep-cache]` — drop a user alias and its cached Markdown.
- `/docs-add`, `/docs-search`, `/docs-list`, `/docs-refresh`, and `/docs-remove` remain compatibility aliases for existing users.
- **Meta-skill** (`docpull-research`): teaches the agent *when* to reach for docpull — so you don't have to remember the tool exists every time you ask about a library, API, vendor, product page, or web source.

## Built-in source aliases

These are fetchable by name without any URL setup: `react`, `nextjs`, `tailwindcss`, `vite`, `hono`, `fastapi`, `express`, `anthropic`, `openai`, `langchain`, `supabase`, `drizzle`, `prisma`.

For anything else, pass an HTTPS URL: `/web-add https://www.python.org/blogs/`.

## Where fetched Markdown is cached

By default, fetched Markdown lives under `$XDG_DATA_HOME/docpull-mcp/docs/` (or `~/.local/share/docpull-mcp/docs/` on macOS/Linux). Override with `DOCPULL_DOCS_DIR` if you want it somewhere else (e.g. one cache per project).

## Limits and privacy

- 100% local. No telemetry. No remote services.
- The plugin only sends HTTP requests to the URLs you ask it to fetch.
- The User-Agent is `docpull/<version> (+https://github.com/raintree-technology/docpull)` — public, identifiable, robots.txt-respecting.
- JavaScript rendering is explicit rather than part of the default fetch path.
- The plugin supports research workflows; it does not certify that a source is
complete, current, or correct.

## Troubleshooting

| Symptom | Fix |
|---------------------------------------------|-----|
| MCP tools missing after install | Run `docpull mcp --help`. If it errors with "requires the 'mcp' package", reinstall with `pip install 'docpull[mcp]'`. |
| `/web-add fastapi` says "unknown source" | Run `mcp__docpull__list_sources()` to see current aliases. Use a URL instead. |
| Slow first fetch | Normal — the first crawl populates the cache. Later runs use the local cache and conditional requests. |
| Want to refresh stale sources | `mcp__docpull__ensure_docs(source="<alias>", force=true)`. |

## Deeper documentation

- [DocPull project guide](../README.md)
- [Surface contract](../docs/surface-contract.md)
- [CLI recipes](../docs/cli-recipes.md)
- [Security policy](../SECURITY.md)

## License

MIT — same as docpull itself. Source: <https://github.com/raintree-technology/docpull>.
11 changes: 11 additions & 0 deletions plugins/docpull/assets/logo.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
57 changes: 57 additions & 0 deletions plugins/docpull/commands/docs-add.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
---
description: Fetch a web source and make its Markdown searchable in this session. Accepts a built-in alias (e.g. "react"), an HTTPS URL, or "name url" to register a custom alias.
argument-hint: <alias> | <https-url> | <name> <https-url>
allowed-tools: mcp__docpull__ensure_docs, mcp__docpull__add_source, mcp__docpull__list_sources
---

# Add a source to this session

Compatibility alias: prefer `/web-add` for new web-source workflows.

The user wants to add a source to docpull's local Markdown index so it's searchable later via `/docs-search` (or directly via the `grep_docs` MCP tool).

User input: **$ARGUMENTS**

## How to handle the input

Inspect `$ARGUMENTS`:

1. **Empty or missing.** Reply with a one-line usage hint and stop:
`Usage: /docs-add <alias>, /docs-add <https-url>, or /docs-add <name> <https-url>. Run /docs-list to see what's already cached.`

2. **One token, no URL scheme** (e.g. `react`, `fastapi`).
- Treat as a built-in alias. Call `ensure_docs(source="<alias>")`. The default `rag` profile is right for most cases — only override if the user mentioned a specific profile.
- If the alias is unknown, the tool will return an error listing available aliases. In that case call `list_sources()` and suggest the closest match by edit distance, or recommend running `/docs-add <name> <url>` with the source URL.

3. **One token, an HTTPS URL** (starts with `https://`).
- Call `list_sources()` first so you can detect both built-in aliases and user-defined aliases, including sources that have not been fetched yet.
- Auto-derive an alias name from the hostname:
1. Take the hostname.
2. Strip a leading `docs.` or `www.` if present.
3. Take the first dot-separated label.
4. Lowercase it.
5. Examples: `https://docs.fastapi.tiangolo.com` → `fastapi`; `https://nextjs.org/docs` → `nextjs`; `https://example.com/api` → `example`.
- If the derived name already appears in `list_sources()`, tell the user and suggest the explicit `/docs-add <name> <url>` form so they pick a unique name. Do not call `add_source` for a derived name that already exists.
- Otherwise call `add_source(name=<derived>, url=<url>)` to register, then `ensure_docs(source=<derived>)` to fetch.

4. **Two tokens, second is an HTTPS URL** (`<name> <url>`).
- Validate the name is a sensible alias (alnum + `_ . -`, ≤128 chars). If not, ask for a cleaner name.
- Call `add_source(name=<name>, url=<url>)`. This intentionally updates an existing user-defined alias with the same name. If it returns "is a builtin source", tell the user that `add_source` refuses to shadow builtins by default (the agent shouldn't pass `force=true` here without explicit user consent).
- Then call `ensure_docs(source=<name>)` to fetch.

## After it succeeds

Report a one-line summary:
- Source name (alias used).
- Pages fetched (from the `ensure_docs` response — pages_fetched / pages_skipped / pages_failed).
- Suggest the next step: `/docs-search <pattern> [source]` or ask the agent to grep for something specific.

## After it fails

Show the error in plain language. Common cases:
- **Unknown built-in alias** → list a few suggestions from `list_sources`.
- **URL rejected** (HTTP, localhost, private IP) → tell the user docpull is HTTPS-only by design and won't fetch internal hosts; suggest a public source URL.
- **`add_source` refused a builtin** → tell the user the alias collides with a built-in; pick a different name.
- **Network / 4xx / 5xx during `ensure_docs`** → show the URL and status code; suggest checking network, the URL itself, or trying a different public path.

Do not use any tools beyond the ones listed in `allowed-tools`. Do not send filler messages while the fetch is running — let the tool output speak for itself.
Loading