feat(web): add page link discovery

README.md

@@ -1,8 +1,8 @@
 # Guion Web
 
 Guion Web is a Node.js web research toolkit. It provides Exa or Brave search,
-Context7 library documentation lookup, Sourcegraph public code search, and two
-page-fetch backends through a CLI, stdio MCP server, Pi extension, and DeepSeek
+Context7 library documentation lookup, Sourcegraph public code search, page-link
+discovery, and two page-fetch backends through a CLI, stdio MCP server, Pi extension, and DeepSeek
 Harness (DSH) integration: direct HTML-to-Markdown extraction and explicit
 `agent-browser` rendering for client-rendered pages on supported hosts.
 
@@ -44,14 +44,16 @@ document on stdout, which is useful for automation.
 web search --provider exa -- "Node AbortSignal"
 web fetch https://example.com/article --tree
 web fetch https://example.com/article --section introduction
+web links https://example.com/article --limit 50
 web docs resolve react
 web docs fetch /facebook/react --topic hooks --tokens 2000
 web sgraph --count 10 -- "repo:^github\\.com/nodejs/node$ AbortSignal"
 ```
 
 Use `--` before a search or Sourcegraph query that begins with a hyphen. `fetch`
 supports `--full`, `--tree`, and `--section`; long extracted documents default to
-a heading tree so a later request can retrieve a stable section ID.
+a heading tree so a later request can retrieve a stable section ID. `links` lists
+up to 100 unique HTTP(S) anchors from the original page DOM.
 
 ## MCP
 
@@ -63,10 +65,10 @@ web mcp
 web mcp --provider brave
 ```
 
-The server exposes five read-only tools: `search`, `fetch`, `docs_resolve`,
+The server exposes six read-only tools: `search`, `fetch`, `links`, `docs_resolve`,
 `docs_fetch`, and `source_search`. Its stdout is reserved for MCP protocol
 messages; diagnostics go to stderr. For a client-rendered page, explicitly call
-`fetch` with `render: "agent-browser"` and an integer `waitMs`; this optional
+`fetch` or `links` with `render: "agent-browser"` and an integer `waitMs`; this optional
 retry requires a host-installed executable and never happens automatically.
 
 ## Pi
@@ -77,11 +79,13 @@ Install the independently bundled Pi extension:
 pi install npm:@guionai/pi-web
 ```
 
-It registers `web_search`, `web_fetch`, `web_docs`, and `web_source_search` and calls
+It registers `web_search`, `web_fetch`, `web_links`, `web_docs`, and `web_source_search` and calls
 the bundled core in-process. Pi and TypeBox are peer dependencies supplied by
 the host; no CLI executable or MCP configuration is required. `web_fetch` uses
 direct fetch by default and can explicitly use `render: "agent-browser"` with
 an integer `waitMs` when its host provides that optional executable.
+`web_links` uses the same explicit rendering contract and lists HTTP(S) anchors
+from the original page DOM.
 
 ## DSH
 
@@ -93,11 +97,13 @@ dsh plugin --profile web add @guionai/dsh-web
 
 The included profile patch routes stock PTC web search through the selected Exa
 or Brave provider. Its settings UI stores provider selection and manages
-namespaced write-only credentials. Fetch, documentation, and Sourcegraph tools
+namespaced write-only credentials. Fetch, link discovery, documentation, and Sourcegraph tools
 also run in-process. The host DSH packages and React are peers supplied by DSH.
 `web_fetch` uses direct fetch by default and can explicitly use
 `render: "agent-browser"` with an integer `waitMs` on a host that supplies the
 optional executable.
+`web_links` uses the same explicit rendering contract and lists HTTP(S) anchors
+from the original page DOM.
 
 ## Page-fetch backends
 
@@ -114,11 +120,16 @@ web fetch https://example.com/app --render=agent-browser --wait=2000
 web fetch https://example.com/app --render=agent-browser --wait=10000
 ```
 
+`web links` uses the same direct or explicit browser-rendered source, but parses
+the original DOM rather than Defuddle output so navigation and other links outside
+the readable article remain discoverable. It returns only HTTP(S) `a[href]`
+destinations, deduplicated and capped at 100 by default.
+
 `--wait` is mandatory with `--render=agent-browser`, including `--wait=0`, and
 accepts only an integer from 0 through 30,000 milliseconds. Direct `fetch`
-requests must not provide `--wait`. The same `render: "agent-browser"` and required
-`waitMs` fields are available on the MCP `fetch`, Pi `web_fetch`, and DSH
-`web_fetch` tools. A direct-fetch failure may return the structured
+or `links` requests must not provide `--wait`. The same `render: "agent-browser"` and required
+`waitMs` fields are available on the MCP `fetch`/`links`, Pi `web_fetch`/`web_links`, and DSH
+`web_fetch`/`web_links` tools. A direct-fetch failure may return the structured
 `javascript_rendering_may_be_required` hint with the 2,000 ms suggestion; the
 agent decides whether to retry with a longer wait or abandon the page.
 

packages/dsh-web/README.md

@@ -18,7 +18,7 @@ settings expose only configured/source/writable metadata.
 
 The published package is a dual host/browser bundle. Its host and client
 artifacts, profile patch, and exact DSH `0.1.0-rc.8` peer contract are included
-in the npm package. Search, direct page fetch, optional agent-browser rendering,
+in the npm package. Search, direct page fetch, page-link discovery, optional agent-browser rendering,
 Context7 documentation, and Sourcegraph all run in-process through the bundled
 Guion Web core. `web_fetch` has two page-fetch backends: direct fetch (the
 default) and explicit `render: "agent-browser"` with required `waitMs` (an
@@ -32,6 +32,11 @@ be directly runnable from `PATH` without a shell. The renderer is supported on
 macOS and Linux, is not an npm dependency, and never reuses persistent browser
 state or credentials.
 
+`web_links` lists up to 100 unique HTTP(S) anchors from the original page DOM,
+so it includes navigation and other links that readable-content extraction drops.
+It uses the same direct default and explicit `render: "agent-browser"` / required
+`waitMs` contract as `web_fetch`.
+
 Rendered requests are bounded and constrained to the requested hostname,
 `*.<requested-hostname>` (the target and its subdomains), and this fixed common
 CDN list: `cdn.jsdelivr.net`, `unpkg.com`, `cdnjs.cloudflare.com`,

packages/dsh-web/src/tools.ts

@@ -1,13 +1,17 @@
 import {
   createWebOperations,
+  DEFAULT_LINK_LIMIT,
   normalizeDocsToolInput,
   formatSize,
+  MAX_LINK_LIMIT,
   truncateHead,
   type Context7Credentials,
   type DocsFetchResult,
   type DocsResolveResult,
   type DocsToolInput,
   type FetchResult,
+  type LinksInput,
+  type LinksResult,
   type SGraphResult,
   type WebOperations,
 } from "@guionai/web-core";
@@ -62,6 +66,29 @@ const fetchParameters = {
   },
 } as const;
 
+const linksParameters = {
+  url: {
+    type: "string",
+    required: true,
+    description: "HTTP or HTTPS URL to inspect",
+  },
+  limit: {
+    type: "integer",
+    default: DEFAULT_LINK_LIMIT,
+    description: `Maximum links to return (1-${MAX_LINK_LIMIT})`,
+  },
+  render: {
+    type: "string",
+    enum: ["fetch", "agent-browser"],
+    description: "Page-fetch backend; defaults to direct fetch",
+  },
+  waitMs: {
+    type: "integer",
+    description:
+      "Required post-load wait for agent-browser rendering (0-30000)",
+  },
+} as const;
+
 const docsParameters = {
   action: {
     type: "string",
@@ -131,6 +158,38 @@ const fetchOutput = {
   ],
 };
 
+const linksOutput = {
+  schema: {
+    type: "object",
+    additionalProperties: false,
+    properties: {
+      url: { type: "string", required: true },
+      links: {
+        type: "array",
+        required: true,
+        items: {
+          type: "object",
+          additionalProperties: false,
+          properties: {
+            text: { type: "string", required: true },
+            url: { type: "string", required: true },
+          },
+        },
+      },
+      truncated: { type: "boolean", required: true },
+    },
+  } as const,
+  render: (_args: unknown, value: LinksResult) => [
+    {
+      type: "text" as const,
+      text: boundedToolText(
+        formatLinks(value),
+        "Use web_fetch to read a selected destination.",
+      ),
+    },
+  ],
+};
+
 const docsOutput = {
   schema: {
     oneOf: [
@@ -234,6 +293,49 @@ function requireString(input: unknown, field: string): string {
   return input[field];
 }
 
+function normalizeLinks(input: unknown): LinksInput {
+  if (!isRecord(input)) throw new Error("web_links input must be an object");
+  rejectUnknownFields(input, Object.keys(linksParameters), "web_links");
+  const url = requireString(input, "url");
+  const limit = input.limit;
+  if (
+    limit !== undefined &&
+    (typeof limit !== "number" ||
+      !Number.isInteger(limit) ||
+      limit < 1 ||
+      limit > MAX_LINK_LIMIT)
+  )
+    throw new Error(
+      `limit must be an integer from 1 through ${MAX_LINK_LIMIT}`,
+    );
+
+  const render = input.render;
+  const waitMs = input.waitMs;
+  if (render !== undefined && render !== "fetch" && render !== "agent-browser")
+    throw new Error('render must be "fetch" or "agent-browser"');
+  if (render !== "agent-browser") {
+    if (waitMs !== undefined)
+      throw new Error("waitMs is only valid with render agent-browser");
+  } else {
+    if (waitMs === undefined)
+      throw new Error("waitMs is required with render agent-browser");
+    if (
+      typeof waitMs !== "number" ||
+      !Number.isInteger(waitMs) ||
+      waitMs < 0 ||
+      waitMs > 30_000
+    )
+      throw new Error("waitMs must be an integer from 0 through 30000");
+  }
+
+  return {
+    url,
+    ...(limit !== undefined ? { limit } : {}),
+    ...(render !== undefined ? { render } : {}),
+    ...(waitMs !== undefined ? { waitMs } : {}),
+  };
+}
+
 function normalizeDocs(input: unknown): DocsToolInput {
   if (!isRecord(input)) throw new Error("web_docs input must be an object");
   return normalizeDocsToolInput(input);
@@ -270,6 +372,15 @@ function formatDocsResolve(result: DocsResolveResult): string {
   return `Found ${result.libraries.length} libraries:\n${result.libraries.map((library) => `- ${library.id}: ${library.title}`).join("\n")}`;
 }
 
+function formatLinks(result: LinksResult): string {
+  if (result.links.length === 0) return "No HTTP(S) links found.";
+  const lines = result.links.map(
+    (link, index) =>
+      `${index + 1}. ${link.text || "(no text)"}\n   URL: ${link.url}`,
+  );
+  return `Found ${result.links.length} link${result.links.length === 1 ? "" : "s"}${result.truncated ? " (truncated)" : ""}:\n\n${lines.join("\n\n")}`;
+}
+
 function webFetchTool(
   dependencies: WebToolDependencies,
   operations: WebOperations,
@@ -301,6 +412,25 @@ function webFetchTool(
   );
 }
 
+function webLinksTool(
+  dependencies: WebToolDependencies,
+  operations: WebOperations,
+): ToolDefinition {
+  return strictDefinition(
+    defineTool({
+      name: "web_links",
+      description:
+        "List HTTP(S) links from a page. Use direct fetch for static pages, or explicit agent-browser rendering with waitMs for client-rendered pages.",
+      parameters: linksParameters,
+      output: linksOutput,
+      isConcurrencySafe: () => true,
+      async execute(args, exec) {
+        return operations.links(normalizeLinks(args), exec.signal);
+      },
+    }),
+  );
+}
+
 function webDocsTool(
   dependencies: WebToolDependencies,
   operations: WebOperations,
@@ -369,10 +499,11 @@ function webSgraphTool(
 
 export function createWebToolDefinitions(
   dependencies: WebToolDependencies,
-): readonly [ToolDefinition, ToolDefinition, ToolDefinition] {
+): readonly [ToolDefinition, ToolDefinition, ToolDefinition, ToolDefinition] {
   const operations = dependencies.operations ?? createWebOperations();
   return [
     webFetchTool(dependencies, operations),
+    webLinksTool(dependencies, operations),
     webDocsTool(dependencies, operations),
     webSgraphTool(dependencies, operations),
   ];

packages/dsh-web/test/artifact.test.ts

@@ -92,7 +92,7 @@ if (command === "open" && args.some((value) => value.includes("/blocked"))) {
   process.exit(1);
 }
 if (command === "eval")
-  console.log(JSON.stringify({ success: true, data: { result: "<html><body><article><p>Packed DSH rendered fixture.</p></article></body></html>" } }));
+  console.log(JSON.stringify({ success: true, data: { result: JSON.stringify({ html: "<html><body><article><p>Packed DSH rendered fixture.</p></article></body></html>", url: "https://93.184.216.34/rendered" }) } }));
 else console.log(JSON.stringify({ success: true, data: {} }));
 `,
   );
@@ -195,6 +195,11 @@ describe("DSH rc.8 packed package contract", () => {
           "<html><body><article><p>Packed DSH browserless fixture.</p></article></body></html>",
           { headers: { "content-type": "text/html" } },
         );
+      if (String(url) === "https://93.184.216.34/links")
+        return new Response(
+          '<html><body><nav><a href="/destination">Packed link</a></nav></body></html>',
+          { headers: { "content-type": "text/html" } },
+        );
       expect(String(url)).toBe("https://api.exa.ai/search");
       expect(
         (init.headers as Headers).get?.("x-api-key") ??
@@ -236,13 +241,28 @@ describe("DSH rc.8 packed package contract", () => {
       );
       if (!fetchTool)
         throw new Error("packed DSH artifact did not register web_fetch");
+      const linksTool = tools.find(
+        (definition) => definition.name === "web_links",
+      );
+      if (!linksTool)
+        throw new Error("packed DSH artifact did not register web_links");
       process.env.PATH = `${browser.bin}:${originalPath ?? ""}`;
       const direct = await fetchTool.execute(
         { url: "https://93.184.216.34/direct", full: true },
         { signal: new AbortController().signal },
       );
       expect(direct.content).toBe("Packed DSH browserless fixture.\n");
       expect(() => readFileSync(browser.log, "utf8")).toThrow();
+      const links = await linksTool.execute(
+        { url: "https://93.184.216.34/links" },
+        { signal: new AbortController().signal },
+      );
+      expect(links.links).toEqual([
+        {
+          text: "Packed link",
+          url: "https://93.184.216.34/destination",
+        },
+      ]);
       const rendered = await fetchTool.execute(
         {
           url: "https://93.184.216.34/rendered",