feat(tooling-ux): minimal 3-tool surface with progressive discovery + escalation

docs/probabilistic-policy.md

@@ -787,3 +787,51 @@ This work is a specific instance of a more general claim about LLM-driven system
 > Most performance engineering today uses hand-tuned heuristics where a real probabilistic policy would do better. The constraint that justified hand-tuning (microsecond budgets, no idle time) doesn't bind in LLM-driven workflows because the LLM itself is the slow part. Everything below the LLM has spare budget for inference that previous-generation systems couldn't afford.
 
 unbrowser is one place to demonstrate this pattern. If it works here, the same shape — frequentist hot path, Bayesian policy layer, persistent priors per workload class — applies to many systems-level decisions in LLM-adjacent infrastructure.
+
+---
+
+## 13. Tool-invocation routing (the same frame, one layer up)
+
+The posteriors above decide what *unbrowser* does internally (run scripts?
+settle? call an API?). The identical math governs the other half of the
+system: which tool the *agent* reaches for next. Every agent decision is
+
+```
+P(call T next | evidence)
+```
+
+and each layer of the stack is one update:
+
+| Update | Mechanism | Owner |
+|---|---|---|
+| Prior | tool name + description (training-data semantics) | us, at design time |
+| Prior shift | MCP `instructions` field at handshake | server |
+| Evidence | `derive_tool_likelihoods()` — page features → per-tool scores | binary |
+| Posterior | `next_tools[]` with confidences | smart layer |
+| MAP estimate | `micro_hint` | smart layer |
+| Suppression | `avoid[]` — hard-absence evidence zeroes mass | smart layer |
+| Ambiguity | `tool_entropy.h` — flat distribution ⇒ "gather info", not argmax | smart layer |
+| New label | `report_outcome` binds success/failure to navigation_id | driver |
+
+Design rules that fall out (all shipped in the smart layer as of 0.0.20-dev):
+
+1. **Calibration beats correctness.** A hint that fires when it shouldn't
+   trains agents to ignore hints. Every advisory branch gates on positive
+   evidence (a table is data only with ≥8 `<td>` cells; a layout table on a
+   docs page must not route to `extract_table`).
+2. **Negative advice saves more than positive advice.** Each avoided call is
+   a full round-trip plus failed-parse cost. `avoid[]` emits only on hard
+   structural absence (no JSON scripts, no tables, no forms), never
+   speculation.
+3. **Suppress argmax under ambiguity.** When the next_tools distribution is
+   flat (normalized entropy > 0.85), `micro_hint` is withheld and the bundle
+   says so — argmax over noise is how hints lose trust.
+4. **Phase B closes the loop.** `report_outcome` labels feed a Beta-Bernoulli
+   per `(page_shape_bucket × tool)` pair; hand-tuned likelihood weights
+   become learned ones. Same sample-efficiency argument as Appendix B: two
+   inspectable parameters per cell, no RL.
+
+The name/description prior is why tool naming is a probabilistic decision,
+not an aesthetic one (`open` routes intent better than `navigate_auto`) —
+but priors are the *smallest* lever we control. The evidence, suppression,
+and calibration layers are where the accuracy actually comes from.

python/pyproject.toml

@@ -46,9 +46,13 @@ solver = ["unchainedsky-cli"]
 # `pip install pyunbrowser` installs the primary `unbrowser` script and a
 # `pyunbrowser` compatibility alias. Both dispatch to the same CLI; the alias
 # makes the Registry-derived `uvx pyunbrowser --mcp` command executable.
+# `unbrowser-smart` is the stdio MCP server for the minimal-3 smart surface
+# (search/open/help) — configured in MCP hosts as `"command": ["unbrowser-smart"]`.
 [project.scripts]
 unbrowser = "unbrowser._cli:main"
 pyunbrowser = "unbrowser._cli:main"
+unbrowser-smart = "unbrowser.smart_mcp:main"
+pyunbrowser-smart = "unbrowser.smart_mcp:main"
 
 # The wheel ships the platform-specific native binary inside unbrowser/_bin/.
 # CI builds the binary first (cargo build --release for each target), copies

python/unbrowser/__init__.py

@@ -15,6 +15,12 @@
 For the `extract` / auto-strategy command, watchdog-bounded `exec_scripts`,
 the cookie handoff for bot-walled sites, and the BlockMap shape: see the
 project README at https://github.com/protostatis/unbrowser.
+
+Smart wrapper (minimal 3-tool progressive discovery): `SmartClient` adds
+`search(query)` (Brave → DDG fallback), `navigate_auto(url, goal=)`
+(open + bounded discover/cards + `escalation`/`micro_hint`/`next_tools`),
+and `help(topic)` (grouped catalog). Runs as an MCP server via the
+`unbrowser-smart` console script or `python -m unbrowser.smart_mcp`.
 """
 
 from __future__ import annotations
@@ -247,6 +253,10 @@ def search(self, query: str, engine: str = "ddg") -> dict:
             bing  — Bing search. Tracker links in results are auto-decoded
                     on click (the binary detects bing.com/ck/a?u=... URLs
                     and follows to the real destination).
+            brave — Brave Search HTML via unbrowser. Prefer the SmartClient
+                    wrapper (``from unbrowser.smart import SmartClient``) for
+                    a parsed ``[{title,url,snippet}]`` result; this base
+                    method returns the raw navigate result for brave as well.
 
         Google is intentionally NOT supported via the cheap path — Google's
         search page returns ~no useful HTML without JS, so it would silently
@@ -260,9 +270,11 @@ def search(self, query: str, engine: str = "ddg") -> dict:
             url = "https://duckduckgo.com/html/?q=" + quote_plus(query)
         elif engine == "bing":
             url = "https://www.bing.com/search?q=" + quote_plus(query)
+        elif engine == "brave":
+            url = "https://search.brave.com/search?q=" + quote_plus(query) + "&source=web"
         else:
             raise UnbrowserError(
-                f"unknown search engine '{engine}'. Supported: ddg, bing. "
+                f"unknown search engine '{engine}'. Supported: ddg, bing, brave. "
                 "Google is intentionally unsupported via the cheap path."
             )
         return self.navigate(url)
@@ -439,3 +451,20 @@ def navigate(url: str, exec_scripts: bool = False, shim_mode: str | None = None)
     """
     with Client(shim_mode=shim_mode) as ub:
         return ub.navigate(url, exec_scripts=exec_scripts)
+
+
+# Lazy re-export of SmartClient (guard against circular import: smart.py imports
+# Client/UnbrowserError from this module, so the import has to come last).
+try:  # pragma: no cover - import guard
+    from .smart import SmartClient as SmartClient
+
+    __all__ = [
+        "Client",
+        "UnbrowserError",
+        "SmartClient",
+        "find_binary",
+        "navigate",
+        "__version__",
+    ]
+except ImportError:  # pragma: no cover - smart.py missing (source checkout/old wheel)
+    pass

python/unbrowser/_cli.py

@@ -5,12 +5,17 @@
 agents and MCP hosts can use directly (e.g. `command: "unbrowser"` in
 .mcp.json).
 
-The wrapper keeps the native binary as the execution engine and exposes a
-useful `--help` surface. Invocations are passed through to the binary.
+The wrapper keeps the native binary as the execution engine. Help follows
+progressive-disclosure conventions (clig.dev): `--help` shows the core path
+plus grouped tool families; `unbrowser help <topic>` drills into any family
+or tool; unknown commands get did-you-mean suggestions on stderr with
+exit code 2.
 """
 
 from __future__ import annotations
 
+import difflib
+import json
 import os
 import subprocess
 import sys
@@ -19,42 +24,137 @@
 from . import find_binary
 
 
+# Grouped tool families — mirrors HELP_CATALOG in unbrowser/smart.py and the
+# Rust MCP surface. Kept as plain data so `--help` renders without importing
+# the smart layer.
+TOOL_FAMILIES: dict[str, list[str]] = {
+    "reading": ["text", "text_main", "text_clean", "blockmap", "body"],
+    "query": ["query", "query_debug", "query_text", "find_text", "text_around"],
+    "extraction": ["extract", "extract_table", "extract_list", "extract_cards", "table_to_json"],
+    "discovery": ["discover", "route_discover", "page_model", "network_extract", "network_stores"],
+    "interaction": ["click", "type", "submit", "activate", "settle", "eval"],
+    "session": ["cookies_set", "cookies_get", "cookies_clear", "report_outcome"],
+}
+
+_KNOWN_COMMANDS = [
+    "navigate", "search", "open", "help", "exec", "session",
+    "router", "cookie-service", "policy-check", "--mcp", "--version",
+    "--list-profiles", "--prefit-info",
+]
+
+
 def _usage() -> None:
+    fams = "\n".join(f"  {fam:<12} {' '.join(tools)}" for fam, tools in TOOL_FAMILIES.items())
     print(
-        """unbrowser
+        f"""unbrowser — web access for LLM agents. One static binary. No Chrome.
 
-Usage:
-  unbrowser session start [--id <id>] [--profile <name>] [--policy=blocklist] [--shims stable|enhanced]
-  unbrowser session exec [--pretty] <id|socket> <method> [params-json | shorthand args]
+START HERE
+  unbrowser navigate <url> [--exec-scripts]    fetch a page -> low-token BlockMap
+  unbrowser search "<query>" [--count N]       web search (Brave->DDG) -> [{{title,url,snippet}}]
+  unbrowser open <url> [--goal G]              fetch + auto-discover + next-step hints
+  unbrowser --mcp                              MCP server mode for agent hosts
+
+MULTI-STEP SESSIONS (cookies + last page persist)
+  unbrowser session start [--id <id>] [--profile <name>] [--policy=blocklist]
   unbrowser exec [--pretty] <id|socket> <method> [params-json | shorthand args]
-  unbrowser session stop <id|socket>
-  unbrowser session list
-  unbrowser session prune
-  unbrowser navigate <url> [--exec-scripts] [--json] [--events] [--shims stable|enhanced]
-  unbrowser router <url> [--cookie-service <url>] [--allow-remote-cookie-service] [--no-auto-cookie-service]
-  unbrowser cookie-service [--headless|--no-headless] [--port <port>] [--allow-host <host>] [--allow-remote-bind]
+  unbrowser session stop <id|socket> | session list | session prune
+
+TOOLS — call via `unbrowser exec <id> <method> '{{...}}'`, or over MCP
+{fams}
+
+  unbrowser help <family|tool>    details + examples (e.g. `unbrowser help extraction`)
+
+MORE
+  unbrowser router <url>                 bot-wall cookie handoff via local Chrome
+  unbrowser cookie-service [--headless]  local solver service (needs [solver] extra)
   unbrowser policy-check <url> [<url>...]
-  unbrowser --list-profiles
-  unbrowser --prefit-info
-  unbrowser [--profile <name>] [--policy=blocklist] [--shims stable|enhanced] [--mcp]
-  unbrowser --version
-
-Examples:
-  unbrowser session start --id demo
-  unbrowser exec demo navigate https://news.ycombinator.com
-  unbrowser exec --pretty demo blockmap
-  unbrowser session stop demo
-  unbrowser navigate https://news.ycombinator.com --json
-  unbrowser cookie-service --headless --profile unbrowser-cookie-service
-  unbrowser router https://example.com/protected
-  unbrowser policy-check https://www.bbc.com/news
-  printf '{\"id\":1,\"method\":\"navigate\",\"params\":{\"url\":\"https://news.ycombinator.com\"}}\n' | unbrowser
-
-`navigate` delegates to the native binary; output is always the binary's JSON.
+  unbrowser --list-profiles | --prefit-info | --version
+
+Every result carries routing hints: micro_hint (the next concrete step),
+next_tools (ranked candidates), avoid (tools with nothing to act on).
 """
     )
 
 
+def _help_topic(topic: str | None) -> int:
+    """Render the grouped catalog, one family, or one tool. Exit 0."""
+    try:
+        from .smart import HELP_CATALOG
+    except ImportError:
+        print("help catalog unavailable in this install", file=sys.stderr)
+        return 1
+    if not topic:
+        for fam, tools in HELP_CATALOG.items():
+            print(f"{fam}:")
+            for name, info in tools.items():
+                print(f"  {name:<16} {info.get('when', '')}")
+        print("\nDrill in: unbrowser help <family|tool>  e.g. unbrowser help extract_table")
+        return 0
+    t = topic.lower()
+    for fam, tools in HELP_CATALOG.items():
+        if t == fam:
+            print(f"{fam}:")
+            for name, info in tools.items():
+                print(f"\n  {name}\n    {info.get('when', '')}")
+                if info.get("example"):
+                    print(f"    e.g. {info['example']}")
+            return 0
+        if t in tools:
+            info = tools[t]
+            print(f"{t}  ({fam})\n  {info.get('when', '')}")
+            if info.get("example"):
+                print(f"  e.g. {info['example']}")
+            return 0
+    # fuzzy fallback
+    matches = difflib.get_close_matches(t, [n for f_ in HELP_CATALOG.values() for n in f_], n=3)
+    if matches:
+        print(f"unknown topic '{topic}'. Did you mean: {', '.join(matches)}?")
+    else:
+        print(f"unknown topic '{topic}'")
+    return 1
+
+
+def _suggest_and_exit(bad: str) -> None:
+    matches = difflib.get_close_matches(bad, _KNOWN_COMMANDS + [n for f_ in TOOL_FAMILIES.values() for n in f_], n=3)
+    hint = f" Did you mean: {', '.join(matches)}?" if matches else ""
+    print(f"unbrowser: unknown command '{bad}'.{hint}\nRun `unbrowser --help` to see what's available.", file=sys.stderr)
+    raise SystemExit(2)
+
+
+def _cmd_search(args: list[str]) -> None:
+    count = 5
+    if "--count" in args:
+        i = args.index("--count")
+        count = int(args[i + 1])
+        del args[i : i + 2]
+    query = " ".join(a for a in args if not a.startswith("-"))
+    if not query:
+        print("usage: unbrowser search \"<query>\" [--count N]", file=sys.stderr)
+        raise SystemExit(2)
+    from .smart import SmartClient
+
+    with SmartClient() as ub:
+        hits = ub.search(query, count=count)
+    print(json.dumps(hits, indent=2))
+
+
+def _cmd_open(args: list[str]) -> None:
+    goal = None
+    if "--goal" in args:
+        i = args.index("--goal")
+        goal = args[i + 1]
+        del args[i : i + 2]
+    url = next((a for a in args if not a.startswith("-")), None)
+    if not url:
+        print("usage: unbrowser open <url> [--goal G]", file=sys.stderr)
+        raise SystemExit(2)
+    from .smart import SmartClient
+
+    with SmartClient() as ub:
+        bundle = ub.navigate_auto(url, goal=goal)
+    print(json.dumps(bundle, indent=2))
+
+
 def _is_help_flag(arg: str) -> bool:
     return arg in {"-h", "--help"}
 
@@ -102,6 +202,17 @@ def main() -> None:
         _usage()
         return
 
+    if argv[0] == "help":
+        raise SystemExit(_help_topic(argv[1] if len(argv) > 1 else None))
+
+    if argv[0] == "search":
+        _cmd_search(argv[1:])
+        return
+
+    if argv[0] == "open":
+        _cmd_open(argv[1:])
+        return
+
     if argv[0] == "navigate":
         _navigate(argv[1:])
         return
@@ -114,9 +225,12 @@ def main() -> None:
         _router(argv[1:])
         return
 
-    binary = find_binary()
-    # Preserve the native binary behavior for every other command.
-    os.execv(binary, ["unbrowser", *argv])
+    # Pass through known binary commands and flags; anything else gets a
+    # did-you-mean instead of a cryptic binary error.
+    if argv[0].startswith("-") or argv[0] in {"session", "exec", "policy-check"}:
+        binary = find_binary()
+        os.execv(binary, ["unbrowser", *argv])
+    _suggest_and_exit(argv[0])
 
 
 if __name__ == "__main__":