Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .cargo/config.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
[target.x86_64-pc-windows-msvc]
rustflags = ["-C", "target-feature=+crt-static"]
36 changes: 36 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,16 @@ jobs:
unzip -Z1 dist/lexmount-webfetch.zip | grep -qx 'SKILL.md'
! unzip -Z1 dist/lexmount-webfetch.zip | grep -q '^lexmount-webfetch/'
! unzip -Z1 dist/lexmount-webfetch.zip | grep -q '^bin/'
expected_files="$(printf '%s\n' \
'SKILL.md' \
'references/authentication.md' \
'references/commands.md' \
'references/troubleshooting.md' \
'scripts/bootstrap.ps1' \
'scripts/bootstrap.sh' \
'scripts/doctor.ps1' \
'scripts/doctor.sh')"
test "$(unzip -Z1 dist/lexmount-webfetch.zip)" = "$expected_files"

windows-bootstrap:
runs-on: windows-latest
Expand All @@ -82,3 +92,29 @@ jobs:
$env:LEXMOUNT_WEBFETCH_CLI_INSTALL_DIR = Join-Path $env:RUNNER_TEMP "webfetch-cli-bootstrap"
& .\skills\lexmount-webfetch\scripts\bootstrap.ps1
& (Join-Path $env:LEXMOUNT_WEBFETCH_CLI_INSTALL_DIR "webfetch-cli.exe") version

windows-release:
runs-on: windows-latest
steps:
- uses: actions/checkout@v5
- uses: dtolnay/rust-toolchain@stable
with:
targets: x86_64-pc-windows-msvc
- run: cargo test --locked --target x86_64-pc-windows-msvc
- run: cargo build --release --locked --target x86_64-pc-windows-msvc
- name: Verify standalone Windows executable
shell: powershell
run: |
$binary = "target/x86_64-pc-windows-msvc/release/webfetch-cli.exe"
& $binary version
if ($LASTEXITCODE -ne 0) { throw "webfetch-cli version failed with exit code $LASTEXITCODE" }

$vswhere = "${env:ProgramFiles(x86)}\Microsoft Visual Studio\Installer\vswhere.exe"
$installationPath = & $vswhere -latest -property installationPath
Import-Module "$installationPath\Common7\Tools\Microsoft.VisualStudio.DevShell.dll"
Enter-VsDevShell -VsInstallPath $installationPath -SkipAutomaticLocation
$dependencies = & dumpbin.exe /dependents $binary
$dependencies | Write-Output
if ($dependencies -match '(?im)^\s+(?:VCRUNTIME|MSVCP)\d*[^\s]*\.dll\s*$') {
throw "Windows release dynamically depends on the Visual C++ Redistributable"
}
2 changes: 1 addition & 1 deletion Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[package]
name = "lexmount-webfetch"
version = "0.1.2"
version = "0.1.3"
edition = "2024"
license = "MIT"
description = "Native Rust SDK and CLI for Lexmount WebFetch"
Expand Down
36 changes: 24 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,11 @@ cargo build --release
```

Credentials come from `LEXMOUNT_API_KEY`, `LEXMOUNT_PROJECT_ID`, optional
`LEXMOUNT_WEBFETCH_BASE_URL`, or `webfetch-cli auth login`. PKCE login stores
credentials at `~/.config/lexmount/webfetch-cli/credentials.json` with mode
`0600` on Unix and never prints the API key.
`LEXMOUNT_WEBFETCH_BASE_URL`, or the Skill-local CLI's `auth login --open` flow.
PKCE login stores credentials at
`~/.config/lexmount/webfetch-cli/credentials.json` with mode `0600` on Unix and
never prints the API key. Pass `--client-name "<name>"` to identify the calling
Agent on the approval page, or omit it to use `Agent`.

## Use

Expand All @@ -25,21 +27,31 @@ Markdown is the default agent-readable output. `--format text` returns plain
text, `--format json` returns a compact response with quality warnings, and
`--format json-full` preserves the API response for debugging.

## WorkBuddy package
## Agent Skill package

The publishable Skill is in `skills/lexmount-webfetch`. Build a deterministic,
direct-upload SkillHub ZIP with:
The publishable Skill is in `skills/lexmount-webfetch`. Build a deterministic
release ZIP with:

```bash
./scripts/package-skill.sh
```

The Skill ZIP contains `SKILL.md`, references, and platform bootstrap scripts;
native executables are published separately. On first use, the matching script
downloads the pinned release from Tencent Cloud COS and verifies its SHA-256
digest. Tagged releases publish the Skill ZIP, `SHA256SUMS`, and exactly two
standalone binaries: signed and notarized macOS ARM64 plus Windows x64. Linux
and macOS Intel are not release platforms.
The Agent host installs the complete ZIP at its selected Skill root; the ZIP
root is the Skill root. Skill installation and status are host responsibilities,
so the Rust CLI does not provide `skill install` or `skill status`. Agents
resolve bundled scripts and binaries from the directory containing the loaded
`SKILL.md`: Codex uses the absolute source path supplied in Skill metadata,
Claude Code uses `${CLAUDE_SKILL_DIR}`, and WorkBuddy/CodeBuddy uses
`${CODEBUDDY_SKILL_DIR}`. Once started, the bootstrap and doctor scripts locate
the Skill directory from their own path.

The ZIP contains exactly eight files: `SKILL.md`, three references, and the
bootstrap/doctor scripts for both platforms. Native executables are published
separately. On first use, the matching script downloads the pinned release from
Tencent Cloud COS and verifies its SHA-256 digest. Tagged releases publish the
Skill ZIP, `SHA256SUMS`, and exactly two standalone binaries: signed and
notarized macOS ARM64 plus Windows x64. Linux and macOS Intel are not release
platforms.

The macOS signing job reads its certificate and notarization credentials from
the `macos-release` GitHub environment. The publish job uploads both platform
Expand Down
41 changes: 34 additions & 7 deletions skills/lexmount-webfetch/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,25 +5,52 @@ description: Use Lexmount WebFetch for lightweight public-page extraction and re

# Lexmount WebFetch

Resolve `<skill-root>` to the directory containing this loaded `SKILL.md` with
the current Agent's Skill locator:

- Codex: use the absolute `SKILL.md` source path supplied in the Skill metadata.
- Claude Code: use `${CLAUDE_SKILL_DIR}`.
- WorkBuddy/CodeBuddy: use `${CODEBUDDY_SKILL_DIR}`.

Do not infer `<skill-root>` from the working directory.

Select the native Rust binary for the current platform:

- macOS arm64: run `${CODEBUDDY_SKILL_DIR}/scripts/bootstrap.sh` when `${CODEBUDDY_SKILL_DIR}/bin/webfetch-cli` is missing, then use that file.
- Windows x64: run `${CODEBUDDY_SKILL_DIR}/scripts/bootstrap.ps1` when `${CODEBUDDY_SKILL_DIR}/bin/webfetch-cli.exe` is missing, then use that file.
- macOS arm64: run `sh "<skill-root>/scripts/bootstrap.sh"` when `<skill-root>/bin/webfetch-cli` is missing, then invoke `"<skill-root>/bin/webfetch-cli"`.
- Windows x64: run `& "<skill-root>\scripts\bootstrap.ps1"` in PowerShell when `<skill-root>\bin\webfetch-cli.exe` is missing, then invoke `& "<skill-root>\bin\webfetch-cli.exe"`.

Both bootstrap scripts download the fixed release version from Tencent Cloud COS
and verify its SHA-256 digest. The Agent-specific locator is needed to form the
initial absolute command. Once started, the bootstrap and doctor scripts locate
the Skill directory from their own file location.

Do not run the binary for the other platform or assume `webfetch-cli` is on `PATH`.

## Setup

Both bootstrap scripts download the fixed release version from Tencent Cloud COS and verify its SHA-256 digest. The examples abbreviate the selected path as `webfetch-cli`.
1. Resolve `<skill-root>` from this `SKILL.md` and select the matching platform paths above.
2. Run the Skill-local bootstrap script if the binary is missing. Then run `sh "<skill-root>/scripts/doctor.sh"` on macOS arm64 or `& "<skill-root>\scripts\doctor.ps1"` in Windows PowerShell.
3. If credentials are missing, run the Skill-local CLI's `auth login --open`.
Pass `--client-name "<agent-name>"` when the current Agent has a user-facing
name; otherwise omit it and the CLI uses `Agent`. Let the user approve in
their browser; never ask them to paste an API key into chat.
4. Run the platform doctor script again after login. Continue only when the
top-level `ok` value is `true` and both the `credentials` and `agent_skill`
checks pass.

## Fast path

Call the target command directly when credentials are already configured:

```bash
webfetch-cli extract --url <url>
webfetch-cli dump-dom --url <url>
"<skill-root>/bin/webfetch-cli" extract --url <url>
"<skill-root>/bin/webfetch-cli" dump-dom --url <url>
```

Do not run setup checks before every extraction. On first use, run the matching bootstrap script if the binary is missing, then run the platform doctor script. Run doctor again after an authentication or API error.
On Windows PowerShell, invoke `& "<skill-root>\bin\webfetch-cli.exe"` with the
same arguments.

If credentials are missing, run `webfetch-cli auth login --open --client-name WorkBuddy`. Let the user approve in their browser; never ask them to paste an API key into chat.
Do not run setup checks before every extraction. Run doctor again after an authentication or API error.

## Output selection

Expand Down
19 changes: 14 additions & 5 deletions skills/lexmount-webfetch/references/authentication.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,19 @@
# Authentication

Run:
Resolve `<skill-root>` from the loaded `SKILL.md`, then run the Skill-local CLI:

```bash
webfetch-cli auth login --open --client-name WorkBuddy
```text
"<skill-root>/bin/webfetch-cli" auth login --open [--client-name "NAME"]
```

The CLI opens a PKCE approval flow, listens on a random loopback port, exchanges the returned code, and stores the credential locally. On Unix the file mode is `0600`.
On Windows PowerShell, invoke `& "<skill-root>\bin\webfetch-cli.exe"` with the
same arguments. Quote and pass the current Agent's user-facing name when
available. Omit `--client-name` when no name is available; the CLI then uses
`Agent`.

The CLI opens a PKCE approval flow, listens on a random loopback port, exchanges
the returned code, and stores the credential locally. On Unix the file mode is
`0600`.

Environment variables take precedence:

Expand All @@ -16,4 +23,6 @@ Environment variables take precedence:
- `LEXMOUNT_WEBFETCH_CONNECT_BASE_URL`
- `LEXMOUNT_WEBFETCH_CREDENTIALS_FILE`

Use `webfetch-cli auth status` to inspect non-secret state. Use `auth clear-credentials` only when the user asks to disconnect or when a stored credential must be replaced.
Use the same Skill-local CLI with `auth status` to inspect non-secret state. Use
`auth clear-credentials` only when the user asks to disconnect or when a stored
credential must be replaced.
28 changes: 19 additions & 9 deletions skills/lexmount-webfetch/references/commands.md
Original file line number Diff line number Diff line change
@@ -1,23 +1,33 @@
# Command reference

Resolve `<skill-root>` from the loaded `SKILL.md`. On macOS, `<webfetch-cli>`
below means `"<skill-root>/bin/webfetch-cli"`; on Windows PowerShell it means
`& "<skill-root>\bin\webfetch-cli.exe"`. These are Skill-local absolute
invocations, not `PATH` lookups.

```text
webfetch-cli version
webfetch-cli doctor --json
webfetch-cli capabilities --json
<webfetch-cli> version
<webfetch-cli> doctor --json
<webfetch-cli> capabilities --json

webfetch-cli auth status
webfetch-cli auth login --open [--client-name WorkBuddy]
<webfetch-cli> auth status
<webfetch-cli> auth login --open [--client-name "NAME"]
[--connect-base-url https://browser.lexmount.cn] [--timeout-seconds 300]
webfetch-cli auth clear-credentials
<webfetch-cli> auth clear-credentials

webfetch-cli extract (--url URL | --dom-id ID) [--timeout-ms MS]
<webfetch-cli> extract (--url URL | --dom-id ID) [--timeout-ms MS]
[--format md|text|json|json-full]
[--include-trace] [--include-raw-dom]

webfetch-cli dump-dom --url URL [--timeout-ms MS]
<webfetch-cli> dump-dom --url URL [--timeout-ms MS]
[--format md|text|json|json-full]
[--engine auto|http|chrome|chrome_cdp|lightmount_lite|lightmount_dcl|lightmount_domstable]
[--filter-scripts-styles]
```

`extract --dom-id` reuses a prior DOM dump when the API returned a DOM ID. Default output is Markdown. Debug flags require `--format json-full` so heavy or sensitive diagnostic fields do not appear accidentally.
Quote and pass the current Agent's user-facing name when available. Omit
`--client-name` when no name is available; the CLI defaults to `Agent`.

`extract --dom-id` reuses a prior DOM dump when the API returned a DOM ID.
Default output is Markdown. Debug flags require `--format json-full` so heavy or
sensitive diagnostic fields do not appear accidentally.
22 changes: 16 additions & 6 deletions skills/lexmount-webfetch/references/troubleshooting.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,18 @@
# Troubleshooting

1. Missing command: run the platform bootstrap script, then the doctor script.
2. Missing or expired credentials: run `webfetch-cli auth login --open --client-name WorkBuddy`.
3. Thin content or HTML warning: retry with `dump-dom`, try an explicit engine, or move to the browser Skill when interaction/rendering is required.
4. API timeout: increase `--timeout-ms` once; do not retry indefinitely.
5. Need trace or raw DOM: add `--format json-full` before the debug flag.
6. Unexpected API shape: use `--format json-full` for diagnosis, but redact secrets before sharing output.
1. Skill root unknown: resolve the directory containing the loaded `SKILL.md`
with the current host's locator: Codex supplies its absolute source path in
Skill metadata, Claude Code provides `${CLAUDE_SKILL_DIR}`, and
WorkBuddy/CodeBuddy provides `${CODEBUDDY_SKILL_DIR}`. Do not infer it from
the working directory or search the user's home directory.
2. Missing command: run `sh "<skill-root>/scripts/bootstrap.sh"` on macOS arm64
or `& "<skill-root>\scripts\bootstrap.ps1"` on Windows x64, then run the
matching doctor script. Invoke only the Skill-local binary afterward; do not
rely on `PATH`.
3. Missing or expired credentials: run the Skill-local CLI's
`auth login --open`. Pass `--client-name "<agent-name>"` when the current
Agent has a user-facing name; otherwise omit it to use `Agent`.
4. Thin content or HTML warning: retry with `dump-dom`, try an explicit engine, or move to the browser Skill when interaction/rendering is required.
5. API timeout: increase `--timeout-ms` once; do not retry indefinitely.
6. Need trace or raw DOM: add `--format json-full` before the debug flag.
7. Unexpected API shape: use `--format json-full` for diagnosis, but redact secrets before sharing output.
2 changes: 1 addition & 1 deletion skills/lexmount-webfetch/scripts/bootstrap.ps1
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ function Invoke-Tls12Download {
}
}

$version = if ($env:LEXMOUNT_WEBFETCH_CLI_VERSION) { $env:LEXMOUNT_WEBFETCH_CLI_VERSION } else { "0.1.2" }
$version = if ($env:LEXMOUNT_WEBFETCH_CLI_VERSION) { $env:LEXMOUNT_WEBFETCH_CLI_VERSION } else { "0.1.3" }
$downloadBaseUrl = if ($env:LEXMOUNT_WEBFETCH_CLI_DOWNLOAD_BASE_URL) { $env:LEXMOUNT_WEBFETCH_CLI_DOWNLOAD_BASE_URL.TrimEnd('/') } else { "https://cli-bin-1377899528.cos.ap-nanjing.myqcloud.com/releases/webfetch-cli" }
$architecture = if ($env:PROCESSOR_ARCHITEW6432) { $env:PROCESSOR_ARCHITEW6432 } else { $env:PROCESSOR_ARCHITECTURE }
if ($architecture -ne "AMD64") { throw "Only Windows x64 is supported" }
Expand Down
2 changes: 1 addition & 1 deletion skills/lexmount-webfetch/scripts/bootstrap.sh
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
#!/bin/sh
set -eu

version="${LEXMOUNT_WEBFETCH_CLI_VERSION:-0.1.2}"
version="${LEXMOUNT_WEBFETCH_CLI_VERSION:-0.1.3}"
download_base_url="${LEXMOUNT_WEBFETCH_CLI_DOWNLOAD_BASE_URL:-https://cli-bin-1377899528.cos.ap-nanjing.myqcloud.com/releases/webfetch-cli}"
repo="${download_base_url%/}/v${version}"
case "$(uname -s)-$(uname -m)" in
Expand Down
Loading