Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/ISSUE_TEMPLATE/config.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
blank_issues_enabled: true
23 changes: 23 additions & 0 deletions .github/ISSUE_TEMPLATE/false-positive.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
name: Wrong finding (false positive or false negative)
description: toolgauge reported something that is not true, or missed something that is.
labels: ["bug", "area: precision"]
body:
- type: input
id: rule
attributes:
label: Rule id
placeholder: MCP/2026-07-28/MUST/... or TOOLGAUGE/STYLE/...
validations:
required: true
- type: textarea
id: observed
attributes:
label: What toolgauge said, and what is actually true
description: Paste the finding. For a spec rule, quote the specification sentence that shows it is wrong.
validations:
required: true
- type: textarea
id: reproduce
attributes:
label: How to reproduce
description: The command you ran. Do not paste real credentials.
16 changes: 16 additions & 0 deletions .github/ISSUE_TEMPLATE/new-rule.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
name: New rule
description: Propose a check toolgauge does not do yet.
labels: ["enhancement", "area: rules"]
body:
- type: textarea
id: requirement
attributes:
label: The requirement
description: For a spec rule, quote the sentence and link the page and heading. Otherwise, explain why it matters.
validations:
required: true
- type: textarea
id: check
attributes:
label: How it can be checked from outside the server
description: What request to send, what answer is a pass, a fail, and "can't tell".
2 changes: 1 addition & 1 deletion .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
#
# Verify a release:
# cosign verify ghcr.io/patsypppe/toolgauge:<version> \
# --certificate-identity-regexp 'https://github.com/patsypppe/sentinel/.github/workflows/release.yml@.*' \
# --certificate-identity-regexp 'https://github.com/patsypppe/toolgauge/.github/workflows/release.yml@.*' \
# --certificate-oidc-issuer https://token.actions.githubusercontent.com
#
# PyPI: Uses PyPI trusted publishing, so no token is stored:
Expand Down
2 changes: 1 addition & 1 deletion .golangci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -41,4 +41,4 @@ formatters:
settings:
goimports:
local-prefixes:
- github.com/patsypppe/sentinel
- github.com/patsypppe/toolgauge
2 changes: 1 addition & 1 deletion .pre-commit-hooks.yaml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Hooks for teams that scan a locally-running MCP server before they push.
#
# repos:
# - repo: https://github.com/patsypppe/sentinel
# - repo: https://github.com/patsypppe/toolgauge
# rev: v0.4.0
# hooks:
# - id: toolgauge-scan
Expand Down
103 changes: 103 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
# Contributing to toolgauge

Thanks for helping. toolgauge checks MCP servers against the specification, so most changes are
one of four kinds: a new or corrected rule, a new server in a benchmark, a fix to how a finding is
reported, or documentation. Each is described below with the exact files to touch.

## Pick something to work on

The [issue tracker](https://github.com/patsypppe/toolgauge/issues) holds scoped tasks, each with what to
change and how to know it is done.

- `good first issue`: small and self-contained, a few hours at most. A good place to start.
- `help wanted`: larger, or needing judgement about the specification.
- Area labels (`area: rules`, `area: benchmarks`, `area: precision`, `area: docs`, `area: transport`)
say which part of the code is involved.

Comment on an issue before starting so two people do not do the same work. If an issue is unclear,
ask in the issue; that is a useful contribution too.

## Set up

You need Python 3.12 or newer and [uv](https://docs.astral.sh/uv/). Go 1.23 or newer is only needed
for the Go parts (`broker/`, `mcp/`, `fleet/`); most harness work never touches them.

```bash
git clone https://github.com/patsypppe/toolgauge
cd toolgauge
uv sync
uv run toolgauge demo # should print one FAIL verdict and one PASS verdict
```

## Test

Python only (what most changes need):

```bash
uv run ruff format . # format
uv run ruff check . # lint
uv run mypy # type-check (strict)
uv run pytest tests -m unit -q -n auto
```

Everything, including Go (what CI runs):

```bash
make check
```

All of it must pass before a pull request is merged. CI runs the same commands on every push.

## Where things live

| Path | What it is |
|---|---|
| `harness/src/toolgauge/catalog/must/`, `should/` | Rules that restate the MCP specification (`MCP/<revision>/MUST/...`, `SHOULD/...`) |
| `harness/src/toolgauge/catalog/beyond/` | Rules that go beyond the spec (`TOOLGAUGE/STYLE`, `SECURITY`, `OPS`) |
| `harness/src/toolgauge/probe/` | The HTTP and stdio transports. Requests are built by hand on purpose; no MCP SDK |
| `harness/src/toolgauge/report/` | Text, JSON, SARIF, HTML, JUnit and Markdown output |
| `fixtures/server/` | Practice servers: `nonconformant` (seeded violations), `conformant`, `partial` |
| `tests/harness/` | Tests. Most run the practice servers in-process |
| `benchmarks/` | Measurements against servers this project did not write |

## Adding or changing a rule

1. **Find the sentence first.** A spec rule restates a sentence in the specification. Quote it in the
pull request, with the page and heading. If you cannot find one, the rule belongs in `beyond/` with a
`rationale`, not in the spec namespace.
2. **Match the severity to the text.** MUST in the spec is a `Severity.MUST` rule; SHOULD is SHOULD.
3. **Never fail what you cannot see.** If the wire cannot tell a violation from an allowed behaviour,
return `RuleResult.indeterminate(...)` and say why. A false FAIL costs more than a missed one.
4. **Rule ids are permanent.** Never rename or delete one. To change a rule's meaning or severity, set
`deprecated_in` and `superseded_by` on the old rule and add a new one.
5. **Test both directions.** The `nonconformant` fixture should fail it (tag the violation with a
`# VIOLATES <rule id>` comment and add the id to `SEEDED_VIOLATIONS`), and the `conformant` fixture
should pass it. `tests/harness/test_fixture_oracle.py` checks both.

## Adding a server to a benchmark

`benchmarks/popular_servers.py` has a `SERVERS` list. Add an entry with the start command from the
server's README (`npx -y ...` or `uvx ...`) and a dummy value for any credential it needs just to
start. Run `uv run python benchmarks/popular_servers.py`, then `uv run python benchmarks/chart.py`, and
commit the regenerated `POPULAR_SERVERS.md`, `.json` and `.svg`. Never add a server's real credentials.

## Checking precision

A finding is only useful if it is right. `benchmarks/PRECISION.md` records hand-checked samples. To add
one: draw findings at random with a fixed seed, read each against the specification text or the tool's
actual description, and record each verdict (correct, low value or wrong) with a one-line reason. Wrong
verdicts are the most valuable result: open an issue for each.

## Sending a change

- One change per pull request, on a branch.
- Commit messages: `<type>: <description>`, where type is one of `feat`, `fix`, `refactor`, `docs`,
`test`, `chore`, `perf`, `ci`.
- In the description, say what changed, why, and how you tested it. For a rule, quote the spec sentence.
- Do not scan servers you do not own or have permission to test, and never publish another project's
findings before its maintainers have seen them.

## Code of conduct

Be kind and assume good faith. Disagreements about what the specification means are normal; settle
them by quoting the text.
2 changes: 1 addition & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ RUN pip install --no-cache-dir build \
&& python -m build --wheel --outdir /dist /src/harness

FROM python:3.12-slim
LABEL org.opencontainers.image.source="https://github.com/patsypppe/sentinel" \
LABEL org.opencontainers.image.source="https://github.com/patsypppe/toolgauge" \
org.opencontainers.image.description="Conformance harness for MCP 2026-07-28" \
org.opencontainers.image.licenses="MIT"
RUN useradd --uid 10001 --no-create-home --shell /usr/sbin/nologin toolgauge
Expand Down
28 changes: 18 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,15 +11,20 @@ You give it a server's URL. It sends the server test requests, checks each answe
rulebook, and prints a pass/fail list. Every failure says what it saw, what to change, and which
page of the spec the rule comes from.

## Try it in 30 seconds

No clone, no setup, nothing to configure. This starts two practice servers on your machine (one
with deliberate mistakes, one correct), scans both, and shows you the difference:
## Install

```bash
uvx --from "git+https://github.com/patsypppe/sentinel@main#subdirectory=harness" toolgauge demo
pip install toolgauge # or: pipx install toolgauge
toolgauge demo
```

No install at all: `uvx toolgauge demo`. Needs Python 3.12 or newer.

## Try it in 30 seconds

`toolgauge demo` starts two practice servers on your machine (one with deliberate mistakes, one
correct), scans both, and shows you the difference:

```
=== Server A: has deliberate mistakes ===
required rules (MUST): 4 pass, 33 fail, 7 could not be checked from outside
Expand All @@ -33,7 +38,7 @@ uvx --from "git+https://github.com/patsypppe/sentinel@main#subdirectory=harness"
verdict: PASS (exit code 0)
```

Prefer a clone? `git clone https://github.com/patsypppe/sentinel && cd sentinel && uv run toolgauge demo`.
From source: `git clone https://github.com/patsypppe/toolgauge && cd toolgauge && uv run toolgauge demo`.

## Lint any server, including stdio ones

Expand All @@ -49,11 +54,8 @@ rule fails, so it drops straight into CI.

## Scan your own server

No clone needed:

```bash
uvx --from "git+https://github.com/patsypppe/sentinel@main#subdirectory=harness" \
toolgauge scan --endpoint https://your-server.example/mcp --gate must
toolgauge scan --endpoint https://your-server.example/mcp --gate must
```

If the server needs a login token, add `--token "$TOKEN"`. Only scan servers you own or have
Expand Down Expand Up @@ -192,6 +194,12 @@ recommendations, so they are reported but never fail the gate. The rule names ca

toolgauge never imports the server code. It only talks HTTP, so it can be pointed at anything.

## Contributing

There is a list of concrete, scoped tasks on the
[issue tracker](https://github.com/patsypppe/toolgauge/issues), many labelled `good first issue`.
[`CONTRIBUTING.md`](CONTRIBUTING.md) explains how to set up, test and send a change.

## Test toolgauge itself

```bash
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ is already tracked. Post it from your own account; it is written in your voice.

Hi, and thanks for maintaining this server.

I'm building [toolgauge](https://github.com/patsypppe/sentinel), a linter for MCP servers, and
I'm building [toolgauge](https://github.com/patsypppe/toolgauge), a linter for MCP servers, and
ran it on `@modelcontextprotocol/server-filesystem` 2.0.0 (also checked against the source at
`51b6382`). Most parameters carry a `.describe()`, but five do not, and a client or model reading the
tool schema gets no explanation for them:
Expand Down Expand Up @@ -38,6 +38,6 @@ glob semantics.) Why it matters: two 2026 studies of MCP tool descriptions
([arXiv 2602.14878](https://arxiv.org/abs/2602.14878), [2602.18914](https://arxiv.org/abs/2602.18914))
found missing parameter semantics are common and change which tool an LLM selects.

Reproduce: `uvx --from "git+https://github.com/patsypppe/sentinel@main#subdirectory=harness" toolgauge lint --command "npx -y @modelcontextprotocol/server-filesystem /tmp"`
Reproduce: `uvx --from "git+https://github.com/patsypppe/toolgauge@main#subdirectory=harness" toolgauge lint --command "npx -y @modelcontextprotocol/server-filesystem /tmp"`

Happy to send a PR with the descriptions if that's welcome.
4 changes: 2 additions & 2 deletions benchmarks/issue-drafts/02-firecrawl-mcp-server.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ limited to what that shows.

**Body:**

Hi, thanks for the MCP server. I ran [toolgauge](https://github.com/patsypppe/sentinel), a linter for MCP
Hi, thanks for the MCP server. I ran [toolgauge](https://github.com/patsypppe/toolgauge), a linter for MCP
servers, on `firecrawl-mcp` 3.34.1 over stdio. Tool descriptions are detailed (they explain when to use
each tool), but in the `inputSchema` of the tools 71 of 194 parameters have no `description`, and they
are not documented in the tool description text either. A model choosing arguments from the schema sees only a name and a type.
Expand All @@ -23,7 +23,7 @@ Two 2026 studies of MCP tool descriptions
([arXiv 2602.14878](https://arxiv.org/abs/2602.14878), [2602.18914](https://arxiv.org/abs/2602.18914))
found missing parameter semantics are common and measurably change which tool and arguments an LLM picks.

Reproduce: `FIRECRAWL_API_KEY=x uvx --from "git+https://github.com/patsypppe/sentinel@main#subdirectory=harness" toolgauge lint --command "npx -y firecrawl-mcp" --env FIRECRAWL_API_KEY=x`
Reproduce: `FIRECRAWL_API_KEY=x uvx --from "git+https://github.com/patsypppe/toolgauge@main#subdirectory=harness" toolgauge lint --command "npx -y firecrawl-mcp" --env FIRECRAWL_API_KEY=x`
(`tools/list` works with a dummy key; no request is made to your API.)

If the schemas are generated from Zod, adding `.describe()` to those fields would cover it. Happy to help.
2 changes: 1 addition & 1 deletion broker/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ COPY broker ./broker
WORKDIR /src/broker
ARG VERSION=0.1.0-dev
RUN CGO_ENABLED=0 go build -trimpath \
-ldflags "-s -w -X github.com/patsypppe/sentinel/broker/internal/version.Version=${VERSION}" \
-ldflags "-s -w -X github.com/patsypppe/toolgauge/broker/internal/version.Version=${VERSION}" \
-o /out/broker ./cmd/broker

FROM alpine:3.20
Expand Down
28 changes: 14 additions & 14 deletions broker/cmd/broker/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -20,20 +20,20 @@ import (
"syscall"
"time"

"github.com/patsypppe/sentinel/broker/internal/config"
"github.com/patsypppe/sentinel/broker/internal/tools/ops"
"github.com/patsypppe/sentinel/broker/internal/tools/warehouse"
"github.com/patsypppe/sentinel/broker/internal/version"
"github.com/patsypppe/sentinel/mcp/audit"
"github.com/patsypppe/sentinel/mcp/authz"
"github.com/patsypppe/sentinel/mcp/envelope"
"github.com/patsypppe/sentinel/mcp/handles"
"github.com/patsypppe/sentinel/mcp/mrtr"
"github.com/patsypppe/sentinel/mcp/otlp"
"github.com/patsypppe/sentinel/mcp/registry"
"github.com/patsypppe/sentinel/mcp/server"
"github.com/patsypppe/sentinel/mcp/store"
"github.com/patsypppe/sentinel/mcp/transport"
"github.com/patsypppe/toolgauge/broker/internal/config"
"github.com/patsypppe/toolgauge/broker/internal/tools/ops"
"github.com/patsypppe/toolgauge/broker/internal/tools/warehouse"
"github.com/patsypppe/toolgauge/broker/internal/version"
"github.com/patsypppe/toolgauge/mcp/audit"
"github.com/patsypppe/toolgauge/mcp/authz"
"github.com/patsypppe/toolgauge/mcp/envelope"
"github.com/patsypppe/toolgauge/mcp/handles"
"github.com/patsypppe/toolgauge/mcp/mrtr"
"github.com/patsypppe/toolgauge/mcp/otlp"
"github.com/patsypppe/toolgauge/mcp/registry"
"github.com/patsypppe/toolgauge/mcp/server"
"github.com/patsypppe/toolgauge/mcp/store"
"github.com/patsypppe/toolgauge/mcp/transport"
)

func main() {
Expand Down
6 changes: 3 additions & 3 deletions broker/go.mod
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
module github.com/patsypppe/sentinel/broker
module github.com/patsypppe/toolgauge/broker

go 1.23.0

require (
github.com/google/uuid v1.6.0
github.com/jackc/pgx/v5 v5.7.4
github.com/patsypppe/sentinel/mcp v0.0.0
github.com/patsypppe/toolgauge/mcp v0.0.0
)

require (
Expand All @@ -22,4 +22,4 @@ require (
// The kit lives beside the broker in this repository. A replace directive (as well
// as go.work) lets `go build` work from the broker directory alone, as the Docker
// build does.
replace github.com/patsypppe/sentinel/mcp => ../mcp
replace github.com/patsypppe/toolgauge/mcp => ../mcp
6 changes: 3 additions & 3 deletions broker/internal/tools/ops/apply.go
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,9 @@ import (
"github.com/google/uuid"
"github.com/jackc/pgx/v5"

"github.com/patsypppe/sentinel/mcp/envelope"
"github.com/patsypppe/sentinel/mcp/mrtr"
"github.com/patsypppe/sentinel/mcp/registry"
"github.com/patsypppe/toolgauge/mcp/envelope"
"github.com/patsypppe/toolgauge/mcp/mrtr"
"github.com/patsypppe/toolgauge/mcp/registry"
)

// Resolver resolves a principal-bound handle. Declared where it is used.
Expand Down
8 changes: 4 additions & 4 deletions broker/internal/tools/ops/ops_integration_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,10 @@ import (

"github.com/jackc/pgx/v5/pgxpool"

"github.com/patsypppe/sentinel/mcp/handles"
"github.com/patsypppe/sentinel/mcp/mrtr"
"github.com/patsypppe/sentinel/mcp/registry"
"github.com/patsypppe/sentinel/mcp/store"
"github.com/patsypppe/toolgauge/mcp/handles"
"github.com/patsypppe/toolgauge/mcp/mrtr"
"github.com/patsypppe/toolgauge/mcp/registry"
"github.com/patsypppe/toolgauge/mcp/store"
)

// The §10 (WP-6) definition of done, end to end through the real tools:
Expand Down
4 changes: 2 additions & 2 deletions broker/internal/tools/ops/plan.go
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,8 @@ import (
"fmt"
"time"

"github.com/patsypppe/sentinel/mcp/envelope"
"github.com/patsypppe/sentinel/mcp/registry"
"github.com/patsypppe/toolgauge/mcp/envelope"
"github.com/patsypppe/toolgauge/mcp/registry"
)

// Minter mints principal-bound state handles. Declared here, where it is used.
Expand Down
4 changes: 2 additions & 2 deletions broker/internal/tools/warehouse/describe.go
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,8 @@ import (
"github.com/jackc/pgx/v5"
"github.com/jackc/pgx/v5/pgxpool"

"github.com/patsypppe/sentinel/mcp/envelope"
"github.com/patsypppe/sentinel/mcp/registry"
"github.com/patsypppe/toolgauge/mcp/envelope"
"github.com/patsypppe/toolgauge/mcp/registry"
)

// DescribeTool is warehouse.describe: the schema of the tables this principal
Expand Down
2 changes: 1 addition & 1 deletion broker/internal/tools/warehouse/format_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ import (
"encoding/json"
"testing"

"github.com/patsypppe/sentinel/mcp/registry"
"github.com/patsypppe/toolgauge/mcp/registry"
)

func sampleRows(n int) Rows {
Expand Down
Loading
Loading