Skip to content

feat(evals): run the MCP host evals through OpenRouter and in CI - #130

Merged
joe4dev merged 13 commits into
mainfrom
devx-1320-run-the-mcp-servers-llm-evals-in-ci-through
Nov 19, 2026
Merged

joe4dev merged 13 commits into
mainfrom
devx-1320-run-the-mcp-servers-llm-evals-in-ci-through

Conversation

@joe4dev

@joe4dev joe4dev commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

Summary

Supersedes #96. Its commit stays as-is, by @bartsdev; the following commits fix the issues the review found and add deterministic tool tests. Part of DEVX-1320, which blocks DEVX-1089: the App Inspector IAM data there must be verified against a live agent workflow.

Two layers of MCP tests, each checking what the other can't:

  • Tool tests (yarn test:mcp:tools, new CI job mcp-tool-tests on every PR): each tool against a real LocalStack, no model. 13 tests call 12 tools in a fixed order (start, App Inspector, deploy, AWS client, state, traces, logs, IAM, chaos latency, a Cloud Pods round trip, replicator, extensions, ephemeral instances). Each test reads back the state a tool leaves behind through another call, not only the tool's own success message, so a tool that skips its API call but still reports success fails. Examples: the Cloud Pods test removes the bucket after the save and expects the load to bring it back; IAM switches to ENFORCED, then SOFT_MODE, and reads each back; a second Cloud Pods delete must fail.
  • Evals (yarn test:mcp:evals, through OpenRouter after every merge into main that changes the server, its tools or the evals, weekly and on demand; default google/gemini-2.5-flash, EVAL_MODEL overrides it): whether a model picks and combines the right tools. Prompts describe a goal, not the tool ("My Lambda keeps failing with AccessDenied… let the calls through and only log missing permissions" → IAM soft mode). 9 cases, one run each: 9 model runs instead of 68.

What changed on top of #96:

  • A case fails when its tool fails. Before, a case passed when the model called the right tool, even if the tool returned ❌ .... Each case now also matches text only a successful tool output contains. The tester matches against the whole simulation result as JSON, so each pattern starts with a prefix that keeps it inside one tool output; otherwise the scenario or the model's answer could satisfy it. On top of that, every required call needs at least one matching call that didn't return an error: the tester drops isError, and a model can still read a usable answer out of an error's text.
  • Every case must pass. feat(evals): run the MCP host evals through OpenRouter and in CI #96 passed the job at 75% of cases, so 3 broken tools could stay green. Iterations lost to infrastructure errors now count as failures.
  • Evals report tokens and cost. mcp-server-tester 1.1.1 reads the old AI SDK usage fields and hardcodes the cost to 0. A patch-package patch (patches/, applied by the prepare script, which doesn't run for npm consumers) reads totalUsage and sums OpenRouter's per-step cost. Dependabot ignores the tester, since a bump would no longer match the patch.
  • Tool versions come from one place. A host-tools stage in the Dockerfile copies the image's own Terraform and uv binaries; the composite action .github/actions/host-tools exports them, installs lstk and optionally the Snowflake CLI from the image's lockfiles (docker/tools/), and puts them on the PATH. Dependabot bumps the FROM lines, and CI gets exactly those binaries; nothing parses the Dockerfile. A pin test fails when a workflow installs Terraform or uv any other way, or when a job that runs the tool tests or evals skips the action.
  • Fork PRs fail loud instead of skipping. GitHub gives them no secrets, so mcp-direct-tests, mcp-tool-tests and the Docker integration test in smoke (amd64) now fail with an error that says what to do: a maintainer pushes the branch to this repo and opens a PR from it. Before, docker.yml skipped its integration test on fork PRs and still showed green; its check now sits right before that test, so forks still get the image build and startup smoke test. Dependabot PRs have the Dependabot secret and run normally.
  • The deployer runs lstk terraform (since feat(deployer): run Terraform, CDK, and SAM through lstk #97), so CI installs lstk instead of tflocal.
  • Cloud Pods runs second in the evals, right after LocalStack starts, while a save is still well under the eval host's 60s tool-call cap. The tool tests call the SDK client directly with a 5-minute timeout instead.
  • Runs don't overlap (the evals share one Cloud Pod name; the tool tests use a pod name per run and attempt), and the results upload includes the .mcp-test-results dot directory.
  • Dropped the unused ai devDependency: the tester brings its own copy. yarn.lock adds the OpenRouter provider and patch-package.

Evidence

  • Full local run (yarn test:mcp, against a separate LocalStack on port 4599 via LOCALSTACK_PORT and MAIN_CONTAINER_NAME): 34/34 in 5m 24s. Direct 12/12, tool tests 13/13 in 1.6 min (including the start), evals 9/9 including Snowflake in 3.4 min: 159,786 input and 8,243 output tokens, $0.045. The traces answer made up half of that: the tool tests had left a long trace table behind, which the model copied.

  • Evals workflow in CI (run 44543116189): 9/9 in 2.7 min, about 4 min for the whole job. Every step passed: the host-tools action including the Snowflake CLI, terraform init, the image pulls, the evals and the results upload (205 KB artifact). 160,803 input and 1,235 output tokens, $0.020. workflow_dispatch only works once the workflow is on main, so a temporary commit added a push trigger for this branch and a second commit reverted it; squashing on merge drops both.

    Case Tool calls Input tokens Output tokens Cost
    management-status 2 12,942 67 $0.0020
    cloud-pods-roundtrip 4 22,248 230 $0.0022
    deploy-terraform-project 1 9,780 71 $0.0021
    aws-client-list-buckets 1 8,463 40 $0.0016
    iam-soft-mode 2 13,051 117 $0.0012
    chaos-latency-cycle 4 21,597 130 $0.0028
    app-inspector-traces 3 17,295 104 $0.0014
    docs-search 1 17,205 243 $0.0037
    snowflake-full-flow 7 38,222 233 $0.0032
    Total 25 160,803 1,235 $0.020
  • PR checks on d79163d: every job passed. mcp-direct-tests 1m 19s, mcp-tool-tests 3m 4s including the start through the management tool, smoke (amd64) 5m 50s with the Docker integration test, smoke (arm64) 2m 5s. The new token checks pass on branches in this repo; their failure path, a fork PR, can't run from inside the repo.

  • Mutation testing: a review agent planted bugs in the tools and ran both suites against them.

    • Tool tests catch 30/30, up from 19/27 before the read-backs.
    • Evals catch 8 of the 10 bugs that change tool behaviour (up from 5). The two they miss, dropped trace rows and a Cloud Pods load that does nothing, are caught by the tool tests.
    • The evals don't catch the 5 description-only mutants: Gemini picks tools by name.
    • No false failures in 3 full runs and about 30 targeted runs.
  • The output prefix caught a false pass: in an earlier run, docs returned ❌ Auth Token Required and the old pattern still passed it.

  • host-tools stage: the exported linux/amd64 binaries report Terraform 1.17.2 and uv 0.13.9.

  • Pin test: fails for setup-uv, pipx install uv, a job without the action, and a Dockerfile without the stage.

  • Unit tests: 425 passed.

Merge Danger

Door: two-way

Only test code, test data, CI config, dev dependencies, a patch to a dev dependency and a Dockerfile stage the image build skips. Reverting the PR restores the Gemini setup.

Blast Radius: CI-only

  • The OPENROUTER_API_KEY and LOCALSTACK_AUTH_TOKEN secrets are set.
  • Every PR now starts LocalStack in mcp-tool-tests and pulls localstack-pro plus the Terraform and uv images.
  • The tool tests save and delete a Cloud Pod in the CI token's account on every run.
  • One run per eval case: a model fluke or a transient docs timeout fails the eval job after a merge or in the weekly run; rerun it. The evals run after the merge, so a model-visible regression turns main red rather than blocking the PR; the per-PR mcp-tool-tests catch broken tools before the merge.
  • Each such merge into main costs about $0.02 (a path filter skips README edits, Dependabot action bumps and the like) in OpenRouter usage. Runs never overlap (they share a Cloud Pod name); merges that land during a run wait, and only the newest waiting run is kept.
  • A tester upgrade needs the patch redone or dropped; yarn install fails when the patch no longer applies.
  • If the IAM or chaos tool itself breaks, its cleanup (through the same tool) may leave IAM enforced or latency set for the rest of the tool-test run. That only affects the run itself.
  • On failure, the eval log prints the full case result, including tool outputs. DEVX-1320 lists a follow-up to print only the case IDs and failed expectations.

Review

Start with tests/mcp/tools.spec.mjs and data/evals/comprehensive.json, then tests/mcp/evals.spec.mjs, .github/actions/host-tools/action.yml and the host-tools stage in the Dockerfile.

🤖 Generated with Claude Code

Bart and others added 2 commits November 18, 2026 17:24
Route the eval dataset through OpenRouter (same google/gemini-2.5-flash
model) instead of the Google provider, add EVAL_MODEL to swap the model,
and declare ai and @openrouter/ai-sdk-provider as dev dependencies.

Add a weekly and on-demand MCP evals workflow modeled on Azure smoke.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… lstk

Every case only checked that the model called the right tool, so a case passed
even when the tool returned an error. Each case now also matches a success
marker inside a tool call's output. The tester matches against the whole
simulation result as JSON, so each pattern starts with a prefix that keeps it
inside one output string; otherwise the model's answer or the scenario could
satisfy it. The spec requires every case to pass, and counts iterations lost to
infrastructure errors (rate limits, 5xx) as failures instead of dropping them.
The Cloud Pods case runs right after LocalStack starts, while a save is still
quick enough for the tester's 60s call limit.

Since #97 the deployer runs `lstk terraform`, so the workflow installs lstk
instead of tflocal and downloads the sample project's providers up front.
lstk and the Snowflake CLI come from the image's lockfiles; Terraform and uv
from the Dockerfile's FROM lines, which a pin test now enforces for every
workflow. Runs no longer overlap, because they share one Cloud Pod name, and
the job timeout leaves room for setup and the upload, which now includes the
results directory (upload-artifact skips dot directories by default).

Drop the unused `ai` devDependency: the tester loads its own copy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@joe4dev
joe4dev marked this pull request as draft November 19, 2026 09:10
joe4dev and others added 3 commits November 19, 2026 10:32
The evals workflow read the Terraform and uv versions out of the Dockerfile's
FROM lines with sed and passed them to setup actions. A new `host-tools` stage
copies the binaries from the same stages the image uses, and a composite action
(.github/actions/host-tools) builds it, installs lstk and optionally the
Snowflake CLI from the image's lockfiles, and puts them on the PATH. The
Dockerfile stays the one place Dependabot bumps these versions, and every job
that needs the tools uses the action. The pin test now fails when a workflow
installs Terraform or uv any other way.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tests/mcp/tools.spec.mjs calls each tool against a running LocalStack in a
fixed order (start, deploy, list, inspect, IAM, chaos, Cloud Pods, replicator,
extensions, ephemeral instances) and checks text only a successful result
contains. It needs no model, so a new CI job runs it on every pull request.
It calls the SDK client directly with a 5-minute timeout: the tester's
callTool applies the SDK's 60s default, which a start or deploy can exceed.

With the tools covered there, the evals only check whether a model picks and
combines the right tools. Their prompts now describe a goal instead of naming
the tool, the cases that only repeated a single tool call moved to the tool
tests, and each case runs once: 9 model runs instead of 68.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…r evals

The pin test only banned the setup actions and download URLs, so a job could
install Terraform or uv with pip, pipx, brew, apt, mise or asdf, or skip the
action and use whatever the runner image ships. It now bans those installs
too, and fails when a job that runs `test:mcp`, `test:mcp:tools` or
`test:mcp:evals` doesn't use .github/actions/host-tools.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@joe4dev

joe4dev commented Nov 19, 2026

Copy link
Copy Markdown
Member Author

FYI: An alternative to using the .github/actions/host-tools/action.yml approach copying binaries from the Docker build would be using .tool-versions but that's not supported by Dependabot and would require migrating to Renovate

joe4dev and others added 7 commits November 19, 2026 11:36
The tester's AI SDK host read `usage.promptTokens` and `completionTokens`,
names the AI SDK dropped in v5, and hardcoded `totalCostUsd: 0`, so every
report showed 0 tokens and $0. A patch-package patch reads `totalUsage`, which
covers every step of a multi-step case, and sums the cost OpenRouter returns
for each step. `prepare` applies it on install; unlike `postinstall` it doesn't
run when users install the published package. When Dependabot bumps the
tester, the patch must be redone or dropped. 2.0 counts tokens but not cost.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Mutation testing showed the tool tests passed when a tool skipped its API call
but still reported success, because they checked the tool's own message. Each
test now reads the state back: IAM and App Inspector switch between two
states and read each back, chaos reads the latency after clearing it, Cloud
Pods removes the bucket after the save and checks the load brings it back, a
second delete must fail, traces must list a row, the logs analysis must report
a request made to fail, the deploy passes name_prefix and checks the outputs
table, and the region-bound SQS queue must be listed.

The evals fail a case when every call of a required tool returned an error
(the tester drops isError; this server's errors start with ❌), and the chaos
and IAM prompts ask for a read-back that the patterns check.

Review fixes: fork PRs skip the jobs that need secrets; the Cloud Pods test
cleans up without masking the first error and names the pod per attempt;
chaos and IAM restore state on failure; Dependabot no longer bumps the
patched tester; the pin test also catches `yarn run`/`npm run` jobs and curl
installers, and no longer flags links to the uv docs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…the jobs

A skipped job looks green, so a fork PR could merge without anyone noticing it was never
tested against LocalStack. Both jobs now fail in their first step with an error that says
how a maintainer can run them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s missing

Same as the CI jobs: on a fork PR, smoke (amd64) skipped the integration test and still
showed green.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The weekly run stays, to catch changes outside this repo such as a new LocalStack image or
model version.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@joe4dev
joe4dev marked this pull request as ready for review November 19, 2026 14:48
@joe4dev
joe4dev marked this pull request as draft November 19, 2026 14:49
… build the image

The eval trigger on main gets a path filter, so README edits or Dependabot action bumps don't
start a run. The Docker token check moves right before the integration test, so a fork PR
still builds the image and runs the startup smoke test before failing loud.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@joe4dev
joe4dev marked this pull request as ready for review November 19, 2026 14:54
@joe4dev
joe4dev merged commit 926d78f into main Nov 19, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant