Repository navigation
feat(evals): run the MCP host evals through OpenRouter and in CI - #130
Merged
joe4dev merged 13 commits intoNov 19, 2026
Merged
Conversation
Route the eval dataset through OpenRouter (same google/gemini-2.5-flash model) instead of the Google provider, add EVAL_MODEL to swap the model, and declare ai and @openrouter/ai-sdk-provider as dev dependencies. Add a weekly and on-demand MCP evals workflow modeled on Azure smoke. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… lstk Every case only checked that the model called the right tool, so a case passed even when the tool returned an error. Each case now also matches a success marker inside a tool call's output. The tester matches against the whole simulation result as JSON, so each pattern starts with a prefix that keeps it inside one output string; otherwise the model's answer or the scenario could satisfy it. The spec requires every case to pass, and counts iterations lost to infrastructure errors (rate limits, 5xx) as failures instead of dropping them. The Cloud Pods case runs right after LocalStack starts, while a save is still quick enough for the tester's 60s call limit. Since #97 the deployer runs `lstk terraform`, so the workflow installs lstk instead of tflocal and downloads the sample project's providers up front. lstk and the Snowflake CLI come from the image's lockfiles; Terraform and uv from the Dockerfile's FROM lines, which a pin test now enforces for every workflow. Runs no longer overlap, because they share one Cloud Pod name, and the job timeout leaves room for setup and the upload, which now includes the results directory (upload-artifact skips dot directories by default). Drop the unused `ai` devDependency: the tester loads its own copy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
joe4dev
marked this pull request as draft
November 19, 2026 09:10
The evals workflow read the Terraform and uv versions out of the Dockerfile's FROM lines with sed and passed them to setup actions. A new `host-tools` stage copies the binaries from the same stages the image uses, and a composite action (.github/actions/host-tools) builds it, installs lstk and optionally the Snowflake CLI from the image's lockfiles, and puts them on the PATH. The Dockerfile stays the one place Dependabot bumps these versions, and every job that needs the tools uses the action. The pin test now fails when a workflow installs Terraform or uv any other way. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tests/mcp/tools.spec.mjs calls each tool against a running LocalStack in a fixed order (start, deploy, list, inspect, IAM, chaos, Cloud Pods, replicator, extensions, ephemeral instances) and checks text only a successful result contains. It needs no model, so a new CI job runs it on every pull request. It calls the SDK client directly with a 5-minute timeout: the tester's callTool applies the SDK's 60s default, which a start or deploy can exceed. With the tools covered there, the evals only check whether a model picks and combines the right tools. Their prompts now describe a goal instead of naming the tool, the cases that only repeated a single tool call moved to the tool tests, and each case runs once: 9 model runs instead of 68. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…r evals The pin test only banned the setup actions and download URLs, so a job could install Terraform or uv with pip, pipx, brew, apt, mise or asdf, or skip the action and use whatever the runner image ships. It now bans those installs too, and fails when a job that runs `test:mcp`, `test:mcp:tools` or `test:mcp:evals` doesn't use .github/actions/host-tools. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Member
Author
|
FYI: An alternative to using the |
The tester's AI SDK host read `usage.promptTokens` and `completionTokens`, names the AI SDK dropped in v5, and hardcoded `totalCostUsd: 0`, so every report showed 0 tokens and $0. A patch-package patch reads `totalUsage`, which covers every step of a multi-step case, and sums the cost OpenRouter returns for each step. `prepare` applies it on install; unlike `postinstall` it doesn't run when users install the published package. When Dependabot bumps the tester, the patch must be redone or dropped. 2.0 counts tokens but not cost. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Mutation testing showed the tool tests passed when a tool skipped its API call but still reported success, because they checked the tool's own message. Each test now reads the state back: IAM and App Inspector switch between two states and read each back, chaos reads the latency after clearing it, Cloud Pods removes the bucket after the save and checks the load brings it back, a second delete must fail, traces must list a row, the logs analysis must report a request made to fail, the deploy passes name_prefix and checks the outputs table, and the region-bound SQS queue must be listed. The evals fail a case when every call of a required tool returned an error (the tester drops isError; this server's errors start with ❌), and the chaos and IAM prompts ask for a read-back that the patterns check. Review fixes: fork PRs skip the jobs that need secrets; the Cloud Pods test cleans up without masking the first error and names the pod per attempt; chaos and IAM restore state on failure; Dependabot no longer bumps the patched tester; the pin test also catches `yarn run`/`npm run` jobs and curl installers, and no longer flags links to the uv docs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…the jobs A skipped job looks green, so a fork PR could merge without anyone noticing it was never tested against LocalStack. Both jobs now fail in their first step with an error that says how a maintainer can run them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s missing Same as the CI jobs: on a fork PR, smoke (amd64) skipped the integration test and still showed green. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The weekly run stays, to catch changes outside this repo such as a new LocalStack image or model version. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
joe4dev
marked this pull request as ready for review
November 19, 2026 14:48
joe4dev
marked this pull request as draft
November 19, 2026 14:49
… build the image The eval trigger on main gets a path filter, so README edits or Dependabot action bumps don't start a run. The Docker token check moves right before the integration test, so a fork PR still builds the image and runs the startup smoke test before failing loud. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
joe4dev
marked this pull request as ready for review
November 19, 2026 14:54
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Supersedes #96. Its commit stays as-is, by @bartsdev; the following commits fix the issues the review found and add deterministic tool tests. Part of DEVX-1320, which blocks DEVX-1089: the App Inspector IAM data there must be verified against a live agent workflow.
Two layers of MCP tests, each checking what the other can't:
yarn test:mcp:tools, new CI jobmcp-tool-testson every PR): each tool against a real LocalStack, no model. 13 tests call 12 tools in a fixed order (start, App Inspector, deploy, AWS client, state, traces, logs, IAM, chaos latency, a Cloud Pods round trip, replicator, extensions, ephemeral instances). Each test reads back the state a tool leaves behind through another call, not only the tool's own success message, so a tool that skips its API call but still reports success fails. Examples: the Cloud Pods test removes the bucket after the save and expects the load to bring it back; IAM switches toENFORCED, thenSOFT_MODE, and reads each back; a second Cloud Pods delete must fail.yarn test:mcp:evals, through OpenRouter after every merge intomainthat changes the server, its tools or the evals, weekly and on demand; defaultgoogle/gemini-2.5-flash,EVAL_MODELoverrides it): whether a model picks and combines the right tools. Prompts describe a goal, not the tool ("My Lambda keeps failing with AccessDenied… let the calls through and only log missing permissions" → IAM soft mode). 9 cases, one run each: 9 model runs instead of 68.What changed on top of #96:
❌ .... Each case now also matches text only a successful tool output contains. The tester matches against the whole simulation result as JSON, so each pattern starts with a prefix that keeps it inside one tool output; otherwise the scenario or the model's answer could satisfy it. On top of that, every required call needs at least one matching call that didn't return an error: the tester dropsisError, and a model can still read a usable answer out of an error's text.patch-packagepatch (patches/, applied by thepreparescript, which doesn't run for npm consumers) readstotalUsageand sums OpenRouter's per-step cost. Dependabot ignores the tester, since a bump would no longer match the patch.host-toolsstage in the Dockerfile copies the image's own Terraform and uv binaries; the composite action.github/actions/host-toolsexports them, installs lstk and optionally the Snowflake CLI from the image's lockfiles (docker/tools/), and puts them on the PATH. Dependabot bumps the FROM lines, and CI gets exactly those binaries; nothing parses the Dockerfile. A pin test fails when a workflow installs Terraform or uv any other way, or when a job that runs the tool tests or evals skips the action.mcp-direct-tests,mcp-tool-testsand the Docker integration test insmoke (amd64)now fail with an error that says what to do: a maintainer pushes the branch to this repo and opens a PR from it. Before,docker.ymlskipped its integration test on fork PRs and still showed green; its check now sits right before that test, so forks still get the image build and startup smoke test. Dependabot PRs have the Dependabot secret and run normally.lstk terraform(since feat(deployer): run Terraform, CDK, and SAM through lstk #97), so CI installs lstk instead of tflocal..mcp-test-resultsdot directory.aidevDependency: the tester brings its own copy.yarn.lockadds the OpenRouter provider and patch-package.Evidence
Full local run (
yarn test:mcp, against a separate LocalStack on port 4599 viaLOCALSTACK_PORTandMAIN_CONTAINER_NAME): 34/34 in 5m 24s. Direct 12/12, tool tests 13/13 in 1.6 min (including the start), evals 9/9 including Snowflake in 3.4 min: 159,786 input and 8,243 output tokens, $0.045. The traces answer made up half of that: the tool tests had left a long trace table behind, which the model copied.Evals workflow in CI (run 44543116189): 9/9 in 2.7 min, about 4 min for the whole job. Every step passed: the host-tools action including the Snowflake CLI,
terraform init, the image pulls, the evals and the results upload (205 KB artifact). 160,803 input and 1,235 output tokens, $0.020.workflow_dispatchonly works once the workflow is onmain, so a temporary commit added apushtrigger for this branch and a second commit reverted it; squashing on merge drops both.PR checks on
d79163d: every job passed.mcp-direct-tests1m 19s,mcp-tool-tests3m 4s including the start through the management tool,smoke (amd64)5m 50s with the Docker integration test,smoke (arm64)2m 5s. The new token checks pass on branches in this repo; their failure path, a fork PR, can't run from inside the repo.Mutation testing: a review agent planted bugs in the tools and ran both suites against them.
The output prefix caught a false pass: in an earlier run, docs returned
❌ Auth Token Requiredand the old pattern still passed it.host-tools stage: the exported linux/amd64 binaries report Terraform 1.17.2 and uv 0.13.9.
Pin test: fails for
setup-uv,pipx install uv, a job without the action, and a Dockerfile without the stage.Unit tests: 425 passed.
Merge Danger
Door: two-way
Only test code, test data, CI config, dev dependencies, a patch to a dev dependency and a Dockerfile stage the image build skips. Reverting the PR restores the Gemini setup.
Blast Radius: CI-only
OPENROUTER_API_KEYandLOCALSTACK_AUTH_TOKENsecrets are set.mcp-tool-testsand pullslocalstack-proplus the Terraform and uv images.mainred rather than blocking the PR; the per-PRmcp-tool-testscatch broken tools before the merge.maincosts about $0.02 (a path filter skips README edits, Dependabot action bumps and the like) in OpenRouter usage. Runs never overlap (they share a Cloud Pod name); merges that land during a run wait, and only the newest waiting run is kept.yarn installfails when the patch no longer applies.Review
Start with
tests/mcp/tools.spec.mjsanddata/evals/comprehensive.json, thentests/mcp/evals.spec.mjs,.github/actions/host-tools/action.ymland thehost-toolsstage in theDockerfile.🤖 Generated with Claude Code