Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 33 additions & 0 deletions .github/actions/host-tools/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
name: Host tools
description: >-
Puts the Docker image's tools on the PATH for jobs that run the MCP server on the host:
always Terraform and uv (from the Dockerfile's host-tools stage) and lstk (docker/tools/npm), and
optionally the Snowflake CLI from docker/tools/snowflake. Versions change only in those files.
inputs:
snowflake:
description: Also install the Snowflake CLI.
required: false
default: "false"
runs:
using: composite
steps:
- name: Copy Terraform and uv out of the Dockerfile
shell: bash
run: |
docker build --target host-tools --output "type=local,dest=$RUNNER_TEMP/host-tools" .
echo "$RUNNER_TEMP/host-tools" >> "$GITHUB_PATH"

- name: Install lstk
shell: bash
run: |
npm ci --prefix docker/tools/npm --omit=dev --ignore-scripts --no-audit --no-fund
echo "$GITHUB_WORKSPACE/docker/tools/npm/node_modules/.bin" >> "$GITHUB_PATH"

- name: Install the Snowflake CLI
if: inputs.snowflake == 'true'
shell: bash
env:
UV_PROJECT_ENVIRONMENT: ${{ runner.temp }}/snowflake-cli
run: |
"$RUNNER_TEMP/host-tools/uv" sync --project docker/tools/snowflake --locked --no-install-project --no-build
echo "$UV_PROJECT_ENVIRONMENT/bin" >> "$GITHUB_PATH"
3 changes: 3 additions & 0 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,9 @@ updates:
- dependency-name: zod
update-types:
- version-update:semver-minor
# patches/ holds a patch for this exact version, and patch-package fails the install in CI
# when it no longer applies. Bump it by hand and redo or drop the patch.
- dependency-name: "@gleanwork/mcp-server-tester"

# CLIs inside the Docker image (aws-cdk, lstk) and the CDK sample app that CI deploys with them.
- package-ecosystem: npm
Expand Down
47 changes: 45 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,14 @@ jobs:
LOCALSTACK_AUTH_TOKEN: ${{ secrets.LOCALSTACK_AUTH_TOKEN }}

steps:
# GitHub gives pull requests from forks no secrets. Fail loud rather than skip, so a
# fork PR never looks tested.
- name: Require the LocalStack auth token
if: env.LOCALSTACK_AUTH_TOKEN == ''
run: |
echo "::error::LOCALSTACK_AUTH_TOKEN is empty. GitHub gives pull requests from forks no secrets: before merging, a maintainer pushes this branch to the repository and opens a pull request from it."
exit 1

- name: Checkout code
uses: actions/checkout@v7

Expand All @@ -54,5 +62,40 @@ jobs:

- name: Run MCP Server Tester
run: yarn test:mcp:direct
env:
LOCALSTACK_AUTH_TOKEN: ${{ secrets.LOCALSTACK_AUTH_TOKEN }}

# Each tool against a real LocalStack, without a model (tests/mcp/tools.spec.mjs). The first
# test starts LocalStack through the management tool.
mcp-tool-tests:
runs-on: ubuntu-26.04
env:
LOCALSTACK_AUTH_TOKEN: ${{ secrets.LOCALSTACK_AUTH_TOKEN }}

steps:
# GitHub gives pull requests from forks no secrets. Fail loud rather than skip, so a
# fork PR never looks tested.
- name: Require the LocalStack auth token
if: env.LOCALSTACK_AUTH_TOKEN == ''
run: |
echo "::error::LOCALSTACK_AUTH_TOKEN is empty. GitHub gives pull requests from forks no secrets: before merging, a maintainer pushes this branch to the repository and opens a pull request from it."
exit 1

- name: Checkout code
uses: actions/checkout@v7

- name: Use Node.js 22
uses: actions/setup-node@v7
with:
node-version: 22.x

- name: Install dependencies
run: yarn install --frozen-lockfile

- name: Install the image's tools
uses: ./.github/actions/host-tools

# Pulled up front so the start stays within the management tool's 120s.
- name: Pull the LocalStack image
run: docker pull localstack/localstack-pro:latest

- name: Run the tool tests
run: yarn test:mcp:tools
19 changes: 15 additions & 4 deletions .github/workflows/docker.yml
Original file line number Diff line number Diff line change
Expand Up @@ -62,12 +62,23 @@ jobs:
docker run -i --rm \
${{ env.IMAGE_NAME }}:ci

# The integration test needs the token, and GitHub gives pull requests from forks no
# secrets. Fail loud rather than skip, so a fork PR never looks tested. Placed after the
# build and the startup smoke test, which need no secret and still run for forks.
- name: Require the LocalStack auth token
# A step's if can't see its own env, so the shell checks the token.
if: matrix.arch == 'amd64'
env:
LOCALSTACK_AUTH_TOKEN: ${{ secrets.LOCALSTACK_AUTH_TOKEN }}
run: |
if [ -z "$LOCALSTACK_AUTH_TOKEN" ]; then
echo "::error::LOCALSTACK_AUTH_TOKEN is empty. GitHub gives pull requests from forks no secrets: before merging, a maintainer pushes this branch to the repository and opens a pull request from it."
exit 1
fi

- name: Integration test Docker runtime
# amd64 only: the deploys take minutes and do not depend on the image's architecture.
# Fork PRs get no secrets, so they skip it.
if: >-
matrix.arch == 'amd64' &&
(github.event_name == 'push' || github.event.pull_request.head.repo.full_name == github.repository)
if: matrix.arch == 'amd64'
env:
LOCALSTACK_AUTH_TOKEN: ${{ secrets.LOCALSTACK_AUTH_TOKEN }}
HARNESS_TOKEN_REAL: ${{ secrets.LOCALSTACK_AUTH_TOKEN != '' && '1' || '0' }}
Expand Down
92 changes: 92 additions & 0 deletions .github/workflows/mcp-evals.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
name: MCP evals

# The LLM-driven eval dataset through OpenRouter: after every merge into main that can change
# the result, weekly (to catch changes outside this repo, such as a new LocalStack image or model
# version) and on demand.
on:
push:
branches: [main]
# What the MCP server, its tools and the evals are built from; README edits or Dependabot
# action bumps don't run the evals.
paths:
- "src/**"
- "data/**"
- "tests/mcp/**"
- "patches/**"
- "package.json"
- "yarn.lock"
- "playwright.config.mjs"
- "Dockerfile"
- "docker/**"
- "xmcp.config.*"
- "tsconfig*.json"
- ".github/workflows/mcp-evals.yml"
- ".github/actions/host-tools/**"
workflow_dispatch:
inputs:
model:
description: "OpenRouter model ID for every case (empty keeps the dataset's model)"
required: false
default: ""
schedule:
- cron: "0 7 * * 1"

permissions:
contents: read

# Runs share the Cloud Pod name mcptester-e2e-pod in one LocalStack account, so they must not overlap.
# Merges that land while a run is going wait; GitHub keeps only the newest waiting run.
concurrency:
group: mcp-evals
cancel-in-progress: false

jobs:
mcp-evals:
runs-on: ubuntu-26.04
# The spec itself times out after 90 minutes; the rest covers setup and the results upload.
timeout-minutes: 120
env:
LOCALSTACK_AUTH_TOKEN: ${{ secrets.LOCALSTACK_AUTH_TOKEN }}
OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
EVAL_MODEL: ${{ inputs.model }}
steps:
- name: Checkout code
uses: actions/checkout@v7

- name: Use Node.js 22
uses: actions/setup-node@v7
with:
node-version: 22.x

- name: Install dependencies
run: yarn install --frozen-lockfile

# Terraform, lstk and the Snowflake CLI at the Docker image's versions.
- name: Install the image's tools
uses: ./.github/actions/host-tools
with:
snowflake: true

# The eval host caps each tool call at the MCP SDK's 60s request timeout, and the
# deployer runs init, apply and output in one call. Download the AWS provider first.
- name: Download the sample project's Terraform providers
run: terraform -chdir=data/sample-terraform init -input=false

# Pulled up front so a start inside a case doesn't time out on the download.
- name: Pull the LocalStack images
run: |
docker pull localstack/localstack-pro:latest
docker pull localstack/snowflake:latest

- name: Run the eval dataset
run: yarn test:mcp:evals

- name: Keep the eval results
if: always()
uses: actions/upload-artifact@v7
with:
name: mcp-evals-results
path: .mcp-test-results
# The directory starts with a dot, which upload-artifact skips by default.
include-hidden-files: true
if-no-files-found: ignore
7 changes: 7 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
# aws-sam-cli docker/tools/sam/uv.lock (uv)
# snowflake-cli docker/tools/snowflake/uv.lock (uv)
# dockerode & co. yarn.lock (npm, same lock as CI)
# The host-tools stage hands the same terraform and uv binaries to CI jobs that run on the host.
# Debian packages (python3, git, curl) come with the node image's bookworm base.
# Both node FROM lines must stay identical, and their major must match `node-version` in the
# workflows; src/core/dependency-pins.test.ts checks this.
Expand All @@ -27,6 +28,12 @@ FROM hashicorp/terraform:1.17.2 AS terraform
# uv installs the Python CLIs below; it is mounted at build time and not part of the image.
FROM ghcr.io/astral-sh/uv:0.13.9 AS uv

# Terraform and uv for CI jobs that run on the host, at the image's versions
# (.github/actions/host-tools): docker build --target host-tools --output type=local,dest=DIR .
FROM scratch AS host-tools
COPY --from=terraform /bin/terraform /
COPY --from=uv /uv /uvx /

FROM node:22.23.5-bookworm-slim AS runtime
ENV DEBIAN_FRONTEND=noninteractive \
PYTHONDONTWRITEBYTECODE=1 \
Expand Down
14 changes: 11 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -262,17 +262,25 @@ This repository includes [MCP Server Tester](https://github.com/gleanwork/mcp-se
```bash
yarn test:mcp:direct
```
- Run Gemini-based MCP host evals:
- Run each tool against a running LocalStack, without a model (deterministic; starts LocalStack if it isn't running):
```bash
export GOOGLE_GENERATIVE_AI_API_KEY="<your-gemini-key>"
export LOCALSTACK_AUTH_TOKEN="<your-localstack-auth-token>"
yarn test:mcp:tools
```
The tests deploy a sample project, switch IAM enforcement to soft mode and save a Cloud Pod, so point them at an instance you don't need: set `LOCALSTACK_PORT` and `MAIN_CONTAINER_NAME` to use one other than the default on 4566. They need Docker, Terraform and [lstk](https://github.com/localstack/lstk) on the `PATH`.
- Run MCP host evals through [OpenRouter](https://openrouter.ai) (default model `google/gemini-2.5-flash`; set `EVAL_MODEL` to any OpenRouter model ID to override it):
```bash
export OPENROUTER_API_KEY="<your-openrouter-key>"
export LOCALSTACK_AUTH_TOKEN="<your-localstack-auth-token>"
yarn test:mcp:evals
```
The evals check whether a model picks and combines the right tools for a goal; their prompts don't name the tools. They start, change and stop LocalStack on port 4566, so don't run them against an instance you're using. They need Docker, Terraform, [lstk](https://github.com/localstack/lstk) and the Snowflake CLI (`snow`) on the `PATH`, and the sample project's providers downloaded with `terraform -chdir=data/sample-terraform init`.
The [MCP evals workflow](./.github/workflows/mcp-evals.yml) runs them after every merge into `main` that changes the server, its tools or the evals, weekly and on demand; it needs the `OPENROUTER_API_KEY` repository secret.
- Open the latest MCP Server Tester HTML report:
```bash
npx mcp-server-tester open
```
- Run both:
- Run all of them:
```bash
yarn test:mcp
```
Expand Down
Loading
Loading