Repository navigation
feat(deployer): run Terraform, CDK, and SAM through lstk - #97
Conversation
Replace tflocal, cdklocal, and samlocal with `lstk terraform|cdk|sam`. The deployer always passes AWS_ENDPOINT_URL so lstk targets the configured LocalStack instead of its own config.toml container, and leaves the S3 endpoint to lstk, which forces path-style S3 for CDK. The Docker image installs lstk 1.4.2 and aws-cdk 2.1145.0, drops the *local wrappers and the aws-cdk path-style patch, and no longer needs the s3.host.docker.internal alias. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Local test: SAM deploy through the MCP serverCI's smoke test covers Terraform and CDK but not SAM, so I deployed the Result: every step passed in both modes.
What this confirms
Setup
Not covered
🤖 Generated with Claude Code |
Local test: SAM with ECR image Lambdas and nested stacksFollow-up to my earlier comment. "Merge Danger" says Result: the ECR regression is real. Nested stacks deploy fine.
ECR image Lambdas: confirmed regression, no workaroundSAM builds the repository URI with the real AWS registry hostname, so Setting a LocalStack repository URI explicitly doesn't help, because SAM's own validation rejects any URI outside
Nested stacks: not a regression
Suggestions
Setup
🤖 Generated with Claude Code |
joe4dev
left a comment
There was a problem hiding this comment.
Reasonable following our deprecation of the local wrapper tools.
I assume the regression (sam fails with ECR images such as image-based Lambda functions) mentioned in the review section is a calculated risk. See testing results #97 (comment)
|
This turns out to be wrong because the build copies the source into the Docker image during build. The build still ships the new changes immediately, but there is no coupling between an (unreleased) NPM version and the shipped Docker image. What stays: any changes merged into main go live immediately in the Docker MCP server |
* feat(evals): run the MCP host evals through OpenRouter and in CI Route the eval dataset through OpenRouter (same google/gemini-2.5-flash model) instead of the Google provider, add EVAL_MODEL to swap the model, and declare ai and @openrouter/ai-sdk-provider as dev dependencies. Add a weekly and on-demand MCP evals workflow modeled on Azure smoke. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(evals): fail when a tool call fails, and run the deployer through lstk Every case only checked that the model called the right tool, so a case passed even when the tool returned an error. Each case now also matches a success marker inside a tool call's output. The tester matches against the whole simulation result as JSON, so each pattern starts with a prefix that keeps it inside one output string; otherwise the model's answer or the scenario could satisfy it. The spec requires every case to pass, and counts iterations lost to infrastructure errors (rate limits, 5xx) as failures instead of dropping them. The Cloud Pods case runs right after LocalStack starts, while a save is still quick enough for the tester's 60s call limit. Since #97 the deployer runs `lstk terraform`, so the workflow installs lstk instead of tflocal and downloads the sample project's providers up front. lstk and the Snowflake CLI come from the image's lockfiles; Terraform and uv from the Dockerfile's FROM lines, which a pin test now enforces for every workflow. Runs no longer overlap, because they share one Cloud Pod name, and the job timeout leaves room for setup and the upload, which now includes the results directory (upload-artifact skips dot directories by default). Drop the unused `ai` devDependency: the tester loads its own copy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci: copy Terraform and uv out of the Dockerfile instead of parsing it The evals workflow read the Terraform and uv versions out of the Dockerfile's FROM lines with sed and passed them to setup actions. A new `host-tools` stage copies the binaries from the same stages the image uses, and a composite action (.github/actions/host-tools) builds it, installs lstk and optionally the Snowflake CLI from the image's lockfiles, and puts them on the PATH. The Dockerfile stays the one place Dependabot bumps these versions, and every job that needs the tools uses the action. The pin test now fails when a workflow installs Terraform or uv any other way. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: check each tool without a model, and evals for tool selection only tests/mcp/tools.spec.mjs calls each tool against a running LocalStack in a fixed order (start, deploy, list, inspect, IAM, chaos, Cloud Pods, replicator, extensions, ephemeral instances) and checks text only a successful result contains. It needs no model, so a new CI job runs it on every pull request. It calls the SDK client directly with a 5-minute timeout: the tester's callTool applies the SDK's 60s default, which a start or deploy can exceed. With the tools covered there, the evals only check whether a model picks and combines the right tools. Their prompts now describe a goal instead of naming the tool, the cases that only repeated a single tool call moved to the tool tests, and each case runs once: 9 model runs instead of 68. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: require the host-tools action in jobs that run the tool tests or evals The pin test only banned the setup actions and download URLs, so a job could install Terraform or uv with pip, pipx, brew, apt, mise or asdf, or skip the action and use whatever the runner image ships. It now bans those installs too, and fails when a job that runs `test:mcp`, `test:mcp:tools` or `test:mcp:evals` doesn't use .github/actions/host-tools. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: record eval tokens and cost by patching mcp-server-tester 1.1.1 The tester's AI SDK host read `usage.promptTokens` and `completionTokens`, names the AI SDK dropped in v5, and hardcoded `totalCostUsd: 0`, so every report showed 0 tokens and $0. A patch-package patch reads `totalUsage`, which covers every step of a multi-step case, and sums the cost OpenRouter returns for each step. `prepare` applies it on install; unlike `postinstall` it doesn't run when users install the published package. When Dependabot bumps the tester, the patch must be redone or dropped. 2.0 counts tokens but not cost. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: check the state tools leave behind, and fix review findings Mutation testing showed the tool tests passed when a tool skipped its API call but still reported success, because they checked the tool's own message. Each test now reads the state back: IAM and App Inspector switch between two states and read each back, chaos reads the latency after clearing it, Cloud Pods removes the bucket after the save and checks the load brings it back, a second delete must fail, traces must list a row, the logs analysis must report a request made to fail, the deploy passes name_prefix and checks the outputs table, and the region-bound SQS queue must be listed. The evals fail a case when every call of a required tool returned an error (the tester drops isError; this server's errors start with ❌), and the chaos and IAM prompts ask for a read-back that the patterns check. Review fixes: fork PRs skip the jobs that need secrets; the Cloud Pods test cleans up without masking the first error and names the pod per attempt; chaos and IAM restore state on failure; Dependabot no longer bumps the patched tester; the pin test also catches `yarn run`/`npm run` jobs and curl installers, and no longer flags links to the uv docs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci: fail fork PRs that lack the LocalStack token instead of skipping the jobs A skipped job looks green, so a fork PR could merge without anyone noticing it was never tested against LocalStack. Both jobs now fail in their first step with an error that says how a maintainer can run them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci: fail the Docker integration test loud when the LocalStack token is missing Same as the CI jobs: on a fork PR, smoke (amd64) skipped the integration test and still showed green. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * TEMPORARY: run the evals workflow on push to the PR branch Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Revert the temporary push trigger for the evals workflow Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci: run the evals after every merge into main The weekly run stays, to catch changes outside this repo such as a new LocalStack image or model version. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci: run the evals only for merges that can change them, and let forks build the image The eval trigger on main gets a path filter, so README edits or Dependabot action bumps don't start a run. The Docker token check moves right before the integration test, so a fork PR still builds the image and runs the startup smoke test before failing loud. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Bart <bart@Barts-MacBook-Pro.local> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Summary
The deployer runs the IaC CLIs through
lstkinstead of the*localwrappers.The child process environment (
buildIacCliEnv) changes to suit lstk:AWS_ENDPOINT_URLis now always set. Without it, lstk only accepts the AWS container named in its ownconfig.toml, which would reject alocalstack-maincontainer started by this server. With it, lstk probes/_localstack/healthand uses that endpoint.AWS_ENDPOINT_URL_S3is left unset. lstk derives the S3 endpoint itself and forces path-style S3 for CDK 2.1138 and newer. A preset S3 endpoint turns that off.aws-cdk 2.1145.0 applies
CDK_S3_FORCE_PATH_STYLE, which lstk sets, in the sames3()client factory the old patch edited. The patch target no longer exists in that version. The S3 alias is now unnecessary, but existing configs that pass it keep working.Evidence
tflocal,cdklocalandsamlocal, and the image patches aws-cdk's source.After: 400/400 unit tests pass, plus new ones:
/_localstack/healthendpoint with no TTY attached:validate-image.mjscovers the Terraform and CDK deploys. It now runs without the S3 alias, and its CDK check looks forError during `lstk cdk`.Merge Danger
Door: two-way
Blast Radius: npx deployer users
lstkplusterraform,cdk(2.177 or newer) orsam(1.95 or newer) on PATH. With only the*localwrappers installed, it returns "Dependency Not Available" with install steps.lstk samdoesn't support container-image (ECR) Lambdas or nested CloudFormation stacks, according to the lstk docs.samlocaldid.🤖 Generated with Claude Code