Skip to content

feat(deployer): run Terraform, CDK, and SAM through lstk - #97

Merged
joe4dev merged 1 commit into
mainfrom
update-docker-image
Nov 18, 2026
Merged

joe4dev merged 1 commit into
mainfrom
update-docker-image

Conversation

@bartsdev

Copy link
Copy Markdown
Member

Summary

The deployer runs the IaC CLIs through lstk instead of the *local wrappers.

 localstack-deployer
   checkDependencies
-    tflocal | cdklocal | samlocal --version
+    lstk --version
+    terraform version | cdk --version | sam --version
   execute
-    tflocal init / apply / output -json / destroy
-    cdklocal bootstrap / deploy / destroy
-    samlocal build / deploy / delete
+    lstk terraform init / apply / output -json / destroy
+    lstk cdk bootstrap / deploy / destroy
+    lstk sam build / deploy / delete

The child process environment (buildIacCliEnv) changes to suit lstk:

 buildIacCliEnv
   dummy credentials, region, CDK_DEFAULT_*
+  AWS_ENDPOINT_URL = http://$LOCALSTACK_HOSTNAME:$PORT    # always; localhost → localhost.localstack.cloud
-  AWS_ENDPOINT_URL_S3, S3_ENDPOINT, AWS_S3_FORCE_PATH_STYLE, EDGE_PORT, AWS_ENVAR_ALLOWLIST
  • AWS_ENDPOINT_URL is now always set. Without it, lstk only accepts the AWS container named in its own config.toml, which would reject a localstack-main container started by this server. With it, lstk probes /_localstack/health and uses that endpoint.
  • AWS_ENDPOINT_URL_S3 is left unset. lstk derives the S3 endpoint itself and forces path-style S3 for CDK 2.1138 and newer. A preset S3 endpoint turns that off.
 Dockerfile
-  pip: terraform-local, aws-sam-cli-local
-  npm: aws-cdk@2.1133.0, aws-cdk-local
-  aws-cdk forcePathStyle source patch
+  npm: aws-cdk@2.1145.0, @localstack/lstk@1.4.2
 docker run / wizard / docs
-  --add-host s3.host.docker.internal:host-gateway

aws-cdk 2.1145.0 applies CDK_S3_FORCE_PATH_STYLE, which lstk sets, in the same s3() client factory the old patch edited. The patch target no longer exists in that version. The S3 alias is now unnecessary, but existing configs that pass it keep working.

Evidence

  • Before: the deployer spawns tflocal, cdklocal and samlocal, and the image patches aws-cdk's source.
    After: 400/400 unit tests pass, plus new ones:
    checkDependencies
      ✓ requires lstk before the wrapped IaC CLI
      ✓ requires the IaC CLI lstk wraps
      ✓ reports both versions when lstk and the IaC CLI are installed
    buildIacCliEnv
      ✓ points lstk at localhost.localstack.cloud for a local LocalStack
      ✓ points lstk at the host gateway when LocalStack runs outside the container  (no AWS_ENDPOINT_URL_S3)
    
  • I ran lstk 1.4.2 against a stub /_localstack/health endpoint with no TTY attached:
    $ AWS_ENDPOINT_URL=http://127.0.0.1:45668 lstk terraform output -json </dev/null
    {}                       # stdout is terraform's JSON only, exit 0, override file removed
    $ LSTK_TF_DRY_RUN=1 lstk terraform plan
    provider "aws" { s3_use_path_style = true  endpoints { s3 = "http://127.0.0.1:45668" … } }
    
  • Not run locally: the image build and real deploys (no Docker daemon). The Docker workflow's validate-image.mjs covers the Terraform and CDK deploys. It now runs without the S3 alias, and its CDK check looks for Error during `lstk cdk`.

Merge Danger

Door: two-way

Blast Radius: npx deployer users

  • npx and from-source users: the deployer now needs lstk plus terraform, cdk (2.177 or newer) or sam (1.95 or newer) on PATH. With only the *local wrappers installed, it returns "Dependency Not Available" with install steps.
  • SAM: lstk sam doesn't support container-image (ECR) Lambdas or nested CloudFormation stacks, according to the lstk docs. samlocal did.
  • Docker image users: nothing to change. The S3 alias still works but is no longer needed.

🤖 Generated with Claude Code

Replace tflocal, cdklocal, and samlocal with `lstk terraform|cdk|sam`.
The deployer always passes AWS_ENDPOINT_URL so lstk targets the
configured LocalStack instead of its own config.toml container, and
leaves the S3 endpoint to lstk, which forces path-style S3 for CDK.

The Docker image installs lstk 1.4.2 and aws-cdk 2.1145.0, drops the
*local wrappers and the aws-cdk path-style patch, and no longer needs
the s3.host.docker.internal alias.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@joe4dev

joe4dev commented Nov 17, 2026

Copy link
Copy Markdown
Member

Local test: SAM deploy through the MCP server

CI's smoke test covers Terraform and CDK but not SAM, so I deployed the s3-lambda sample from appinspector-samples through this branch's MCP server, in both npx and Docker mode. I drove it over stdio with the MCP SDK, the way an agent would.

Result: every step passed in both modes.

Step (MCP tool) npx mode Docker mode
Start LocalStack (localstack-management) ✅ at localhost:4610 ✅ at host.docker.internal:4610
lstk sam build --cached (localstack-deployer) ✅ ✅
lstk sam deploy … --resolve-s3 ✅ CREATE_COMPLETE ✅ CREATE_COMPLETE
Stack, Lambda and buckets exist (localstack-aws-client) ✅ ✅
Put an object in the source bucket → the Lambda writes it to the target bucket ✅ ✅
lstk sam delete --no-prompts ✅ DELETE_COMPLETE ✅ DELETE_COMPLETE
Stop LocalStack ✅ ✅

What this confirms

  • lstk sam works as a drop-in replacement for samlocal for a plain zip-packaged SAM app: build, deploy, invoke and delete.
  • Docker mode needs no S3 alias. The run passed only --add-host host.docker.internal:host-gateway. SAM uploaded its build artifacts (17 MB) to the bucket --resolve-s3 creates. botocore uses path-style URLs for any custom endpoint, so SAM never needs s3.host.docker.internal.
  • A non-default port works. With LOCALSTACK_PORT=4610, the endpoint buildIacCliEnv builds reached the right instance in both modes.
  • The image builds and runs on arm64 (Apple silicon) with lstk 1.4.2. CI skips the multi-arch build on PRs.

Setup

  • npx mode: this branch (06fa75b) built locally, with lstk 1.4.3 and sam 1.171.0 on the host. The sample was used unmodified.
  • Docker mode: the image built from this branch's Dockerfile (lstk 1.4.2, aws-sam-cli 1.142.1). The sample's runtime was changed from python3.13 to python3.11, because the image ships only Python 3.11. Without that change, sam build fails in Docker mode for any python3.13 app. That gap already exists on main and isn't caused by this PR, but it may deserve a follow-up.

Not covered

  • SAM apps with container-image (ECR) Lambdas or nested stacks. lstk documents both as unsupported, so the regression mentioned under Merge Danger still needs a decision.
  • Terraform and CDK. CI's smoke test already covers them.

🤖 Generated with Claude Code

@joe4dev

joe4dev commented Nov 17, 2026

Copy link
Copy Markdown
Member

Local test: SAM with ECR image Lambdas and nested stacks

Follow-up to my earlier comment. "Merge Danger" says lstk sam doesn't support container-image (ECR) Lambdas or nested stacks. I tested both, through this branch's MCP server (lstk sam) and with samlocal, called the same way the deployer on main called it.

Result: the ECR regression is real. Nested stacks deploy fine.

Scenario lstk sam (this PR, via MCP) samlocal (main)
ECR image Lambda, resolve_image_repos = true ❌ deploy fails ✅ deploys, invoke returns 200
ECR image Lambda, LocalStack repository URI set as image_repository ❌ SAM rejects the URI not tested
Nested stack (AWS::Serverless::Application with a local child template): deploy ✅ parent and child Lambdas invoke with 200 ✅ same
Nested stack: delete ❌ sam delete crashes ❌ the same crash

ECR image Lambdas: confirmed regression, no workaround

SAM builds the repository URI with the real AWS registry hostname, so docker push goes to AWS ECR instead of LocalStack:

The push refers to repository [000000000000.dkr.ecr.us-east-1.amazonaws.com/ecrlambda3296c98d/imagefunction069e6a78repo]
unknown: unexpected status from HEAD request to https://000000000000.dkr.ecr.us-east-1.amazonaws.com/v2/…: 400 Bad Request
Error: Unable to upload artifact imagefunction:v1 referenced by ImageUri parameter of ImageFunction resource.

Setting a LocalStack repository URI explicitly doesn't help, because SAM's own validation rejects any URI outside amazonaws.com:

Error: Invalid value for '--image-repository': Invalid Image Repository ECR URI: 000000000000.dkr.ecr.us-east-1.localhost.localstack.cloud:4610/ecr-lambda-explicit

samlocal patches both the URI and that check. lstk sam patches neither, so users currently have no workaround.

Nested stacks: not a regression

lstk sam deploy creates the parent and child stacks, and both functions run. sam delete then fails while parsing the path-style template URL it uploaded:

Error: URL given to the parse method is not a valid S3 url http://localhost.localstack.cloud:4610/aws-sam-cli-managed-default-samclisourcebucket-70cbe2a8/6b1b628db78e826eef633086c46e4ab3.template

samlocal delete fails with exactly the same error, so this exists on main too and isn't caused by this PR. lstk's help text (cmd/sam.go) and this PR's "Merge Danger" section overstate the nested-stack limitation.

Suggestions

  1. In "Merge Danger", name only ECR image Lambdas, and drop nested stacks.
  2. For PackageType: Image templates, either fall back to samlocal when it is installed, or refuse them with a clear message. Today the user gets SAM's push error, which mentions amazonaws.com.
  3. Also seen on main: when a deploy or destroy fails, the deployer still returns isError: false, and the error appears only inside the markdown. A failed destroy even ends with "has been deleted". An agent can read that as success.

Setup

  • npx mode: this branch (06fa75b) built locally, with lstk 1.4.3 and SAM CLI 1.171.0. The baseline used aws-sam-cli-local 1.71.0 on SAM CLI 1.172.1. Apple silicon (arm64).
  • LocalStack Pro ran on port 4610 with GATEWAY_LISTEN=:4610 and LOCALSTACK_HOST=localhost.localstack.cloud:4610. ECR only works when the host and container ports match.
  • Both test apps are minimal. The ECR app is a public.ecr.aws/lambda/python:3.12 image with an arm64 handler. The nested app has a parent function plus an AWS::Serverless::Application whose child template sits in the same folder and defines one function.

🤖 Generated with Claude Code

@joe4dev joe4dev left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reasonable following our deprecation of the local wrapper tools.

I assume the regression (sam fails with ECR images such as image-based Lambda functions) mentioned in the review section is a calculated risk. See testing results #97 (comment)

@joe4dev

joe4dev commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

⚠️ Merging this PR breaks the deployer via Docker because the wrapper tools are not installed anymore. This will remain an issue for users on older MCP versions, but even breaks latest because the Docker image is pushed on every build but NPM is released manually 😬

This turns out to be wrong because the build copies the source into the Docker image during build. The build still ships the new changes immediately, but there is no coupling between an (unreleased) NPM version and the shipped Docker image.

What stays: any changes merged into main go live immediately in the Docker MCP server ⚠️ , but require manual releasing via NPM

@joe4dev
joe4dev merged commit 5b91909 into main Nov 18, 2026
4 checks passed
joe4dev added a commit that referenced this pull request Nov 19, 2026
* feat(evals): run the MCP host evals through OpenRouter and in CI

Route the eval dataset through OpenRouter (same google/gemini-2.5-flash
model) instead of the Google provider, add EVAL_MODEL to swap the model,
and declare ai and @openrouter/ai-sdk-provider as dev dependencies.

Add a weekly and on-demand MCP evals workflow modeled on Azure smoke.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(evals): fail when a tool call fails, and run the deployer through lstk

Every case only checked that the model called the right tool, so a case passed
even when the tool returned an error. Each case now also matches a success
marker inside a tool call's output. The tester matches against the whole
simulation result as JSON, so each pattern starts with a prefix that keeps it
inside one output string; otherwise the model's answer or the scenario could
satisfy it. The spec requires every case to pass, and counts iterations lost to
infrastructure errors (rate limits, 5xx) as failures instead of dropping them.
The Cloud Pods case runs right after LocalStack starts, while a save is still
quick enough for the tester's 60s call limit.

Since #97 the deployer runs `lstk terraform`, so the workflow installs lstk
instead of tflocal and downloads the sample project's providers up front.
lstk and the Snowflake CLI come from the image's lockfiles; Terraform and uv
from the Dockerfile's FROM lines, which a pin test now enforces for every
workflow. Runs no longer overlap, because they share one Cloud Pod name, and
the job timeout leaves room for setup and the upload, which now includes the
results directory (upload-artifact skips dot directories by default).

Drop the unused `ai` devDependency: the tester loads its own copy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci: copy Terraform and uv out of the Dockerfile instead of parsing it

The evals workflow read the Terraform and uv versions out of the Dockerfile's
FROM lines with sed and passed them to setup actions. A new `host-tools` stage
copies the binaries from the same stages the image uses, and a composite action
(.github/actions/host-tools) builds it, installs lstk and optionally the
Snowflake CLI from the image's lockfiles, and puts them on the PATH. The
Dockerfile stays the one place Dependabot bumps these versions, and every job
that needs the tools uses the action. The pin test now fails when a workflow
installs Terraform or uv any other way.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: check each tool without a model, and evals for tool selection only

tests/mcp/tools.spec.mjs calls each tool against a running LocalStack in a
fixed order (start, deploy, list, inspect, IAM, chaos, Cloud Pods, replicator,
extensions, ephemeral instances) and checks text only a successful result
contains. It needs no model, so a new CI job runs it on every pull request.
It calls the SDK client directly with a 5-minute timeout: the tester's
callTool applies the SDK's 60s default, which a start or deploy can exceed.

With the tools covered there, the evals only check whether a model picks and
combines the right tools. Their prompts now describe a goal instead of naming
the tool, the cases that only repeated a single tool call moved to the tool
tests, and each case runs once: 9 model runs instead of 68.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: require the host-tools action in jobs that run the tool tests or evals

The pin test only banned the setup actions and download URLs, so a job could
install Terraform or uv with pip, pipx, brew, apt, mise or asdf, or skip the
action and use whatever the runner image ships. It now bans those installs
too, and fails when a job that runs `test:mcp`, `test:mcp:tools` or
`test:mcp:evals` doesn't use .github/actions/host-tools.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: record eval tokens and cost by patching mcp-server-tester 1.1.1

The tester's AI SDK host read `usage.promptTokens` and `completionTokens`,
names the AI SDK dropped in v5, and hardcoded `totalCostUsd: 0`, so every
report showed 0 tokens and $0. A patch-package patch reads `totalUsage`, which
covers every step of a multi-step case, and sums the cost OpenRouter returns
for each step. `prepare` applies it on install; unlike `postinstall` it doesn't
run when users install the published package. When Dependabot bumps the
tester, the patch must be redone or dropped. 2.0 counts tokens but not cost.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: check the state tools leave behind, and fix review findings

Mutation testing showed the tool tests passed when a tool skipped its API call
but still reported success, because they checked the tool's own message. Each
test now reads the state back: IAM and App Inspector switch between two
states and read each back, chaos reads the latency after clearing it, Cloud
Pods removes the bucket after the save and checks the load brings it back, a
second delete must fail, traces must list a row, the logs analysis must report
a request made to fail, the deploy passes name_prefix and checks the outputs
table, and the region-bound SQS queue must be listed.

The evals fail a case when every call of a required tool returned an error
(the tester drops isError; this server's errors start with ❌), and the chaos
and IAM prompts ask for a read-back that the patterns check.

Review fixes: fork PRs skip the jobs that need secrets; the Cloud Pods test
cleans up without masking the first error and names the pod per attempt;
chaos and IAM restore state on failure; Dependabot no longer bumps the
patched tester; the pin test also catches `yarn run`/`npm run` jobs and curl
installers, and no longer flags links to the uv docs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci: fail fork PRs that lack the LocalStack token instead of skipping the jobs

A skipped job looks green, so a fork PR could merge without anyone noticing it was never
tested against LocalStack. Both jobs now fail in their first step with an error that says
how a maintainer can run them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci: fail the Docker integration test loud when the LocalStack token is missing

Same as the CI jobs: on a fork PR, smoke (amd64) skipped the integration test and still
showed green.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* TEMPORARY: run the evals workflow on push to the PR branch

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Revert the temporary push trigger for the evals workflow

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci: run the evals after every merge into main

The weekly run stays, to catch changes outside this repo such as a new LocalStack image or
model version.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci: run the evals only for merges that can change them, and let forks build the image

The eval trigger on main gets a path filter, so README edits or Dependabot action bumps don't
start a run. The Docker token check moves right before the integration test, so a fork PR
still builds the image and runs the startup smoke test before failing loud.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Bart <bart@Barts-MacBook-Pro.local>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants