From d47ac4512384e42f5fb7ceffa3f48b68f5d92bcd Mon Sep 17 00:00:00 2001 From: Joel Scheuner Date: Wed, 18 Nov 2026 14:50:57 +0100 Subject: [PATCH 01/13] docs: add agent design rules, approved exceptions and review rules AGENTS.md records the design rules for the MCP servers (compute in LocalStack, public interfaces only) and the approved client-side exceptions, starting with grouping App Inspector failed requests. Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 220 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ CLAUDE.md | 1 + REVIEW.md | 107 ++++++++++++++++++++++++++ 3 files changed, 328 insertions(+) create mode 100644 AGENTS.md create mode 100644 CLAUDE.md create mode 100644 REVIEW.md diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..5b5c97a --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,220 @@ +# AGENTS.md + +The LocalStack MCP server lets an MCP client (Claude Code, Cursor, VS Code, …) start LocalStack, deploy +infrastructure, run cloud CLIs, and inspect the emulator. It is a TypeScript server built on +[xmcp](https://xmcp.dev), published to npm as `@localstack/localstack-mcp-server` and as the Docker image +`localstack/localstack-mcp-server`. + +Read [REVIEW.md](REVIEW.md) before opening a PR. It turns the rules below into review checks. + +## Commands + +```bash +yarn install +yarn build # xmcp build + esbuild CLI bundle → dist/ +yarn dev # xmcp dev server +yarn test # Jest unit tests (*.test.ts next to the code) +yarn test:mcp:direct # Playwright: calls the built server's tools directly (needs Docker + LOCALSTACK_AUTH_TOKEN) +yarn test:mcp:evals # Playwright: LLM evals against the tools (needs a Gemini key) +yarn format # Prettier +``` + +## Layout + +| Path | Contents | +|------|----------| +| `src/tools/` | One file per MCP tool. xmcp discovers them: each exports `schema` (zod), `metadata`, and a default handler. | +| `src/prompts/` | MCP prompts (workflow templates such as `infrastructure-tester`). | +| `src/core/` | Shared plumbing: `preflight.ts`, `response-builder.ts`, `analytics.ts`, `command-runner.ts`, `config.ts`, `http-client.ts`. | +| `src/lib//` | Tool logic. Pure, testable logic lives in `*.logic.ts` with a matching `*.logic.test.ts`. | +| `src/lib/docker/` | Docker Engine API client (find, start, stop, exec, logs). | +| `src/lib/localstack/` | LocalStack HTTP clients and the container spec. | +| `src/cli/` | The `init` / `remove` setup wizard and the stdio entry point. | +| `tests/mcp/` | Playwright tests against the built server. | +| `tests/docker/` | Validation of the Docker image. | +| `docs/` | Topic docs (`AZURE.md`, `DOCKER.md`). | + +## Writing a tool + +Follow the shape of an existing tool, for example `src/tools/localstack-aws-client.ts`: + +1. Wrap the handler in `withToolAnalytics(, args, …)`. Add the tool to the field allowlist in + `src/core/analytics.ts`, and list only fields that can never hold secrets or user data. +2. Run preflights first (`runPreflights([requireAuthToken(), requireLocalStackRunning(), …])`). +3. Return every result through `ResponseBuilder` (`markdown`, `error`, …). +4. Put parsing, validation and decision logic in `src/lib//*.logic.ts` and unit-test it there. +5. Never pass user input to a shell. Spawn binaries with argument arrays (`runCommand`), and sanitize CLI strings + like `src/lib/aws/aws-cli-sanitizer.ts` does. +6. Update the tool and prerequisite tables in `README.md`. + +## Design rules + +### 1. Work with lstk, never with the legacy CLI + +[lstk](https://github.com/localstack/lstk) ([docs](https://docs.localstack.cloud/aws/developer-tools/running-localstack/lstk/)) +is the LocalStack CLI. Do not call, document or recommend the legacy `localstack` CLI +([legacy docs](https://docs.localstack.cloud/aws/developer-tools/running-localstack/localstack-cli/)): no +`localstack start`, `localstack extensions`, `localstack pod`, and no hints that tell users to install it. + +The MCP server keeps its own lifecycle code: it starts and stops the emulator through the Docker API, so it works +without lstk. That container must be interchangeable with one lstk starts: + +- **Same container.** Container name, image, ports, labels, volume layout and auth env match what lstk creates. + The reference is lstk's `internal/config/containers.go` and `internal/container/`. When lstk changes them, follow. +- **Either tool can take over.** `lstk stop` / `lstk status` work on an emulator the MCP server started, and the MCP + server's `stop` / `status` work on one lstk started. Find the container the way lstk marks it (name and labels), + not by a name only this server uses. +- **Same endpoint.** When the MCP server calls lstk, it passes its own target (`LOCALSTACK_HOSTNAME` / + `LOCALSTACK_PORT`) as `LSTK_ENDPOINT_URL`, so both talk to the same emulator. + +### 2. Prefer lstk's cloud and IaC commands over dedicated wrappers + +lstk wraps the cloud CLIs and IaC tools +([docs](https://docs.localstack.cloud/aws/developer-tools/running-localstack/lstk/cloud-and-iac-commands/)): +`lstk aws`, `lstk az`, `lstk terraform`, `lstk cdk`, `lstk sam`. Use them instead of `awslocal`, `tflocal`, +`cdklocal`, `samlocal`, or a hand-configured `az`. + +- **lstk first, legacy fallback.** When `lstk` is on `PATH`, use it. Otherwise fall back to the dedicated wrapper, + so a setup without lstk keeps working. Resolve this in one shared helper, not per tool. +- **Say which path ran.** The tool response names the command that ran (`lstk aws …` or `awslocal …`), so a user + can reproduce it. +- **Hint toward lstk.** When the fallback runs, or no runner is found, the message recommends installing lstk. It + never recommends the legacy CLI. +- **Keep the guards.** Moving a tool to lstk does not remove its safety checks: the AWS CLI sanitizer, and the Azure + tool's profile isolation, egress guard and command policy (`src/lib/azure/`). +- **Docker image.** The image bundles lstk and the AWS CLI, so the lstk path also runs there. +- **Snowflake is exempt.** The Snowflake tool uses the `snow` CLI until lstk wraps it. + +### 3. Prefer LocalStack APIs over parsing logs + +Get data from LocalStack's +[internal endpoints](https://docs.localstack.cloud/aws/customization/networking/internal-endpoints/) (`/_localstack/*`, +`/_aws/*`). The emulator publishes their OpenAPI spec at `/_localstack/swagger`. Log text is not an interface: its +format changes without notice, lines rotate out of the window, and regexes miss cases. + +- **Structured answers come from an API.** Which calls failed, which IAM checks denied a request, which services ran: + read these from an endpoint, for example App Inspector traces, spans and IAM events (`/_localstack/appinspector`). +- **No API, no feature.** If no endpoint exposes the data, ask for one in LocalStack rather than adding a log parser. + A tool may explain how to enable the feature that provides the data (for example App Inspector), but must not fall + back to logs. +- **Logs are for showing logs.** `localstack-logs-analysis` returns raw or filtered log lines (startup failures, stack + traces). It does not classify lines into services, operations, status codes or IAM denials. + +### 4. Use only public interfaces + +This repository is public. Do not reference code from private LocalStack repositories: + +- no Python module paths (`localstack.pro.*`), private file paths, or class names in code, comments or docs +- no undocumented endpoints, and no scripts executed inside the LocalStack container to reach internals +- no undocumented configuration variables + +Allowed: documented endpoints (in the OpenAPI spec or the docs), documented configuration variables (for example +`EXTENSION_AUTO_INSTALL`), the public LocalStack Platform API, and public repositories such as lstk. + +Exception: an endpoint listed under [Gaps in LocalStack](#gaps-in-localstack) may be used before it is documented. +Each such endpoint is one LocalStack's own clients (the web app, App Inspector) already depend on. Describe its +request and response in this repository's code (types and tests); do not link to or quote its implementation. + +### 5. Compute in LocalStack, render in the MCP server + +The MCP server is a client of LocalStack, next to the App Inspector UI and the coding agents that read its output. +If two clients could compute the same answer differently, the answer belongs in LocalStack. + +- **LocalStack decides:** which policy denied a request, which policy to change and how, whether a change fixes the + request, which resource holds a resource-based policy. +- **The MCP server presents:** it fetches, filters, formats and explains. It may choose wording and structure for the + agent, but it does not parse policy documents, match actions or resources, or synthesize policies. +- **Missing data:** if LocalStack does not return what a tool needs, request a field in LocalStack. A tool may say + the answer is unavailable on an older emulator, but must not compute it itself as a fallback. +- **Exceptions for a better experience:** when computing something in the client would clearly improve what the agent + or user gets, and LocalStack cannot offer it yet, it may be done as a documented exception, but only with the + user's explicit approval. This is not a default way out of the rule: first propose the LocalStack change, then + ask the user, naming the computation, what it improves, and what it would take in LocalStack. Never assume + approval. An approved exception gets a row in [Approved exceptions](#approved-exceptions) and a + [Gaps in LocalStack](#gaps-in-localstack) row for the API that would replace it; delete both once that API + ships. Exceptions never cover decisions about IAM (which policy denied, what fixes it), and never read logs + (rule 3). + +## Features + +### IAM policies (`localstack-app-inspector`) + +`localstack-app-inspector` owns IAM. `localstack-iam-policy-analyzer` is deprecated: its behaviour is unchanged +until it is removed, but its description and every response name the app-inspector action that replaces it. + +The IAM actions have three jobs. Each reads a LocalStack API; without the feature behind it, the response explains +what is missing instead of returning an empty answer. + +1. **Why was a request denied, and what fixes it.** `list-iam-denials`, `explain-iam`, `get-iam-policy`. Source: + App Inspector IAM events (`/_localstack/appinspector/v1/traces/{t}/spans/{s}/events/iam`, `iam.policy.denied` for + denials, also in soft mode with `enforcement: allow_with_warning`). The event's `payload` attribute is a JSON + string holding the decisions (`explicit_allows`, `explicit_denies`, `implicit_denies`, each with `caused_by` + policy keys and statement positions), the `policies` map, and for a denial LocalStack's `suggestion` (version 1: + `outcome`, `entries`, `hints`, `near_misses`). `explain-iam` renders them in the section order of the App + Inspector UI's "Copy agent prompt", then the payload as sent. Do not recompute, merge or reorder the changes. + A policy stored truncated (`statement_count` without `document`) is fetched on demand with `get-iam-policy` + (`/v1/events/{event_id}/policies/{key}`), never automatically. + - Without the `localstack.appinspector/view_iam_policy_errors` entitlement, LocalStack hides IAM events and + returns an empty list. Only spans say so (`iam_errors_suppressed`), so an empty IAM result reads one span of + the scope and reports suppression, or else the IAM mode. `/_localstack/licenseinfo` cannot tell: it lists + plugins, not entitlements. + - A denial without `suggestion` comes from an older LocalStack: say so and suggest upgrading. A suggestion + `version` other than 1 is shown as JSON only. +2. **Which policies does the workload need.** `get-iam-policy-summary`, `reset-iam-policy-summary`. Source: the IAM + policy summary (`GET`/`DELETE /_aws/iam/policies/summary`, feature `localstack.platform.plugin/iam-stream`; a 404 + means the feature is not licensed or not enabled). It is global to the emulator: there is no per-trace summary + (see [Gaps in LocalStack](#gaps-in-localstack)). Each entry is one policy for one ARN: `resource`, + `policy_type` (`identity` or `resource`) and `policy_document`. Present it the way App Inspector's Policies view + does ([localstack-appinspector-ui#341](https://github.com/localstack/localstack-appinspector-ui/pull/341)): + - one entry per ARN, with its name (the ARN's last part), its type, its services and its number of actions + - the policy JSON exactly as sent: a role's trust statement and its permissions stay in one policy + - always labelled as a suggestion, derived from the requests seen since the last reset, not as a verified policy +3. **How do the deployed policies differ from what the workload needs.** For each ARN in the summary: the + permissions to add, and the granted permissions the run never used. This compares policy documents, so + LocalStack computes it (rule 5). LocalStack does not offer it yet (see [Gaps in LocalStack](#gaps-in-localstack)), + so the tool does not either. + +The enforcement mode (`get-iam-mode`, `set-iam-mode`, through `/_aws/iam/config`) is a separate action. Denials and +the policy summary are only recorded while the mode is `ENFORCED` or `SOFT_MODE`. + +The agentic evaluation in `evals/agentic/` checks that a coding agent fixes common IAM failures with these actions. + +## Gaps in LocalStack + +Things this server needs from LocalStack that are missing or undocumented. Request them in LocalStack, and remove a +row once it ships. + +| Need | Status | Used by | +|------|--------|---------| +| Document the extensions API (`GET /_localstack/extensions/list`, `POST …/install`, `POST …/uninstall`) in the OpenAPI spec and the docs | The LocalStack web app's Extension Manager depends on it; neither the spec nor the docs name it. lstk has no extensions commands. | `localstack-extensions` (allowed under the rule 4 exception) | +| Document the IAM policy summary and stream (`GET`/`DELETE /_aws/iam/policies/summary`, `GET /_aws/iam/policies/stream`) | App Inspector's Policies view and the web app's IAM Policy Stream depend on them; the spec documents only `/_aws/iam/config` and `/_aws/iam/check-actions-allowed`. | `localstack-app-inspector` IAM job 2 (allowed under the rule 4 exception) | +| Filter the IAM policy summary by trace (or time range), so a run's policies don't mix with earlier requests | Not implemented: the summary is global and only resettable. | `get-iam-policy-summary` | +| Say on the App Inspector events page when IAM events are hidden by the license (as spans do with `iam_errors_suppressed`) | Not implemented: the events routes return an empty list. The tool probes one span instead. | `list-iam-denials`, `explain-iam` | +| Compare suggested policies with deployed ones: permissions to add and permissions granted but unused, per ARN | Not implemented. Listed as a follow-up in localstack-appinspector-ui#341. | IAM job 3 | +| A failed-requests summary in App Inspector: failed spans grouped by service, operation, HTTP status and AWS error code, with counts, first and last time, and an example trace and span | Not implemented. | `lstk mcp`'s `localstack-app-inspector` `failed-requests` (an [approved exception](#approved-exceptions) until then) | + +## Approved exceptions + +Client-side computations allowed under rule 5's exception, each approved by the user. Remove a row when the +LocalStack API that replaces it ships. + +| Where | Computation | Why | Approved | Replaced by | +|-------|-------------|-----|----------|-------------| +| `lstk mcp`, `localstack-app-inspector` action `failed-requests` (lstk `internal/mcp/appinspector_render.go`) | Groups App Inspector's failed spans (`status_code=2`) by service, operation, HTTP status and AWS error code, with counts, first and last time, and the latest example. Reads at most 5,000 spans. | Replaces the log-based error summary of `localstack-logs-analysis` with one from App Inspector's data; without it an agent pages through raw spans or greps logs. | 2026-11-17, by the user, during the `lstk mcp` prototype | The failed-requests summary in [Gaps in LocalStack](#gaps-in-localstack) | + +## Known deviations + +The code does not follow every rule yet. Do not copy these patterns; remove an entry when the code is fixed. + +| Where | Deviation | Target | +|-------|-----------|--------| +| `src/lib/localstack/container-spec.logic.ts`, `src/lib/docker/docker.client.ts` | Names its container `localstack-main` (the legacy CLI's name); lstk uses `localstack-aws`. | Match lstk's container spec (rule 1). | +| `src/tools/localstack-aws-client.ts` | Runs `awslocal` inside the container via `docker exec`. | `lstk aws` first, `awslocal` fallback (rule 2). | +| `src/tools/localstack-deployer.ts` | Runs `tflocal`, `cdklocal`, `samlocal` only. | `lstk terraform` / `cdk` / `sam` first (rule 2). | +| `src/tools/localstack-azure-client.ts` | Configures the host `az` itself. | `lstk az` first, keeping the existing guards (rule 2). | +| `Dockerfile` | Bundles neither lstk nor the AWS CLI. | Bundle lstk and the AWS CLI (rule 2). | +| `src/tools/localstack-iam-policy-analyzer.ts` | Deprecated. Finds IAM denials by parsing container logs. | Remove it once the deprecation notice period ends; `localstack-app-inspector` covers its jobs (rule 3). | +| `src/lib/iam/iam-policy.logic.ts` | Synthesizes IAM policies from denials, for the deprecated analyzer. | Remove with the analyzer; `src/lib/iam/iam-explain.logic.ts` presents LocalStack's suggestions and summary (rule 5). | +| `src/lib/logs/log-retriever.ts` | Regex-classifies log lines (service, operation, status, IAM). | Raw and filtered lines only (rule 3). | +| `src/lib/localstack/extensions.logic.ts` | Runs a private Python module inside the container (the same one the legacy CLI runs). | The extensions HTTP API: `GET /_localstack/extensions/list`, `POST …/install`, `POST …/uninstall` (newline-delimited JSON progress events; LocalStack restarts itself afterwards). See [Gaps in LocalStack](#gaps-in-localstack). | +| `README.md` | Advertises log parsing for IAM policies and lists `tflocal`/`cdklocal`/`samlocal` as prerequisites. | Update with the code. | diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..43c994c --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1 @@ +@AGENTS.md diff --git a/REVIEW.md b/REVIEW.md new file mode 100644 index 0000000..844f701 --- /dev/null +++ b/REVIEW.md @@ -0,0 +1,107 @@ +# Review Rules: LocalStack MCP Server + +Rules for reviewing a change to this repository. The design rules they enforce are in [AGENTS.md](AGENTS.md). + +Report each finding as BLOCKER, SHOULD-FIX or NIT, with file and line and a concrete failure scenario (inputs and +state, then the wrong result). Call something a BLOCKER only after confirming it in the code. + +A change may touch code listed under "Known deviations" in AGENTS.md without fixing it. It must not add new code of +the same kind, and it must not extend a deviation to another tool. + +## 1. lstk, not the legacy CLI + +- **BLOCKER:** the change calls, documents or recommends the legacy `localstack` CLI, in code, error hints, README or + tool descriptions. +- **BLOCKER:** the container the server starts differs from lstk's in name, labels, ports, volumes or auth env, so + `lstk stop` or `lstk status` cannot find or manage it, or the server cannot manage an lstk-started emulator. +- **SHOULD-FIX:** a change to the container spec has no test that compares it with lstk's defaults. +- **SHOULD-FIX:** the server calls lstk without passing its own endpoint as `LSTK_ENDPOINT_URL`, so a non-default + `LOCALSTACK_HOSTNAME` or `LOCALSTACK_PORT` sends the command to a different emulator. + +## 2. Cloud and IaC commands + +- **BLOCKER:** a new code path runs `awslocal`, `tflocal`, `cdklocal`, `samlocal` or a hand-configured `az` without + trying the lstk command first. +- **BLOCKER:** the lstk/fallback choice is made per tool instead of in the shared helper. +- **SHOULD-FIX:** tests cover only one path. Both the lstk path and the fallback need a test, including "neither is + installed". +- **SHOULD-FIX:** the response does not say which command ran, or the missing-runner hint recommends anything other + than lstk. +- **BLOCKER:** moving a tool to lstk drops a safety check: the AWS CLI sanitizer, or the Azure profile isolation, + egress guard or command policy. + +## 3. APIs over logs + +- **BLOCKER:** new code derives structured facts from log text: services, operations, status codes, errors per + request, IAM denials. +- **BLOCKER:** a tool falls back to log parsing when an API is unavailable. It should explain how to enable the + feature instead. +- **SHOULD-FIX:** the change reads data from logs that a documented endpoint already returns. + +## 4. Public interfaces only + +This repository is public, so these are BLOCKERs: + +- a reference to private LocalStack code: Python module paths (`localstack.pro.*`), private file paths or class names, + in code, comments, tests or docs +- an endpoint that is neither in the emulator's OpenAPI spec (`/_localstack/swagger`) nor in the public docs, unless + it is listed under "Gaps in LocalStack" in AGENTS.md +- a script or command run inside the LocalStack container to reach internals +- an undocumented configuration variable + +## 5. Compute in LocalStack, render here + +Flag new code that decides what is true instead of presenting what LocalStack returned. Red flags: + +- parsing ARNs (`split(':')`, regexes) to learn an account, service, resource type or principal kind +- reading policy documents, or matching actions and resources against statements +- building or merging IAM policies +- choosing between remedies (which policy to edit, attach versus inline) + +The fix is a field in the LocalStack payload. A tool may report that an older emulator lacks the field, but must +not compute it as a fallback. + +## 6. IAM policies + +See "IAM policies" in AGENTS.md. + +- **BLOCKER:** the tool changes LocalStack's suggestion for a denied request: drops, merges or reorders its changes, + or reports a different `outcome`. What the agent applies must be what LocalStack verified. +- **BLOCKER:** the tool computes a diff between suggested and deployed policies itself, instead of presenting one + that LocalStack returned. +- **SHOULD-FIX:** a summary policy is presented as verified or complete, instead of as a suggestion from the + requests seen so far; or "unused permissions" is not qualified as unused in this run. +- **SHOULD-FIX:** a summary policy is reshaped, for example by splitting a role's trust statement from its + permissions. +- **SHOULD-FIX:** a missing license feature (a 404 from the policy summary, or App Inspector unavailable) shows as an + error instead of an explanation of the feature and how to get it. + +## 7. Tool contract + +- **BLOCKER:** user input reaches a shell (`shell: true`, string commands, unescaped interpolation), or a CLI string + bypasses its sanitizer. +- **BLOCKER:** a secret (auth token, AWS or Azure credentials, replicator source keys) can reach a tool response, an + error message or analytics. +- **SHOULD-FIX:** the handler skips `withToolAnalytics`, preflights or `ResponseBuilder`, or a new tool argument is + missing from (or wrongly added to) the analytics allowlist in `src/core/analytics.ts`. +- **SHOULD-FIX:** decision or parsing logic lives in the tool file instead of a tested `*.logic.ts` module. +- **SHOULD-FIX:** a tool's `schema` descriptions or `metadata.description` no longer match what the tool does. Agents + choose tools from these strings. +- **SHOULD-FIX:** a breaking change to a tool's arguments or actions without a note in the PR. MCP clients and saved + prompts depend on them. + +## 8. Run modes + +The server runs through `npx` on the host and as a Docker image that reaches the host's Docker daemon through the +mounted socket. + +- **SHOULD-FIX:** the change works in one mode only. Check paths (host paths versus mounted paths), hostnames + (`localhost` versus `host.docker.internal`) and binaries the image must bundle. +- **SHOULD-FIX:** a new host binary is needed but neither README prerequisites nor the `Dockerfile` are updated. + +## 9. Tests and docs + +- **SHOULD-FIX:** new `*.logic.ts` code without unit tests, or a new tool or action without a case in + `tests/mcp/direct.spec.mjs`. +- **SHOULD-FIX:** `README.md` tool or prerequisite tables are out of date. +- **NIT:** a fixed known deviation still listed in AGENTS.md, or a new one not listed. From 3523876676310d915505b8b9acbe6b2727ccbcba Mon Sep 17 00:00:00 2001 From: Joel Scheuner Date: Thu, 19 Nov 2026 16:59:02 +0100 Subject: [PATCH 02/13] docs: rules for security, dependencies, image size and testing; align with main MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds REVIEW.md §0 (secret leaks through CI, external and Dependabot PRs), dependency pinning for Dependabot, Docker image size, and testing rules for MCP tools. Known deviations now cover existing code on main only; the IAM actions are documented as design and the agentic eval as a follow-up. Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 315 ++++++++++++++++++++++++++++++++++++++++++------------ REVIEW.md | 143 +++++++++++++++++++++---- 2 files changed, 371 insertions(+), 87 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 5b5c97a..4687de7 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -13,9 +13,10 @@ Read [REVIEW.md](REVIEW.md) before opening a PR. It turns the rules below into r yarn install yarn build # xmcp build + esbuild CLI bundle → dist/ yarn dev # xmcp dev server -yarn test # Jest unit tests (*.test.ts next to the code) -yarn test:mcp:direct # Playwright: calls the built server's tools directly (needs Docker + LOCALSTACK_AUTH_TOKEN) -yarn test:mcp:evals # Playwright: LLM evals against the tools (needs a Gemini key) +yarn test # Jest unit tests: *.test.ts next to the code, handler tests in src/tools-tests/ +yarn test:mcp:direct # Playwright: calls the built server's tools directly, plus the Azure offline suite (needs Docker + LOCALSTACK_AUTH_TOKEN) +yarn test:mcp:tools # Playwright: each tool against a real LocalStack, no model (needs Docker + LOCALSTACK_AUTH_TOKEN) +yarn test:mcp:evals # Playwright: LLM evals through OpenRouter (needs OPENROUTER_API_KEY, LOCALSTACK_AUTH_TOKEN, and terraform, lstk and snow on PATH; EVAL_MODEL swaps the model) yarn format # Prettier ``` @@ -25,11 +26,12 @@ yarn format # Prettier |------|----------| | `src/tools/` | One file per MCP tool. xmcp discovers them: each exports `schema` (zod), `metadata`, and a default handler. | | `src/prompts/` | MCP prompts (workflow templates such as `infrastructure-tester`). | -| `src/core/` | Shared plumbing: `preflight.ts`, `response-builder.ts`, `analytics.ts`, `command-runner.ts`, `config.ts`, `http-client.ts`. | +| `src/core/` | Shared plumbing: `preflight.ts`, `response-builder.ts`, `analytics.ts`, `command-runner.ts`, `localstack-env.ts`, `config.ts`, `http-client.ts`. | | `src/lib//` | Tool logic. Pure, testable logic lives in `*.logic.ts` with a matching `*.logic.test.ts`. | | `src/lib/docker/` | Docker Engine API client (find, start, stop, exec, logs). | | `src/lib/localstack/` | LocalStack HTTP clients and the container spec. | | `src/cli/` | The `init` / `remove` setup wizard and the stdio entry point. | +| `src/tools-tests/` | Handler tests, one file per tool (2 of 15 so far). They live outside `src/tools/` because xmcp registers every file there as a tool. | | `tests/mcp/` | Playwright tests against the built server. | | `tests/docker/` | Validation of the Docker image. | | `docs/` | Topic docs (`AZURE.md`, `DOCKER.md`). | @@ -39,13 +41,114 @@ yarn format # Prettier Follow the shape of an existing tool, for example `src/tools/localstack-aws-client.ts`: 1. Wrap the handler in `withToolAnalytics(, args, …)`. Add the tool to the field allowlist in - `src/core/analytics.ts`, and list only fields that can never hold secrets or user data. + `src/core/analytics.ts`, and list only fields that can never hold secrets or user data. Some existing entries + break this (see [Known deviations](#known-deviations)); do not copy them. 2. Run preflights first (`runPreflights([requireAuthToken(), requireLocalStackRunning(), …])`). 3. Return every result through `ResponseBuilder` (`markdown`, `error`, …). 4. Put parsing, validation and decision logic in `src/lib//*.logic.ts` and unit-test it there. 5. Never pass user input to a shell. Spawn binaries with argument arrays (`runCommand`), and sanitize CLI strings like `src/lib/aws/aws-cli-sanitizer.ts` does. -6. Update the tool and prerequisite tables in `README.md`. +6. Test it as described in [Testing](#testing). +7. Update the tool and prerequisite tables in `README.md`. + +## Testing + +Test every tool at these levels; many do not have all of them yet (see [Known deviations](#known-deviations)). Write +the failing test first when fixing a bug. + +| Level | Where | Runs in CI | What it proves | +|-------|-------|------------|----------------| +| Logic | `src/lib//*.logic.test.ts` | always | Parsing, validation, rendering and sanitizing, as pure functions. | +| Handler | `src/tools-tests/.test.ts` | always | The tool's actions end to end inside the process, with Docker, LocalStack and spawned binaries mocked at the module boundary. No existing handler test follows every rule below yet; do not copy their mocks of internal modules. | +| Tools | `tests/mcp/tools.spec.mjs` | every PR, with the auth token | Each tool over MCP stdio against a real emulator, without a model. Each case reads the state the tool left behind back through another call. | +| Image | `tests/mcp/direct.spec.mjs`, `tests/docker/validate-image.mjs` | every PR (`docker.yml` only when the image's inputs change; its integration test on amd64 only), with the auth token | The built server and the Docker image: tool list, prompts, setup wizard, and real calls through the image. | + +Rules for tool tests: + +- **One handler test per action**, covering its success path and each error path. Errors come back through + `ResponseBuilder.error` with `isError: true`; assert that, not just the text. +- **Test the gates.** Each preflight the tool runs gets a case: no auth token, LocalStack not running, lstk missing, + feature not licensed. Keep `runPreflights` real and stub only the checks that reach Docker or the license API. + A missing license or feature must read as an explanation (see "IAM policies"), and the test should say so. +- **Assert on what leaves the process**: the response (text, `isError`), the argument array passed to the spawned + binary, and the HTTP method, path, query and body sent to LocalStack. Do not assert on internal calls. +- **Schema.** Invalid arguments are rejected by the zod schema; test a missing required field and a wrong enum + value for each action that takes them. +- **Adversarial input.** For a tool that renders LocalStack or workload strings into markdown, add a case whose + payload contains a code fence, a table pipe and an instruction-like line, and assert it cannot escape. For a tool + that builds a command line, add cases that try shell chaining, option injection and path traversal. +- **No secrets in output.** Run the handler with a recognizable fake token and credentials, and assert they appear + in neither the response, the error, nor the analytics payload. +- **Real-emulator coverage.** A new tool goes into `EXPECTED_TOOLS` in `tests/mcp/direct.spec.mjs`. A new action + gets a case in `tests/mcp/tools.spec.mjs` that checks its effect through another call, not just its success + message, so a tool that skips its API call but reports success fails. Add a case to + `tests/docker/validate-image.mjs` when the action depends on the image (bundled binaries, mounted paths). +- **Local by default.** No real AWS or Azure resources. A test that creates something on the LocalStack platform + (a Cloud Pod) deletes it in a `finally` and uses a name unique to the run, as `tools.spec.mjs` does, or a fixed + name only when a workflow `concurrency` group keeps runs from overlapping, as `mcp-evals.yml` does. Costly ones + (ephemeral instances) stay opt-in (`HARNESS_RUN_EPHEMERAL`). +- **Missing secrets fail loudly.** A job that needs a secret fails with a message naming it when it is empty, as + `ci.yml` does for fork PRs, so a fork PR never looks tested. Only an opt-in scenario's own credentials (the + replicator's source AWS keys) or an outside service being down (the docs search) may skip a case, and the + output says why. +- **Evals are not PR gates.** `test:mcp:evals` depends on a model. `mcp-evals.yml` runs it after merges to `main`, + weekly and on demand. Run them yourself when a tool's description, schema or + response wording changes, and report the result in the PR. + +## Dependencies + +Every dependency version lives in exactly one file that Dependabot updates (`.github/dependabot.yml`), so CVE fixes +arrive as PRs instead of waiting for someone to notice: + +| What | Pinned in | +|------|-----------| +| Server dependencies | `package.json` + `yarn.lock` | +| CLIs in the image (aws-cdk, lstk) | `docker/tools/npm/package-lock.json` | +| Python CLIs in the image (SAM, Snowflake) | `docker/tools//uv.lock` | +| Base images, Terraform, uv | `FROM` lines in `Dockerfile` | +| GitHub Actions | `uses:` in `.github/workflows/` | + +- **No version outside these files.** No `npm install -g `, `pip install `, `curl … | sh`, `latest` tags + (except the LocalStack emulator images, which track the newest release on purpose), or `npx ` for a package + that is not in `package.json`, in the Dockerfile, workflows or scripts. A new tool gets a lockfile in one of the + places above. +- **New ecosystem or directory, new Dependabot entry.** Keep the 7-day cooldown and the minor/patch grouping. +- **CI uses the image's versions.** Jobs that run tools on the host install them through + `.github/actions/host-tools`, from the same files the image uses. `src/core/dependency-pins.test.ts` checks that + the jobs running the tool tests or the evals use it, and that `ci.yml`, `docker.yml` and `mcp-evals.yml` run on + the same pinned Ubuntu. +- **Hand-managed pins** (Node's major line, `zod` up to xmcp's copy, `@gleanwork/mcp-server-tester` while + `patches/` patches it) are listed as `ignore` entries in `dependabot.yml` with the reason, and + `src/core/dependency-pins.test.ts` keeps the ones it can in sync. +- **Fix advisories by upgrading.** `audit-ci --high --skip-dev` gates CI. If no fixed version exists, an allowlist + entry names the advisory, why it does not apply, and when to revisit it. +- **Actions pinned to a commit SHA**, with the version in a comment (`uses: actions/checkout@ # v7`). + Dependabot updates both. + +## Docker image size + +Every user of the image downloads it, often in CI on every run, so its size is part of the startup time. Keep it +small, and treat growth as a decision for the user, not a side effect. + +- **Ask before a larger change.** Before adding anything that grows the image by more than about 50 MB + (uncompressed, per platform) or brings in a new runtime, toolchain or base image (a JVM, a second Python, a cloud + CLI, a browser), stop and ask the user. Name what it is for, which tools and actions need it, its size on amd64 + and arm64, and the alternatives: leaving it to host mode with a clear error in the image, a separate image + variant, or a lighter tool. Record the approval in the PR. +- **Measure every image change.** A PR that touches `Dockerfile`, `docker/` or `.dockerignore`, or adds a runtime + dependency, reports the image size before and after for both platforms (`docker image inspect -f + '{{.Size}}'`; `docker history` for the layer that grew). +- **Keep the runtime stage lean.** + - Start from a `-slim` base and build in earlier stages; copy only the artifacts into the runtime stage, never + compilers, build caches or dev dependencies. + - Install system packages with `--no-install-recommends`, and remove package lists and caches in the same `RUN` + that creates them; a later `RUN rm` does not shrink the image. + - Install production dependencies only (`--omit=dev`), clean the npm and yarn caches, and prune test directories + from Python environments, as the existing stages do. + - Use `RUN --mount` (bind, cache) for files only the build needs. + - Keep `.dockerignore` excluding everything the image does not need. +- **Do not trade correctness for size.** Removing something a tool calls breaks it only in the image. Run + `tests/docker/validate-image.mjs` after any change that removes files. ## Design rules @@ -59,30 +162,39 @@ is the LocalStack CLI. Do not call, document or recommend the legacy `localstack The MCP server keeps its own lifecycle code: it starts and stops the emulator through the Docker API, so it works without lstk. That container must be interchangeable with one lstk starts: -- **Same container.** Container name, image, ports, labels, volume layout and auth env match what lstk creates. - The reference is lstk's `internal/config/containers.go` and `internal/container/`. When lstk changes them, follow. -- **Either tool can take over.** `lstk stop` / `lstk status` work on an emulator the MCP server started, and the MCP - server's `stop` / `status` work on one lstk started. Find the container the way lstk marks it (name and labels), - not by a name only this server uses. -- **Same endpoint.** When the MCP server calls lstk, it passes its own target (`LOCALSTACK_HOSTNAME` / - `LOCALSTACK_PORT`) as `LSTK_ENDPOINT_URL`, so both talk to the same emulator. - -### 2. Prefer lstk's cloud and IaC commands over dedicated wrappers +- **Converge on lstk's container.** The reference is lstk's `internal/config/containers.go`, `internal/container/` + and `internal/runtime/docker.go`. The server's container still differs from it in several ways (name, labels, + volume directory, ports, environment; see [Known deviations](#known-deviations), which lists the differences known + so far, not all of them). A change to the container spec must not add a difference: compare it with lstk's and + move toward it. When lstk changes, follow. +- **Either tool can take over.** `lstk stop` / `status` / `logs` / `restart` work on an emulator the MCP server + started, and the MCP server's `stop` / `status` work on one lstk started. Today this holds only through fallbacks: + the server detects `localstack-aws` by name, and lstk, finding no container under its own name, matches a running + LocalStack image that serves the gateway on container port 4566. That interoperability is a known gap. Do not + widen the difference: keep both names detected, keep LocalStack's own images, and keep the gateway on container + port 4566. +- **Same endpoint, for the commands that take one.** `lstk aws`, `az`, `terraform`, `cdk`, `sam`, `status`, + `inspect`, `reset` and `snapshot` (save, load, remove, and list of an `s3://` remote) accept an endpoint. Pass + the server's target (`LOCALSTACK_HOSTNAME` / `LOCALSTACK_PORT`) to them so both talk to the same emulator. Set `LSTK_ENDPOINT_URL`: lstk prefers it over `AWS_ENDPOINT_URL`, so a value already in the + user's environment would otherwise win. lstk refuses an endpoint for the local lifecycle commands (`start`, + `stop`, `restart`, `logs`, `volume`), so setting one there makes the command fail. Which commands + get the endpoint is not decided yet: ask the user before wiring lstk calls (see [Open questions](#open-questions)). + +### 2. Use lstk's cloud and IaC commands lstk wraps the cloud CLIs and IaC tools ([docs](https://docs.localstack.cloud/aws/developer-tools/running-localstack/lstk/cloud-and-iac-commands/)): -`lstk aws`, `lstk az`, `lstk terraform`, `lstk cdk`, `lstk sam`. Use them instead of `awslocal`, `tflocal`, -`cdklocal`, `samlocal`, or a hand-configured `az`. - -- **lstk first, legacy fallback.** When `lstk` is on `PATH`, use it. Otherwise fall back to the dedicated wrapper, - so a setup without lstk keeps working. Resolve this in one shared helper, not per tool. -- **Say which path ran.** The tool response names the command that ran (`lstk aws …` or `awslocal …`), so a user - can reproduce it. -- **Hint toward lstk.** When the fallback runs, or no runner is found, the message recommends installing lstk. It - never recommends the legacy CLI. +`lstk aws`, `lstk az`, `lstk terraform`, `lstk cdk`, `lstk sam`. Use them, not `awslocal`, `tflocal`, `cdklocal`, +`samlocal`, or a hand-configured `az`. + +- **lstk is required.** There is no fallback to the dedicated wrappers. When lstk is missing, the tool says so and + shows how to install it (as the deployer does with `LSTK_INSTALL_MESSAGE` in `src/lib/deployment/`). It never + recommends the legacy CLI. +- **Say what ran.** The tool response names the command that ran (`lstk aws …`), so a user can reproduce it. - **Keep the guards.** Moving a tool to lstk does not remove its safety checks: the AWS CLI sanitizer, and the Azure tool's profile isolation, egress guard and command policy (`src/lib/azure/`). -- **Docker image.** The image bundles lstk and the AWS CLI, so the lstk path also runs there. +- **Docker image.** The image bundles lstk and everything its commands call (the AWS CLI, Terraform, CDK, SAM, and + the Azure CLI once the Azure tool moves). - **Snowflake is exempt.** The Snowflake tool uses the `snow` CLI until lstk wraps it. ### 3. Prefer LocalStack APIs over parsing logs @@ -108,8 +220,19 @@ This repository is public. Do not reference code from private LocalStack reposit - no undocumented endpoints, and no scripts executed inside the LocalStack container to reach internals - no undocumented configuration variables -Allowed: documented endpoints (in the OpenAPI spec or the docs), documented configuration variables (for example -`EXTENSION_AUTO_INSTALL`), the public LocalStack Platform API, and public repositories such as lstk. +Allowed: documented endpoints, documented configuration variables (for example `EXTENSION_AUTO_INSTALL`), the +public LocalStack Platform API, and public repositories such as lstk. An endpoint is documented when it is in the +public docs or in one of the OpenAPI specs a running emulator serves: + +- `/_localstack/swagger` for the core and platform endpoints (`/_localstack/*`, `/_aws/*`) +- `/_localstack/appinspector/v1/openapi.yaml` (browsable at `/_localstack/appinspector/v1/swagger`) for App + Inspector + +Ask the user before adopting an API this server does not use yet, even a documented one, and name the endpoint and +what the tool would do with it. The App Inspector routes the server uses today: the traces, spans and events +routes under `/v1/traces` (list, get and delete, down to `/v1/traces/{t}/spans/{s}/events/{e}` and `/events/iam`), +and `/_localstack/appinspector/status` (not in the spec yet, see [Gaps in LocalStack](#gaps-in-localstack)). The +IAM design below adds `/v1/events/{event_id}/policies/{key}` and the IAM policy summary. Exception: an endpoint listed under [Gaps in LocalStack](#gaps-in-localstack) may be used before it is documented. Each such endpoint is one LocalStack's own clients (the web app, App Inspector) already depend on. Describe its @@ -130,43 +253,48 @@ If two clients could compute the same answer differently, the answer belongs in or user gets, and LocalStack cannot offer it yet, it may be done as a documented exception, but only with the user's explicit approval. This is not a default way out of the rule: first propose the LocalStack change, then ask the user, naming the computation, what it improves, and what it would take in LocalStack. Never assume - approval. An approved exception gets a row in [Approved exceptions](#approved-exceptions) and a - [Gaps in LocalStack](#gaps-in-localstack) row for the API that would replace it; delete both once that API - ships. Exceptions never cover decisions about IAM (which policy denied, what fixes it), and never read logs + approval. An approved exception gets a row in [Known deviations](#known-deviations), saying who approved it and + when, and a [Gaps in LocalStack](#gaps-in-localstack) row for the API that would replace it; delete both once + that API ships. Exceptions never cover decisions about IAM (which policy denied, what fixes it), and never read logs (rule 3). ## Features ### IAM policies (`localstack-app-inspector`) -`localstack-app-inspector` owns IAM. `localstack-iam-policy-analyzer` is deprecated: its behaviour is unchanged -until it is removed, but its description and every response name the app-inspector action that replaces it. +Today `localstack-iam-policy-analyzer` handles IAM by parsing logs and synthesizing policies (known deviations), +with its `set-mode` and `get-status` actions for the enforcement mode. IAM moves to `localstack-app-inspector`. None +of the actions below exist yet; this is the design they are built to. Once its actions ship, the analyzer is +deprecated: its behaviour stays unchanged until it is removed, but its description and every response name the +app-inspector action that replaces it. The IAM actions have three jobs. Each reads a LocalStack API; without the feature behind it, the response explains what is missing instead of returning an empty answer. 1. **Why was a request denied, and what fixes it.** `list-iam-denials`, `explain-iam`, `get-iam-policy`. Source: App Inspector IAM events (`/_localstack/appinspector/v1/traces/{t}/spans/{s}/events/iam`, `iam.policy.denied` for - denials, also in soft mode with `enforcement: allow_with_warning`). The event's `payload` attribute is a JSON - string holding the decisions (`explicit_allows`, `explicit_denies`, `implicit_denies`, each with `caused_by` + denials, also when they were not enforced, with `enforcement: allow_with_warning`). The event's `payload` + attribute is a JSON string holding the decisions (`explicit_allows`, `explicit_denies`, `implicit_denies`, each with `caused_by` policy keys and statement positions), the `policies` map, and for a denial LocalStack's `suggestion` (version 1: - `outcome`, `entries`, `hints`, `near_misses`). `explain-iam` renders them in the section order of the App + `outcome`, `entries`, `hints`, `near_misses`). `explain-iam` is to render them in the section order of the App Inspector UI's "Copy agent prompt", then the payload as sent. Do not recompute, merge or reorder the changes. A policy stored truncated (`statement_count` without `document`) is fetched on demand with `get-iam-policy` (`/v1/events/{event_id}/policies/{key}`), never automatically. - Without the `localstack.appinspector/view_iam_policy_errors` entitlement, LocalStack hides IAM events and - returns an empty list. Only spans say so (`iam_errors_suppressed`), so an empty IAM result reads one span of - the scope and reports suppression, or else the IAM mode. `/_localstack/licenseinfo` cannot tell: it lists - plugins, not entitlements. + returns an empty list. Spans say so (`iam_errors_suppressed`) and the policy route answers 403 + `not_entitled`, but the event lists don't, so an empty IAM result reads one span of the scope and reports + suppression, or else the IAM mode. `/_localstack/licenseinfo` cannot tell: it lists plugins, not + entitlements. - A denial without `suggestion` comes from an older LocalStack: say so and suggest upgrading. A suggestion `version` other than 1 is shown as JSON only. 2. **Which policies does the workload need.** `get-iam-policy-summary`, `reset-iam-policy-summary`. Source: the IAM - policy summary (`GET`/`DELETE /_aws/iam/policies/summary`, feature `localstack.platform.plugin/iam-stream`; a 404 - means the feature is not licensed or not enabled). It is global to the emulator: there is no per-trace summary + policy summary (`GET`/`DELETE /_aws/iam/policies/summary`; a 404 means the feature is not licensed or not + enabled). It is global to the emulator: there is no per-trace summary (see [Gaps in LocalStack](#gaps-in-localstack)). Each entry is one policy for one ARN: `resource`, `policy_type` (`identity` or `resource`) and `policy_document`. Present it the way App Inspector's Policies view does ([localstack-appinspector-ui#341](https://github.com/localstack/localstack-appinspector-ui/pull/341)): - - one entry per ARN, with its name (the ARN's last part), its type, its services and its number of actions + - one entry per ARN, with its type LocalStack does not send a display name, services or action count yet (see + [Gaps in LocalStack](#gaps-in-localstack)); do not derive them from the ARN or the policy - the policy JSON exactly as sent: a role's trust statement and its permissions stay in one policy - always labelled as a suggestion, derived from the requests seen since the last reset, not as a verified policy 3. **How do the deployed policies differ from what the workload needs.** For each ARN in the summary: the @@ -174,10 +302,17 @@ what is missing instead of returning an empty answer. LocalStack computes it (rule 5). LocalStack does not offer it yet (see [Gaps in LocalStack](#gaps-in-localstack)), so the tool does not either. -The enforcement mode (`get-iam-mode`, `set-iam-mode`, through `/_aws/iam/config`) is a separate action. Denials and -the policy summary are only recorded while the mode is `ENFORCED` or `SOFT_MODE`. +The enforcement mode is to get separate actions (`get-iam-mode`, `set-iam-mode`, through `/_aws/iam/config`). Enforcement +is not needed to record denials. With App Inspector enabled and enforcement off, `/_aws/iam/config` reports +`ENGINE_ONLY` (a documented state), App Inspector records denials with `enforcement: allow_with_warning`, and the +policy summary grows, but nothing is blocked. `ENFORCED` only makes denials fail the +request; `SOFT_MODE` keeps them passing. Never tell a user to switch on enforcement just to see denials, since +`ENFORCED` can break a workload that worked. -The agentic evaluation in `evals/agentic/` checks that a coding agent fixes common IAM failures with these actions. +Render every payload string escaped (REVIEW §7): an ARN, a resource name or an error message must not be able to +close a code fence, break a table, or read as an instruction. + +An agentic evaluation of these actions is a follow-up (see [Follow-ups](#follow-ups)). ## Gaps in LocalStack @@ -187,34 +322,80 @@ row once it ships. | Need | Status | Used by | |------|--------|---------| | Document the extensions API (`GET /_localstack/extensions/list`, `POST …/install`, `POST …/uninstall`) in the OpenAPI spec and the docs | The LocalStack web app's Extension Manager depends on it; neither the spec nor the docs name it. lstk has no extensions commands. | `localstack-extensions` (allowed under the rule 4 exception) | -| Document the IAM policy summary and stream (`GET`/`DELETE /_aws/iam/policies/summary`, `GET /_aws/iam/policies/stream`) | App Inspector's Policies view and the web app's IAM Policy Stream depend on them; the spec documents only `/_aws/iam/config` and `/_aws/iam/check-actions-allowed`. | `localstack-app-inspector` IAM job 2 (allowed under the rule 4 exception) | -| Filter the IAM policy summary by trace (or time range), so a run's policies don't mix with earlier requests | Not implemented: the summary is global and only resettable. | `get-iam-policy-summary` | -| Say on the App Inspector events page when IAM events are hidden by the license (as spans do with `iam_errors_suppressed`) | Not implemented: the events routes return an empty list. The tool probes one span instead. | `list-iam-denials`, `explain-iam` | +| Document the IAM evaluation event payload (the `payload` attribute of App Inspector's IAM events: the decisions, the `policies` map and the `suggestion`) in the App Inspector spec | The spec types the attribute as a string; its fields are not documented anywhere public. The App Inspector UI depends on them. Describe them in this repository only as types and tests. | `explain-iam`, planned (allowed under the rule 4 exception) | +| Document the IAM policy summary and stream (`GET`/`DELETE /_aws/iam/policies/summary`, `GET /_aws/iam/policies/stream`) | App Inspector's Policies view and the web app's IAM Policy Stream depend on them; the spec documents only `/_aws/iam/config` and `/_aws/iam/check-actions-allowed`. | `localstack-app-inspector` IAM job 2, planned (allowed under the rule 4 exception) | +| Filter the IAM policy summary by trace (or time range), so a run's policies don't mix with earlier requests | Not implemented: the summary is global and only resettable. | `get-iam-policy-summary` (planned) | +| Say on the App Inspector events page when IAM events are hidden by the license (as spans do with `iam_errors_suppressed`) | Not implemented: the events routes return an empty list, so a tool has to probe one span. | `list-iam-denials`, `explain-iam` (planned) | | Compare suggested policies with deployed ones: permissions to add and permissions granted but unused, per ARN | Not implemented. Listed as a follow-up in localstack-appinspector-ui#341. | IAM job 3 | -| A failed-requests summary in App Inspector: failed spans grouped by service, operation, HTTP status and AWS error code, with counts, first and last time, and an example trace and span | Not implemented. | `lstk mcp`'s `localstack-app-inspector` `failed-requests` (an [approved exception](#approved-exceptions) until then) | - -## Approved exceptions +| Document App Inspector's status route (`GET`/`PUT /_localstack/appinspector/status`) in its OpenAPI spec | Not in `/_localstack/appinspector/v1/openapi.yaml`. lstk's `inspect on` / `off` uses it too. | `get-status`, `set-status` (allowed under the rule 4 exception) | +| Presentation fields for IAM: a display name, the services and the action count per policy summary entry; the principal kind (ARN, service, account) on suggestion hints; the statements a decision names, resolved | Not implemented. Until it ships, show the ARN, the type and the policy JSON, name statement positions without resolving them, and word hints for any kind of caller. | `explain-iam`, `get-iam-policy-summary` (planned) | +| Document `GET /_localstack/licenseinfo`, or name the documented way to tell whether a licensed feature is available | In none of the emulator's specs and not in the docs. Whether a LocalStack client depends on it is unverified: lstk and the App Inspector UI don't. | `requireProFeature` in every Pro tool (allowed under the rule 4 exception until LocalStack answers) | +| A failed-requests summary in App Inspector: failed spans grouped by service, operation, HTTP status and AWS error code, with counts, first and last time, and an example trace and span | Not implemented. | `lstk mcp`'s `localstack-app-inspector` `failed-requests`, planned (an approved exception, see [Follow-ups](#follow-ups)) | -Client-side computations allowed under rule 5's exception, each approved by the user. Remove a row when the -LocalStack API that replaces it ships. +## Known deviations -| Where | Computation | Why | Approved | Replaced by | -|-------|-------------|-----|----------|-------------| -| `lstk mcp`, `localstack-app-inspector` action `failed-requests` (lstk `internal/mcp/appinspector_render.go`) | Groups App Inspector's failed spans (`status_code=2`) by service, operation, HTTP status and AWS error code, with counts, first and last time, and the latest example. Reads at most 5,000 spans. | Replaces the log-based error summary of `localstack-logs-analysis` with one from App Inspector's data; without it an agent pages through raw spans or greps logs. | 2026-11-17, by the user, during the `lstk mcp` prototype | The failed-requests summary in [Gaps in LocalStack](#gaps-in-localstack) | +Existing code that does not follow the rules yet. This list is for code that is already on `main`; it is not a way +to let new code through. -## Known deviations +- **New code follows the rules.** A change that needs to break one raises it with the user before it is written, + and its PR records the decision. Only then does it get a row here. +- **A change that touches a listed deviation raises it.** Say in the PR whether the change fixes the deviation, + leaves it as it is, or why it cannot fix it yet. Do not copy the pattern and do not extend it to another tool. +- **Remove a row when the code is fixed.** -The code does not follow every rule yet. Do not copy these patterns; remove an entry when the code is fixed. +Approved exceptions to rule 5 are listed here too, with who approved them (by role or GitHub handle) and when; they +stay until the LocalStack API that replaces them ships. | Where | Deviation | Target | |-------|-----------|--------| -| `src/lib/localstack/container-spec.logic.ts`, `src/lib/docker/docker.client.ts` | Names its container `localstack-main` (the legacy CLI's name); lstk uses `localstack-aws`. | Match lstk's container spec (rule 1). | -| `src/tools/localstack-aws-client.ts` | Runs `awslocal` inside the container via `docker exec`. | `lstk aws` first, `awslocal` fallback (rule 2). | -| `src/tools/localstack-deployer.ts` | Runs `tflocal`, `cdklocal`, `samlocal` only. | `lstk terraform` / `cdk` / `sam` first (rule 2). | -| `src/tools/localstack-azure-client.ts` | Configures the host `az` itself. | `lstk az` first, keeping the existing guards (rule 2). | -| `Dockerfile` | Bundles neither lstk nor the AWS CLI. | Bundle lstk and the AWS CLI (rule 2). | -| `src/tools/localstack-iam-policy-analyzer.ts` | Deprecated. Finds IAM denials by parsing container logs. | Remove it once the deprecation notice period ends; `localstack-app-inspector` covers its jobs (rule 3). | -| `src/lib/iam/iam-policy.logic.ts` | Synthesizes IAM policies from denials, for the deprecated analyzer. | Remove with the analyzer; `src/lib/iam/iam-explain.logic.ts` presents LocalStack's suggestions and summary (rule 5). | -| `src/lib/logs/log-retriever.ts` | Regex-classifies log lines (service, operation, status, IAM). | Raw and filtered lines only (rule 3). | -| `src/lib/localstack/extensions.logic.ts` | Runs a private Python module inside the container (the same one the legacy CLI runs). | The extensions HTTP API: `GET /_localstack/extensions/list`, `POST …/install`, `POST …/uninstall` (newline-delimited JSON progress events; LocalStack restarts itself afterwards). See [Gaps in LocalStack](#gaps-in-localstack). | -| `README.md` | Advertises log parsing for IAM policies and lists `tflocal`/`cdklocal`/`samlocal` as prerequisites. | Update with the code. | +| `src/lib/localstack/container-spec.logic.ts`, `src/lib/docker/docker.client.ts` | Its header comment describes the container as replicating the legacy `localstack start` (rule 1). The container differs from lstk's; the differences known so far, not an exhaustive list: the name (`localstack-main` for every emulator; lstk uses `localstack-aws`, `localstack-snowflake`, `localstack-azure`, or `localstack--`), and the server detects only `localstack-main` and `localstack-aws`; no `cloud.localstack.lstk` label; its own volume directory (`/localstack/volume` or the named volume `localstack-mcp`; lstk: `/lstk/volume/`), so emulator state does not carry over; service ports 4510–4560 (lstk: 4510–4559); and the environment (lstk sets `LOCALSTACK_HOST`, `LOCALSTACK_CLIENT_NAME=lstk` and, for Snowflake, `SF_S3_ENDPOINT`; the server sets none of them and adds `EXTERNAL_SERVICE_PORTS_*`). Interoperability rests on the server detecting both AWS names and on lstk's fallback, which matches a known LocalStack image serving the gateway on container port 4566 (not the host port, so a non-default `LOCALSTACK_PORT` is fine). It breaks when the server runs an image lstk does not know (a mirror or custom build), a `GATEWAY_LISTEN` moves the gateway off 4566 inside the container, or the emulator is Snowflake or Azure: lstk then reports no emulator running, and `lstk start` starts a second one beside it. | Converge on lstk's container spec; decide later whether to adopt its names. Until then, keep both AWS names detected (rule 1). | +| `src/tools/localstack-aws-client.ts` | Runs `awslocal` inside the container via `docker exec`. | `lstk aws`, no fallback (rule 2). | +| `src/tools/localstack-deployer.ts`, `src/prompts/infrastructure-tester.ts` | The deployer runs `awslocal cloudformation` through `docker exec` for CloudFormation deploys and deletes, and the prompt tells the agent to use `awslocal`. | `lstk aws cloudformation` (rule 2); the prompt names lstk. | +| `src/tools/localstack-chaos-injector.ts` | Builds some `❌` responses by hand instead of through `ResponseBuilder`, so they are not flagged with `isError`. | `ResponseBuilder.error` ("Writing a tool"). | +| `src/core/localstack-env.ts` | Passes the server's target to `lstk terraform` / `cdk` / `sam` as `AWS_ENDPOINT_URL`, but keeps an `AWS_ENDPOINT_URL` already in the user's environment, and a `LSTK_ENDPOINT_URL` there overrides both: either sends the command to a different emulator. | Depends on the endpoint-scoping [open question](#open-questions). | +| `src/tools/localstack-azure-client.ts` | Configures the host `az` itself. | `lstk az`, keeping the existing guards (rule 2). | +| `Dockerfile` | Bundles lstk, Terraform, CDK and SAM, but not the AWS CLI. | Bundle the AWS CLI for `lstk aws` (rule 2). This is a larger image change: ask the user first ("Docker image size"). | +| `src/core/analytics.ts` | Sends data that can hold secrets (REVIEW §0): the full `command` of `localstack-aws-client` and `localstack-azure-client` (for example `secretsmanager create-secret --secret-string …`), and the first line of every error response as `error_message`. `looksSensitiveValue` only catches a value that is a whole key or token. The allowlist also sends user data: the deployer's `directory`, `stackName`, `templatePath` and `s3Bucket`, the docs `query`, the logs `filter`, the Cloud Pods `pod_name`, the ephemeral instance `name` and `cloudPod`, and the extension `name`. | Only fields that can never hold secrets or user data ("Writing a tool"): the command's service and operation instead of the command, an error code instead of the message. | +| `.github/workflows/*.yml` | Actions are pinned to tags (`@v7`), not commit SHAs. | SHA pins with a version comment ("Dependencies"). | +| `.github/workflows/ci.yml`, `docker.yml` | No `permissions:` block, so jobs get the repository's default token permissions. | `permissions: contents: read` at the top, widened per job only where needed (REVIEW §0). | +| `src/tools/localstack-iam-policy-analyzer.ts` | Finds IAM denials by parsing container logs (rule 3). | The IAM actions of `localstack-app-inspector` ("IAM policies"); then deprecate the analyzer and remove it after a notice period. | +| `.github/workflows/azure-smoke.yml`, `ci.yml` (`mcp-direct-tests`), `tests/mcp/azure-offline.spec.mjs` | `azure-smoke.yml` runs on `ubuntu-latest`; it and `mcp-direct-tests` use the runner image's Azure CLI, whose version no file here pins. `azure-offline` skips its cases when `az` is missing, a skip "Testing" does not allow. | The pinned Ubuntu of the other workflows, and the Azure CLI from a pinned file, as `host-tools` does for Terraform, uv and lstk. | +| `.github/workflows/ci.yml`, `mcp-evals.yml`, `azure-smoke.yml` | Set `LOCALSTACK_AUTH_TOKEN` (and in `mcp-evals.yml` `OPENROUTER_API_KEY`) at job level, so `yarn install` runs dependency install scripts with them, and so do third-party actions (REVIEW §0). | Secrets only on the steps that need them, after dependencies are installed. | +| `data/sample-terraform/` | The AWS provider (`~> 5.0`) has no lockfile and no Dependabot entry; CI downloads it with secrets in the job's environment. | A committed `.terraform.lock.hcl` and a `terraform` Dependabot entry ("Dependencies"). | +| `playwright.config.mjs` | Passes the whole environment to the server under test, so in `mcp-evals.yml` the `OPENROUTER_API_KEY` reaches the server and the `lstk` processes it starts (REVIEW §0). | Pass the server only what it needs; the model keys stay with the eval host. | +| `Dockerfile` | `# syntax=docker/dockerfile:1` floats to the newest 1.x frontend. | Pin the frontend version where Dependabot can update it. | +| `src/tools-tests/`, `src/lib/**/*.logic.test.ts` | Only `localstack-management` and `localstack-azure-client` have handler tests. `localstack-management.test.ts` stubs `withToolAnalytics` and mocks the internal `localstack.utils` module, asserting on its calls. `app-inspector.logic.test.ts`, `aws-replicator.logic.test.ts` and `state-management.logic.test.ts` test tool handlers, not a `*.logic.ts` module. | Handler tests for every tool, following "Testing"; move handler tests to `src/tools-tests/`. | +| `src/lib/iam/iam-policy.logic.ts` | Synthesizes IAM policies from denials, for the analyzer (rule 5). | Remove with the analyzer; the IAM actions present LocalStack's suggestions and summary instead. | +| `src/lib/logs/log-retriever.ts`, `src/tools/localstack-logs-analysis.ts` | Regex-classifies log lines (service, operation, status, IAM), which the logs tool's `summary`, `errors` and `requests` modes present. | Raw and filtered lines only (rule 3); App Inspector for failed requests. | +| `src/lib/localstack/extensions.logic.ts` | Runs a private Python module inside the container (the same one the legacy CLI runs), and names that private module and the container's internal paths in public code (rule 4). | The extensions HTTP API: `GET /_localstack/extensions/list`, `POST …/install`, `POST …/uninstall` (newline-delimited JSON progress events; LocalStack restarts itself afterwards). See [Gaps in LocalStack](#gaps-in-localstack). | +| `README.md` | Describes the analyzer's log search and policy generation, the aws-client's `awslocal`, and the logs tool's classification as features, and refers to the legacy "LocalStack CLI" for the volume path. | Update with the code; name lstk (rule 1). | + +## Open questions + +Decisions not settled yet. Ask the user before acting on one, and move the answer into the rules once it is made. + +- **Endpoint scoping for lstk calls (rule 1).** Rule 1 passes the server's endpoint as `LSTK_ENDPOINT_URL` only to + the lstk commands that accept one (`aws`, `az`, `terraform`, `cdk`, `sam`, `status`, `inspect`, `reset`, + `snapshot`), and never sets one for `start`, `stop`, `restart`, `logs` or `volume`, because lstk refuses those when `--endpoint-url`, `LSTK_ENDPOINT_URL` or `AWS_ENDPOINT_URL` + is set. Open: + - Is this scoping the intended contract? + - If the server delegates a lifecycle command to lstk, must it remove all three variables from the child's + environment, so an endpoint inherited from the user's shell does not make lstk refuse the command? + +## Follow-ups + +Planned work that is not on `main` yet. Ask the user before starting one. + +- **Agentic evaluation of the IAM actions.** A coding agent (Claude Code headless) fixes common IAM failures in small + projects using only this server, graded by deterministic checks, on demand and weekly. Before it lands it must + meet REVIEW §0 and "Dependencies": the agent and the scripts it runs get no secrets, the MCP config holding the + auth token is private and deleted after the run, the agent can read and edit only its project and cannot change + the scripts it may run, results are redacted and checked before upload, workflow inputs go through `env:`, and + Claude Code is pinned in a lockfile Dependabot updates. +- **`lstk mcp`'s `failed-requests` action (lstk, unreleased).** It groups App Inspector's failed spans + (`status_code=2`) by service, operation, HTTP status and AWS error code, with counts, first and last time and the + latest example, reading at most 5,000 spans. That computation is an approved exception to rule 5: approved by the + project maintainer on 2026-11-17, because it replaces the log-based error summary of `localstack-logs-analysis` + with one from App Inspector's data. It is recorded here because the code is not on lstk's `main` yet; when it + lands, record it where lstk's MCP server documents its design. It goes away once the failed-requests summary in + [Gaps in LocalStack](#gaps-in-localstack) ships. diff --git a/REVIEW.md b/REVIEW.md index 844f701..6eca214 100644 --- a/REVIEW.md +++ b/REVIEW.md @@ -5,28 +5,85 @@ Rules for reviewing a change to this repository. The design rules they enforce a Report each finding as BLOCKER, SHOULD-FIX or NIT, with file and line and a concrete failure scenario (inputs and state, then the wrong result). Call something a BLOCKER only after confirming it in the code. -A change may touch code listed under "Known deviations" in AGENTS.md without fixing it. It must not add new code of -the same kind, and it must not extend a deviation to another tool. +"Known deviations" in AGENTS.md lists existing code on `main` only. For a change under review: + +- New code that breaks a rule is a finding at the rule's severity, even if the same pattern is listed. A new row in + "Known deviations" excuses it only if the PR records that the user approved the deviation. +- A change may touch listed code without fixing it, but the PR must say so: fixed, left as is, or why it cannot be + fixed yet (SHOULD-FIX if it does not). It must not extend a deviation to another tool. + +## 0. Security leaks (always BLOCKER) + +This repository is public and takes PRs from external collaborators. Check these on every PR, before anything +else, and before approving a workflow run for a first-time contributor. Treat every changed file that CI executes +or reads as code that runs with whatever the job exposes. That includes `.github/workflows/`, `.github/actions/`, +`package.json` (scripts and dependencies), `yarn.lock`, `patches/` (applied by `yarn install`), `src/` (the server +runs with the auth token), `tests/`, `data/` (sample projects CI installs and deploys, eval datasets), +`playwright.config.mjs`, `jest.config.js`, `xmcp.config.ts`, `tsconfig*.json`, `Dockerfile`, `docker/`, and anything +else a workflow, action or install script reads. + +Each of these is a BLOCKER: + +- **Untrusted code with secrets.** A workflow triggered by `pull_request_target`, `workflow_run` or + `issue_comment` checks out or runs the PR's code: those triggers run with the repository's secrets + (`LOCALSTACK_AUTH_TOKEN`, `OPENROUTER_API_KEY`, `DOCKER_PUSH_TOKEN`, …) and a write token. Under `pull_request`, + GitHub gives fork PRs no secrets, so the jobs fail on the token check instead (`ci.yml`, `docker.yml`). Do not + work around that with a trigger that has secrets. +- **Secrets for Dependabot.** Dependabot PRs get no Actions secrets either, so token-gated jobs fail on them. Do not + add `LOCALSTACK_AUTH_TOKEN` or another secret as a Dependabot secret: it would reach the install scripts of the + packages the PR just bumped. Review the bump, then run it with secrets the same way as an external PR. +- **An external PR pushed unreviewed.** For a fork or Dependabot PR to be tested, a maintainer pushes its branch to this + repository, where every workflow runs with secrets. Review the whole diff under this section before pushing it, + and push the reviewed commit, not a branch the contributor can still change. +- **Secrets printed or stored.** A step or script echoes a secret, runs `set -x`, `env` or `printenv`, logs request + headers, or writes a secret to a file. GitHub masks secrets in logs only: an uploaded artifact (test results, + Playwright traces, eval transcripts, the server's stderr), a cache entry, or a built image layer is not masked. + Check what each `upload-artifact` path can contain. +- **Secrets handed to something that records or runs untrusted input.** A coding agent, a model-driven tool, a + wrapped CLI or the LocalStack container inherits the whole environment when it needs one variable, or can read a + file that holds a secret. (A secret set at job level in a trusted workflow, where one step would do, is a + SHOULD-FIX.) +- **Write access by default.** A new workflow, or a new job or trigger in an existing one, runs without a + `permissions:` block, or a job that runs PR code gets more than it needs (`contents: write`, `packages: write`, + `id-token: write`). A version bump of an action in a listed workflow is not such a change. +- **Publishing from a PR.** `npm publish`, `docker push` or a release step can run on a `pull_request` event. +- **User input in a shell.** `${{ github.event.* }}`, `${{ inputs.* }}` or a branch or PR title reaches a `run:` + step directly instead of through `env:`. +- **Secrets in the repository or the image.** A committed token, key, `.env` file, or a test fixture, snapshot or + eval result that holds a real credential; a build `ARG`/`ENV` or `COPY` that bakes one into the image; a removed + `.dockerignore` entry for `.env`. +- **Secrets in tool output.** An auth token, AWS or Azure credentials or replicator source keys can reach a tool + response, an error message, analytics or the server's logs. +- **New third-party action or package from an unknown publisher**, added without saying in the PR why it is + trusted. (Pinning is checked in §10.) + +A finding here blocks the PR even if the code is otherwise fine. If a secret may already have leaked (a log, an +artifact, a pushed commit), say so in the review so a maintainer rotates it; deleting the artifact is not enough. ## 1. lstk, not the legacy CLI - **BLOCKER:** the change calls, documents or recommends the legacy `localstack` CLI, in code, error hints, README or tool descriptions. -- **BLOCKER:** the container the server starts differs from lstk's in name, labels, ports, volumes or auth env, so - `lstk stop` or `lstk status` cannot find or manage it, or the server cannot manage an lstk-started emulator. +- **BLOCKER:** a change to the container spec adds a difference from lstk's container (name, image, labels, ports, + volumes, environment, auth env) or makes an existing one worse, so `lstk stop` or `lstk status` can find or manage + fewer of the server's emulators, or the server fewer of lstk's; or the change drops detection of `localstack-aws`. + Differences that already exist are not findings; the "Known deviations" list of them is not exhaustive. - **SHOULD-FIX:** a change to the container spec has no test that compares it with lstk's defaults. -- **SHOULD-FIX:** the server calls lstk without passing its own endpoint as `LSTK_ENDPOINT_URL`, so a non-default - `LOCALSTACK_HOSTNAME` or `LOCALSTACK_PORT` sends the command to a different emulator. +- Endpoint handling for lstk calls is an open question in AGENTS.md. Until it is decided, report these as + questions for the user, not as rule violations, unless the failure is confirmed: + - an lstk call can reach a different emulator than the server's other tools, for example because a + `LSTK_ENDPOINT_URL` or `AWS_ENDPOINT_URL` in the user's environment wins (SHOULD-FIX if confirmed) + - an endpoint reaches `lstk start`, `stop`, `restart`, `logs` or `volume`, through the server or the inherited + environment; lstk refuses these commands when one is set (BLOCKER if confirmed: the command fails) ## 2. Cloud and IaC commands -- **BLOCKER:** a new code path runs `awslocal`, `tflocal`, `cdklocal`, `samlocal` or a hand-configured `az` without - trying the lstk command first. -- **BLOCKER:** the lstk/fallback choice is made per tool instead of in the shared helper. -- **SHOULD-FIX:** tests cover only one path. Both the lstk path and the fallback need a test, including "neither is - installed". -- **SHOULD-FIX:** the response does not say which command ran, or the missing-runner hint recommends anything other - than lstk. +- **BLOCKER:** new code runs `awslocal`, `tflocal`, `cdklocal`, `samlocal` or a hand-configured `az`, or adds a + fallback to them. lstk is required. +- **SHOULD-FIX:** no test for lstk being missing, or the missing-lstk message recommends anything other than + installing lstk. +- **SHOULD-FIX:** the response does not say which command ran. +- **SHOULD-FIX:** a tool moves to lstk but the Docker image lacks a binary that lstk command calls. - **BLOCKER:** moving a tool to lstk drops a safety check: the AWS CLI sanitizer, or the Azure profile isolation, egress guard or command policy. @@ -44,11 +101,15 @@ This repository is public, so these are BLOCKERs: - a reference to private LocalStack code: Python module paths (`localstack.pro.*`), private file paths or class names, in code, comments, tests or docs -- an endpoint that is neither in the emulator's OpenAPI spec (`/_localstack/swagger`) nor in the public docs, unless - it is listed under "Gaps in LocalStack" in AGENTS.md +- an endpoint that is in neither the public docs nor one of the emulator's OpenAPI specs (`/_localstack/swagger`; + `/_localstack/appinspector/v1/openapi.yaml` for App Inspector), unless it is listed under "Gaps in LocalStack" in + AGENTS.md - a script or command run inside the LocalStack container to reach internals - an undocumented configuration variable +Also a BLOCKER: the change adopts an API the server did not use before, and the PR does not record that the user +approved it (AGENTS.md rule 4). + ## 5. Compute in LocalStack, render here Flag new code that decides what is true instead of presenting what LocalStack returned. Red flags: @@ -80,8 +141,10 @@ See "IAM policies" in AGENTS.md. - **BLOCKER:** user input reaches a shell (`shell: true`, string commands, unescaped interpolation), or a CLI string bypasses its sanitizer. -- **BLOCKER:** a secret (auth token, AWS or Azure credentials, replicator source keys) can reach a tool response, an - error message or analytics. +- **BLOCKER:** a string from LocalStack or the user's workload (resource names, ARNs, error messages, policy JSON, + log lines) reaches an agent-facing response without escaping. A value must not be able to close a code fence, + break a table, or read as an instruction. Size code fences to their content, and escape markdown in table cells + and inline text. - **SHOULD-FIX:** the handler skips `withToolAnalytics`, preflights or `ResponseBuilder`, or a new tool argument is missing from (or wrongly added to) the analytics allowlist in `src/core/analytics.ts`. - **SHOULD-FIX:** decision or parsing logic lives in the tool file instead of a tested `*.logic.ts` module. @@ -98,10 +161,50 @@ mounted socket. - **SHOULD-FIX:** the change works in one mode only. Check paths (host paths versus mounted paths), hostnames (`localhost` versus `host.docker.internal`) and binaries the image must bundle. - **SHOULD-FIX:** a new host binary is needed but neither README prerequisites nor the `Dockerfile` are updated. +- **BLOCKER:** the image grows by more than about 50 MB per platform, or gains a new runtime, toolchain or base + image, and the PR does not record that the user approved it (AGENTS.md "Docker image size"). +- **SHOULD-FIX:** a change to `Dockerfile`, `docker/` or `.dockerignore`, or a new runtime dependency, without the + image size before and after for amd64 and arm64. +- **SHOULD-FIX:** the runtime stage keeps build tools, dev dependencies, package caches or package lists, or cleans + them in a later layer than the one that created them. ## 9. Tests and docs -- **SHOULD-FIX:** new `*.logic.ts` code without unit tests, or a new tool or action without a case in - `tests/mcp/direct.spec.mjs`. +See "Testing" in AGENTS.md. + +- **BLOCKER:** a bug fix without a test that fails before the fix. +- **SHOULD-FIX:** a new or changed action without a handler test in `src/tools-tests/.test.ts` covering its + success path, each error path (`isError: true`) and each preflight it runs. +- **SHOULD-FIX:** new `*.logic.ts` code without unit tests. +- **SHOULD-FIX:** a tool that renders payload or workload strings, or builds a command line, without an adversarial + test (fence and table injection; shell chaining, option injection, path traversal). +- **SHOULD-FIX:** a tool that handles a secret without a test asserting the secret is absent from the response, + the error and analytics. +- **SHOULD-FIX:** tests assert on internal calls instead of the response, the spawned argument array or the HTTP + request sent to LocalStack. +- **SHOULD-FIX:** a new tool missing from `EXPECTED_TOOLS` in `tests/mcp/direct.spec.mjs`, or a new action without + a case in `tests/mcp/tools.spec.mjs` that checks its effect through another call. +- **SHOULD-FIX:** a test reaches real AWS or Azure resources; creates a LocalStack platform resource without + cleanup, or with a fixed name outside a `concurrency` group; or skips or passes silently when a required secret + is absent (AGENTS.md "Testing" names the only skips allowed). +- **SHOULD-FIX:** a tool's description, schema or response wording changed and the PR does not report an eval run. - **SHOULD-FIX:** `README.md` tool or prerequisite tables are out of date. -- **NIT:** a fixed known deviation still listed in AGENTS.md, or a new one not listed. +- **NIT:** a fixed known deviation still listed in AGENTS.md. (New code that breaks a rule is a finding at that + rule's severity; see the top of this file.) + +## 10. Dependencies and CVEs + +See "Dependencies" in AGENTS.md. + +- **BLOCKER:** a dependency or tool installed without a pinned version in a file Dependabot updates: `npm install -g`, + `pip install`, `curl … | sh`, a `latest` tag (other than the LocalStack emulator images), or `npx` of a package + not in `package.json`. +- **BLOCKER:** a known high or critical advisory is silenced (an `audit-ci` allowlist entry, a removed audit step) + without the advisory ID, why it does not apply, and a date to revisit. +- **SHOULD-FIX:** a new manifest, lockfile, ecosystem or Dockerfile stage without a matching entry in + `.github/dependabot.yml`. +- **SHOULD-FIX:** a version pinned by hand that Dependabot could manage, or a hand-managed pin without an `ignore` + entry giving the reason. +- **SHOULD-FIX:** a new `uses:` (a new action, or an existing one in a new step) not pinned to a commit SHA with a + version comment. Dependabot bumping a tag pin that is a known deviation is fine. +- **SHOULD-FIX:** a lockfile changed without the manifest change that explains it, or the reverse. From e6dcdfbadede8de4facbccca7e6252a525360665 Mon Sep 17 00:00:00 2001 From: Joel Scheuner Date: Thu, 19 Nov 2026 17:24:32 +0100 Subject: [PATCH 03/13] docs: keep inventories out of AGENTS.md Replace counts, versions, port ranges and field lists with rules and pointers to the source, and add "Maintaining this file": lists the code can check belong in ratchet tests, deviations are described by pattern. Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 39 +++++++++++++++++++++++++++------------ REVIEW.md | 2 ++ 2 files changed, 29 insertions(+), 12 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 4687de7..39c78c4 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -31,7 +31,7 @@ yarn format # Prettier | `src/lib/docker/` | Docker Engine API client (find, start, stop, exec, logs). | | `src/lib/localstack/` | LocalStack HTTP clients and the container spec. | | `src/cli/` | The `init` / `remove` setup wizard and the stdio entry point. | -| `src/tools-tests/` | Handler tests, one file per tool (2 of 15 so far). They live outside `src/tools/` because xmcp registers every file there as a tool. | +| `src/tools-tests/` | Handler tests, one file per tool. They live outside `src/tools/` because xmcp registers every file there as a tool. | | `tests/mcp/` | Playwright tests against the built server. | | `tests/docker/` | Validation of the Docker image. | | `docs/` | Topic docs (`AZURE.md`, `DOCKER.md`). | @@ -229,10 +229,9 @@ public docs or in one of the OpenAPI specs a running emulator serves: Inspector Ask the user before adopting an API this server does not use yet, even a documented one, and name the endpoint and -what the tool would do with it. The App Inspector routes the server uses today: the traces, spans and events -routes under `/v1/traces` (list, get and delete, down to `/v1/traces/{t}/spans/{s}/events/{e}` and `/events/iam`), -and `/_localstack/appinspector/status` (not in the spec yet, see [Gaps in LocalStack](#gaps-in-localstack)). The -IAM design below adds `/v1/events/{event_id}/policies/{key}` and the IAM policy summary. +what the tool would do with it. The APIs the server uses today are the ones its clients call +(`src/lib/localstack/*.client.ts`, `src/core/config.ts`); an endpoint missing there is new. The IAM design below +adds the event policy route and the IAM policy summary. Exception: an endpoint listed under [Gaps in LocalStack](#gaps-in-localstack) may be used before it is documented. Each such endpoint is one LocalStack's own clients (the web app, App Inspector) already depend on. Describe its @@ -348,23 +347,23 @@ stay until the LocalStack API that replaces them ships. | Where | Deviation | Target | |-------|-----------|--------| -| `src/lib/localstack/container-spec.logic.ts`, `src/lib/docker/docker.client.ts` | Its header comment describes the container as replicating the legacy `localstack start` (rule 1). The container differs from lstk's; the differences known so far, not an exhaustive list: the name (`localstack-main` for every emulator; lstk uses `localstack-aws`, `localstack-snowflake`, `localstack-azure`, or `localstack--`), and the server detects only `localstack-main` and `localstack-aws`; no `cloud.localstack.lstk` label; its own volume directory (`/localstack/volume` or the named volume `localstack-mcp`; lstk: `/lstk/volume/`), so emulator state does not carry over; service ports 4510–4560 (lstk: 4510–4559); and the environment (lstk sets `LOCALSTACK_HOST`, `LOCALSTACK_CLIENT_NAME=lstk` and, for Snowflake, `SF_S3_ENDPOINT`; the server sets none of them and adds `EXTERNAL_SERVICE_PORTS_*`). Interoperability rests on the server detecting both AWS names and on lstk's fallback, which matches a known LocalStack image serving the gateway on container port 4566 (not the host port, so a non-default `LOCALSTACK_PORT` is fine). It breaks when the server runs an image lstk does not know (a mirror or custom build), a `GATEWAY_LISTEN` moves the gateway off 4566 inside the container, or the emulator is Snowflake or Azure: lstk then reports no emulator running, and `lstk start` starts a second one beside it. | Converge on lstk's container spec; decide later whether to adopt its names. Until then, keep both AWS names detected (rule 1). | +| `src/lib/localstack/container-spec.logic.ts`, `src/lib/docker/docker.client.ts` | The header comment describes the container as replicating the legacy `localstack start` (rule 1). The container spec differs from lstk's in several ways, among them the name (`localstack-main` for every emulator), labels, volume directory, ports and environment, so emulator state does not carry over and only AWS containers are found in both directions. Compare with lstk's `internal/config/containers.go` and `internal/container/start.go` for the current differences. Interoperability rests on the server detecting lstk's AWS container name and on lstk's fallback, which matches a known LocalStack image serving the gateway on container port 4566; it breaks for other images, a gateway moved off 4566 inside the container, and the other emulators. | Converge on lstk's container spec; decide later whether to adopt its names. Until then, keep lstk's AWS name detected (rule 1). | | `src/tools/localstack-aws-client.ts` | Runs `awslocal` inside the container via `docker exec`. | `lstk aws`, no fallback (rule 2). | | `src/tools/localstack-deployer.ts`, `src/prompts/infrastructure-tester.ts` | The deployer runs `awslocal cloudformation` through `docker exec` for CloudFormation deploys and deletes, and the prompt tells the agent to use `awslocal`. | `lstk aws cloudformation` (rule 2); the prompt names lstk. | | `src/tools/localstack-chaos-injector.ts` | Builds some `❌` responses by hand instead of through `ResponseBuilder`, so they are not flagged with `isError`. | `ResponseBuilder.error` ("Writing a tool"). | | `src/core/localstack-env.ts` | Passes the server's target to `lstk terraform` / `cdk` / `sam` as `AWS_ENDPOINT_URL`, but keeps an `AWS_ENDPOINT_URL` already in the user's environment, and a `LSTK_ENDPOINT_URL` there overrides both: either sends the command to a different emulator. | Depends on the endpoint-scoping [open question](#open-questions). | | `src/tools/localstack-azure-client.ts` | Configures the host `az` itself. | `lstk az`, keeping the existing guards (rule 2). | | `Dockerfile` | Bundles lstk, Terraform, CDK and SAM, but not the AWS CLI. | Bundle the AWS CLI for `lstk aws` (rule 2). This is a larger image change: ask the user first ("Docker image size"). | -| `src/core/analytics.ts` | Sends data that can hold secrets (REVIEW §0): the full `command` of `localstack-aws-client` and `localstack-azure-client` (for example `secretsmanager create-secret --secret-string …`), and the first line of every error response as `error_message`. `looksSensitiveValue` only catches a value that is a whole key or token. The allowlist also sends user data: the deployer's `directory`, `stackName`, `templatePath` and `s3Bucket`, the docs `query`, the logs `filter`, the Cloud Pods `pod_name`, the ephemeral instance `name` and `cloudPod`, and the extension `name`. | Only fields that can never hold secrets or user data ("Writing a tool"): the command's service and operation instead of the command, an error code instead of the message. | -| `.github/workflows/*.yml` | Actions are pinned to tags (`@v7`), not commit SHAs. | SHA pins with a version comment ("Dependencies"). | +| `src/core/analytics.ts` | Sends data that can hold secrets (REVIEW §0): free-form arguments such as the cloud CLIs' `command` (for example `--secret-string …`), and error text. `looksSensitiveValue` only catches a value that is a whole key or token. The allowlist also sends arguments that hold user data: names, paths, queries and filters. | Only fields that can never hold secrets or user data ("Writing a tool"): for example the command's service and operation instead of the command, an error code instead of the message. | +| `.github/workflows/*.yml` | Actions are pinned to version tags, not commit SHAs. | SHA pins with a version comment ("Dependencies"). | | `.github/workflows/ci.yml`, `docker.yml` | No `permissions:` block, so jobs get the repository's default token permissions. | `permissions: contents: read` at the top, widened per job only where needed (REVIEW §0). | | `src/tools/localstack-iam-policy-analyzer.ts` | Finds IAM denials by parsing container logs (rule 3). | The IAM actions of `localstack-app-inspector` ("IAM policies"); then deprecate the analyzer and remove it after a notice period. | | `.github/workflows/azure-smoke.yml`, `ci.yml` (`mcp-direct-tests`), `tests/mcp/azure-offline.spec.mjs` | `azure-smoke.yml` runs on `ubuntu-latest`; it and `mcp-direct-tests` use the runner image's Azure CLI, whose version no file here pins. `azure-offline` skips its cases when `az` is missing, a skip "Testing" does not allow. | The pinned Ubuntu of the other workflows, and the Azure CLI from a pinned file, as `host-tools` does for Terraform, uv and lstk. | | `.github/workflows/ci.yml`, `mcp-evals.yml`, `azure-smoke.yml` | Set `LOCALSTACK_AUTH_TOKEN` (and in `mcp-evals.yml` `OPENROUTER_API_KEY`) at job level, so `yarn install` runs dependency install scripts with them, and so do third-party actions (REVIEW §0). | Secrets only on the steps that need them, after dependencies are installed. | -| `data/sample-terraform/` | The AWS provider (`~> 5.0`) has no lockfile and no Dependabot entry; CI downloads it with secrets in the job's environment. | A committed `.terraform.lock.hcl` and a `terraform` Dependabot entry ("Dependencies"). | +| `data/sample-terraform/` | The AWS provider has only a version constraint, no lockfile, and no Dependabot entry; CI downloads it with secrets in the job's environment. | A committed `.terraform.lock.hcl` and a `terraform` Dependabot entry ("Dependencies"). | | `playwright.config.mjs` | Passes the whole environment to the server under test, so in `mcp-evals.yml` the `OPENROUTER_API_KEY` reaches the server and the `lstk` processes it starts (REVIEW §0). | Pass the server only what it needs; the model keys stay with the eval host. | -| `Dockerfile` | `# syntax=docker/dockerfile:1` floats to the newest 1.x frontend. | Pin the frontend version where Dependabot can update it. | -| `src/tools-tests/`, `src/lib/**/*.logic.test.ts` | Only `localstack-management` and `localstack-azure-client` have handler tests. `localstack-management.test.ts` stubs `withToolAnalytics` and mocks the internal `localstack.utils` module, asserting on its calls. `app-inspector.logic.test.ts`, `aws-replicator.logic.test.ts` and `state-management.logic.test.ts` test tool handlers, not a `*.logic.ts` module. | Handler tests for every tool, following "Testing"; move handler tests to `src/tools-tests/`. | +| `Dockerfile` | The `# syntax=` line floats to the newest frontend of its major version. | Pin the frontend version where Dependabot can update it. | +| `src/tools-tests/`, `src/lib/**/*.logic.test.ts` | Most tools have no handler test. The existing handler tests stub `withToolAnalytics` or mock internal modules and assert on their calls, and some `*.logic.test.ts` files test tool handlers instead of a `*.logic.ts` module. | Handler tests for every tool, following "Testing", with tool handler tests in `src/tools-tests/`. A ratchet test that lists the tools still without one would replace this row (see "Maintaining this file"). | | `src/lib/iam/iam-policy.logic.ts` | Synthesizes IAM policies from denials, for the analyzer (rule 5). | Remove with the analyzer; the IAM actions present LocalStack's suggestions and summary instead. | | `src/lib/logs/log-retriever.ts`, `src/tools/localstack-logs-analysis.ts` | Regex-classifies log lines (service, operation, status, IAM), which the logs tool's `summary`, `errors` and `requests` modes present. | Raw and filtered lines only (rule 3); App Inspector for failed requests. | | `src/lib/localstack/extensions.logic.ts` | Runs a private Python module inside the container (the same one the legacy CLI runs), and names that private module and the container's internal paths in public code (rule 4). | The extensions HTTP API: `GET /_localstack/extensions/list`, `POST …/install`, `POST …/uninstall` (newline-delimited JSON progress events; LocalStack restarts itself afterwards). See [Gaps in LocalStack](#gaps-in-localstack). | @@ -394,8 +393,24 @@ Planned work that is not on `main` yet. Ask the user before starting one. Claude Code is pinned in a lockfile Dependabot updates. - **`lstk mcp`'s `failed-requests` action (lstk, unreleased).** It groups App Inspector's failed spans (`status_code=2`) by service, operation, HTTP status and AWS error code, with counts, first and last time and the - latest example, reading at most 5,000 spans. That computation is an approved exception to rule 5: approved by the + latest example. That computation is an approved exception to rule 5: approved by the project maintainer on 2026-11-17, because it replaces the log-based error summary of `localstack-logs-analysis` with one from App Inspector's data. It is recorded here because the code is not on lstk's `main` yet; when it lands, record it where lstk's MCP server documents its design. It goes away once the failed-requests summary in [Gaps in LocalStack](#gaps-in-localstack) ships. + +## Maintaining this file + +Write what stays true while features land, so this file changes with decisions, not with every PR. + +- **No inventories.** No counts ("2 of 15"), version numbers, line numbers, port numbers, or lists of files, fields + or routes that grow with the code. State the rule and point to where the current answer lives: a file, a + command, or the upstream source to compare with. +- **Let code hold the lists.** A rule the code can check belongs in a test, not here. For existing code that does + not follow it yet, the test keeps an allowlist that only shrinks: it fails when new code breaks the rule and + when a listed entry no longer needs to be listed. This file then names the test instead of the entries, as + `src/core/dependency-pins.test.ts` does for dependency versions. +- **Deviations by pattern.** A row in [Known deviations](#known-deviations) names the pattern, where it lives and + the target, not every instance. +- **Name a file only when it defines the convention** (`container-spec.logic.ts` for the container, + `response-builder.ts` for responses). diff --git a/REVIEW.md b/REVIEW.md index 6eca214..3ae370d 100644 --- a/REVIEW.md +++ b/REVIEW.md @@ -189,6 +189,8 @@ See "Testing" in AGENTS.md. is absent (AGENTS.md "Testing" names the only skips allowed). - **SHOULD-FIX:** a tool's description, schema or response wording changed and the PR does not report an eval run. - **SHOULD-FIX:** `README.md` tool or prerequisite tables are out of date. +- **NIT:** a change adds a count, version, line number or growing list to AGENTS.md instead of a rule and a + pointer, or a list a test could hold (AGENTS.md, "Maintaining this file"). - **NIT:** a fixed known deviation still listed in AGENTS.md. (New code that breaks a rule is a finding at that rule's severity; see the top of this file.) From a3f419386102941de5e0bfc55ce102f50da61f0a Mon Sep 17 00:00:00 2001 From: Joel Scheuner Date: Thu, 19 Nov 2026 18:00:02 +0100 Subject: [PATCH 04/13] docs: address review round findings (CI secrets, child env, licenseinfo, endpoint proposal) Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 37 +++++++++++++++++++++---------------- REVIEW.md | 10 ++++++---- 2 files changed, 27 insertions(+), 20 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 39c78c4..83cd378 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -61,7 +61,7 @@ the failing test first when fixing a bug. | Logic | `src/lib//*.logic.test.ts` | always | Parsing, validation, rendering and sanitizing, as pure functions. | | Handler | `src/tools-tests/.test.ts` | always | The tool's actions end to end inside the process, with Docker, LocalStack and spawned binaries mocked at the module boundary. No existing handler test follows every rule below yet; do not copy their mocks of internal modules. | | Tools | `tests/mcp/tools.spec.mjs` | every PR, with the auth token | Each tool over MCP stdio against a real emulator, without a model. Each case reads the state the tool left behind back through another call. | -| Image | `tests/mcp/direct.spec.mjs`, `tests/docker/validate-image.mjs` | every PR (`docker.yml` only when the image's inputs change; its integration test on amd64 only), with the auth token | The built server and the Docker image: tool list, prompts, setup wizard, and real calls through the image. | +| Image | `tests/mcp/direct.spec.mjs`, `tests/docker/validate-image.mjs` | every PR (`docker.yml` when its path filter matches; its integration test on amd64 only), with the auth token | The built server and the Docker image: tool list, prompts, setup wizard, and real calls through the image. | Rules for tool tests: @@ -79,7 +79,8 @@ Rules for tool tests: that builds a command line, add cases that try shell chaining, option injection and path traversal. - **No secrets in output.** Run the handler with a recognizable fake token and credentials, and assert they appear in neither the response, the error, nor the analytics payload. -- **Real-emulator coverage.** A new tool goes into `EXPECTED_TOOLS` in `tests/mcp/direct.spec.mjs`. A new action +- **Real-emulator coverage.** A new tool goes into `EXPECTED_TOOLS` in `tests/mcp/direct.spec.mjs` and in + `tests/docker/validate-image.mjs`. A new action gets a case in `tests/mcp/tools.spec.mjs` that checks its effect through another call, not just its success message, so a tool that skips its API call but reports success fails. Add a case to `tests/docker/validate-image.mjs` when the action depends on the image (bundled binaries, mounted paths). @@ -88,9 +89,9 @@ Rules for tool tests: name only when a workflow `concurrency` group keeps runs from overlapping, as `mcp-evals.yml` does. Costly ones (ephemeral instances) stay opt-in (`HARNESS_RUN_EPHEMERAL`). - **Missing secrets fail loudly.** A job that needs a secret fails with a message naming it when it is empty, as - `ci.yml` does for fork PRs, so a fork PR never looks tested. Only an opt-in scenario's own credentials (the - replicator's source AWS keys) or an outside service being down (the docs search) may skip a case, and the - output says why. + `ci.yml` does for fork PRs, so a fork PR never looks tested. Only an opt-in scenario (one that needs its own + credentials, such as the replicator's source AWS keys, or a flag such as `HARNESS_RUN_REMOTE`) or an outside + service being down (the docs search) may skip a case, and the output says why. - **Evals are not PR gates.** `test:mcp:evals` depends on a model. `mcp-evals.yml` runs it after merges to `main`, weekly and on demand. Run them yourself when a tool's description, schema or response wording changes, and report the result in the PR. @@ -106,7 +107,7 @@ arrive as PRs instead of waiting for someone to notice: | CLIs in the image (aws-cdk, lstk) | `docker/tools/npm/package-lock.json` | | Python CLIs in the image (SAM, Snowflake) | `docker/tools//uv.lock` | | Base images, Terraform, uv | `FROM` lines in `Dockerfile` | -| GitHub Actions | `uses:` in `.github/workflows/` | +| GitHub Actions | `uses:` in `.github/workflows/` and `.github/actions/*/action.yml` (each directory needs a Dependabot entry) | - **No version outside these files.** No `npm install -g `, `pip install `, `curl … | sh`, `latest` tags (except the LocalStack emulator images, which track the newest release on purpose), or `npx ` for a package @@ -175,8 +176,9 @@ without lstk. That container must be interchangeable with one lstk starts: port 4566. - **Same endpoint, for the commands that take one.** `lstk aws`, `az`, `terraform`, `cdk`, `sam`, `status`, `inspect`, `reset` and `snapshot` (save, load, remove, and list of an `s3://` remote) accept an endpoint. Pass - the server's target (`LOCALSTACK_HOSTNAME` / `LOCALSTACK_PORT`) to them so both talk to the same emulator. Set `LSTK_ENDPOINT_URL`: lstk prefers it over `AWS_ENDPOINT_URL`, so a value already in the - user's environment would otherwise win. lstk refuses an endpoint for the local lifecycle commands (`start`, + the server's target (`LOCALSTACK_HOSTNAME` / `LOCALSTACK_PORT`) to them so both talk to the same emulator. The current proposal is + `LSTK_ENDPOINT_URL`, which lstk prefers over `AWS_ENDPOINT_URL`, so a value already in the user's environment + would not win. lstk refuses an endpoint for the local lifecycle commands (`start`, `stop`, `restart`, `logs`, `volume`), so setting one there makes the command fail. Which commands get the endpoint is not decided yet: ask the user before wiring lstk calls (see [Open questions](#open-questions)). @@ -231,7 +233,8 @@ public docs or in one of the OpenAPI specs a running emulator serves: Ask the user before adopting an API this server does not use yet, even a documented one, and name the endpoint and what the tool would do with it. The APIs the server uses today are the ones its clients call (`src/lib/localstack/*.client.ts`, `src/core/config.ts`); an endpoint missing there is new. The IAM design below -adds the event policy route and the IAM policy summary. +would add the event policy route and the IAM policy summary; adopting them still needs the user's approval, +recorded in the PR that adds them. Exception: an endpoint listed under [Gaps in LocalStack](#gaps-in-localstack) may be used before it is documented. Each such endpoint is one LocalStack's own clients (the web app, App Inspector) already depend on. Describe its @@ -278,7 +281,7 @@ what is missing instead of returning an empty answer. `outcome`, `entries`, `hints`, `near_misses`). `explain-iam` is to render them in the section order of the App Inspector UI's "Copy agent prompt", then the payload as sent. Do not recompute, merge or reorder the changes. A policy stored truncated (`statement_count` without `document`) is fetched on demand with `get-iam-policy` - (`/v1/events/{event_id}/policies/{key}`), never automatically. + (`/v1/events/{event_id}/policies/{policy_key}`), never automatically. - Without the `localstack.appinspector/view_iam_policy_errors` entitlement, LocalStack hides IAM events and returns an empty list. Spans say so (`iam_errors_suppressed`) and the policy route answers 403 `not_entitled`, but the event lists don't, so an empty IAM result reads one span of the scope and reports @@ -321,14 +324,14 @@ row once it ships. | Need | Status | Used by | |------|--------|---------| | Document the extensions API (`GET /_localstack/extensions/list`, `POST …/install`, `POST …/uninstall`) in the OpenAPI spec and the docs | The LocalStack web app's Extension Manager depends on it; neither the spec nor the docs name it. lstk has no extensions commands. | `localstack-extensions` (allowed under the rule 4 exception) | -| Document the IAM evaluation event payload (the `payload` attribute of App Inspector's IAM events: the decisions, the `policies` map and the `suggestion`) in the App Inspector spec | The spec types the attribute as a string; its fields are not documented anywhere public. The App Inspector UI depends on them. Describe them in this repository only as types and tests. | `explain-iam`, planned (allowed under the rule 4 exception) | +| Document the IAM evaluation event payload (the `payload` attribute of App Inspector's IAM events: the decisions, the `policies` map and the `suggestion`) in the App Inspector spec | The spec types the attribute as a string; its fields are not documented anywhere public. The App Inspector UI depends on them. Until they are documented, describe them in this repository's code as types and tests written from observed responses, not from LocalStack's code; this file names them only as far as the design needs. | `explain-iam`, planned (allowed under the rule 4 exception) | | Document the IAM policy summary and stream (`GET`/`DELETE /_aws/iam/policies/summary`, `GET /_aws/iam/policies/stream`) | App Inspector's Policies view and the web app's IAM Policy Stream depend on them; the spec documents only `/_aws/iam/config` and `/_aws/iam/check-actions-allowed`. | `localstack-app-inspector` IAM job 2, planned (allowed under the rule 4 exception) | | Filter the IAM policy summary by trace (or time range), so a run's policies don't mix with earlier requests | Not implemented: the summary is global and only resettable. | `get-iam-policy-summary` (planned) | | Say on the App Inspector events page when IAM events are hidden by the license (as spans do with `iam_errors_suppressed`) | Not implemented: the events routes return an empty list, so a tool has to probe one span. | `list-iam-denials`, `explain-iam` (planned) | | Compare suggested policies with deployed ones: permissions to add and permissions granted but unused, per ARN | Not implemented. Listed as a follow-up in localstack-appinspector-ui#341. | IAM job 3 | | Document App Inspector's status route (`GET`/`PUT /_localstack/appinspector/status`) in its OpenAPI spec | Not in `/_localstack/appinspector/v1/openapi.yaml`. lstk's `inspect on` / `off` uses it too. | `get-status`, `set-status` (allowed under the rule 4 exception) | | Presentation fields for IAM: a display name, the services and the action count per policy summary entry; the principal kind (ARN, service, account) on suggestion hints; the statements a decision names, resolved | Not implemented. Until it ships, show the ARN, the type and the policy JSON, name statement positions without resolving them, and word hints for any kind of caller. | `explain-iam`, `get-iam-policy-summary` (planned) | -| Document `GET /_localstack/licenseinfo`, or name the documented way to tell whether a licensed feature is available | In none of the emulator's specs and not in the docs. Whether a LocalStack client depends on it is unverified: lstk and the App Inspector UI don't. | `requireProFeature` in every Pro tool (allowed under the rule 4 exception until LocalStack answers) | +| Document `GET /_localstack/licenseinfo`, or name the documented way to tell whether a licensed feature is available | In none of the emulator's specs and not in the docs. Whether a LocalStack client depends on it is unverified: lstk and the App Inspector UI don't. | `requireProFeature` in every Pro tool. The rule 4 exception does not cover it, since no LocalStack client is known to depend on it; see [Known deviations](#known-deviations). | | A failed-requests summary in App Inspector: failed spans grouped by service, operation, HTTP status and AWS error code, with counts, first and last time, and an example trace and span | Not implemented. | `lstk mcp`'s `localstack-app-inspector` `failed-requests`, planned (an approved exception, see [Follow-ups](#follow-ups)) | ## Known deviations @@ -350,7 +353,7 @@ stay until the LocalStack API that replaces them ships. | `src/lib/localstack/container-spec.logic.ts`, `src/lib/docker/docker.client.ts` | The header comment describes the container as replicating the legacy `localstack start` (rule 1). The container spec differs from lstk's in several ways, among them the name (`localstack-main` for every emulator), labels, volume directory, ports and environment, so emulator state does not carry over and only AWS containers are found in both directions. Compare with lstk's `internal/config/containers.go` and `internal/container/start.go` for the current differences. Interoperability rests on the server detecting lstk's AWS container name and on lstk's fallback, which matches a known LocalStack image serving the gateway on container port 4566; it breaks for other images, a gateway moved off 4566 inside the container, and the other emulators. | Converge on lstk's container spec; decide later whether to adopt its names. Until then, keep lstk's AWS name detected (rule 1). | | `src/tools/localstack-aws-client.ts` | Runs `awslocal` inside the container via `docker exec`. | `lstk aws`, no fallback (rule 2). | | `src/tools/localstack-deployer.ts`, `src/prompts/infrastructure-tester.ts` | The deployer runs `awslocal cloudformation` through `docker exec` for CloudFormation deploys and deletes, and the prompt tells the agent to use `awslocal`. | `lstk aws cloudformation` (rule 2); the prompt names lstk. | -| `src/tools/localstack-chaos-injector.ts` | Builds some `❌` responses by hand instead of through `ResponseBuilder`, so they are not flagged with `isError`. | `ResponseBuilder.error` ("Writing a tool"). | +| `src/tools/localstack-chaos-injector.ts`, `src/tools/localstack-ephemeral-instances.ts`, `src/tools/localstack-aws-replicator.ts` | Tools that skip parts of "Writing a tool": hand-built `❌` responses instead of `ResponseBuilder` (`withToolAnalytics` flags them as errors only by their prefix), a handler without `runPreflights`, and ARN parsing in the tool file (rule 5). | Follow "Writing a tool". | | `src/core/localstack-env.ts` | Passes the server's target to `lstk terraform` / `cdk` / `sam` as `AWS_ENDPOINT_URL`, but keeps an `AWS_ENDPOINT_URL` already in the user's environment, and a `LSTK_ENDPOINT_URL` there overrides both: either sends the command to a different emulator. | Depends on the endpoint-scoping [open question](#open-questions). | | `src/tools/localstack-azure-client.ts` | Configures the host `az` itself. | `lstk az`, keeping the existing guards (rule 2). | | `Dockerfile` | Bundles lstk, Terraform, CDK and SAM, but not the AWS CLI. | Bundle the AWS CLI for `lstk aws` (rule 2). This is a larger image change: ask the user first ("Docker image size"). | @@ -359,15 +362,17 @@ stay until the LocalStack API that replaces them ships. | `.github/workflows/ci.yml`, `docker.yml` | No `permissions:` block, so jobs get the repository's default token permissions. | `permissions: contents: read` at the top, widened per job only where needed (REVIEW §0). | | `src/tools/localstack-iam-policy-analyzer.ts` | Finds IAM denials by parsing container logs (rule 3). | The IAM actions of `localstack-app-inspector` ("IAM policies"); then deprecate the analyzer and remove it after a notice period. | | `.github/workflows/azure-smoke.yml`, `ci.yml` (`mcp-direct-tests`), `tests/mcp/azure-offline.spec.mjs` | `azure-smoke.yml` runs on `ubuntu-latest`; it and `mcp-direct-tests` use the runner image's Azure CLI, whose version no file here pins. `azure-offline` skips its cases when `az` is missing, a skip "Testing" does not allow. | The pinned Ubuntu of the other workflows, and the Azure CLI from a pinned file, as `host-tools` does for Terraform, uv and lstk. | -| `.github/workflows/ci.yml`, `mcp-evals.yml`, `azure-smoke.yml` | Set `LOCALSTACK_AUTH_TOKEN` (and in `mcp-evals.yml` `OPENROUTER_API_KEY`) at job level, so `yarn install` runs dependency install scripts with them, and so do third-party actions (REVIEW §0). | Secrets only on the steps that need them, after dependencies are installed. | +| `.github/workflows/` | Secrets reach third-party code in CI (REVIEW §0): jobs set `LOCALSTACK_AUTH_TOKEN` (and, in `mcp-evals.yml`, the OpenRouter key) at job level, so `yarn install` runs dependency install scripts and third-party actions run with them; `docker.yml` runs `npm ci` for `data/sample-cdk`, with install scripts, in a step that holds the token; and `actions/checkout` leaves the GitHub token in `.git/config` for later steps. | Install dependencies with no secret in the environment (or with `--ignore-scripts`), set secrets only on the steps that need them, and check out with `persist-credentials: false`. | +| `.github/workflows/docker.yml` | Its path filter misses files the image build reads (`patches/`, `xmcp.config.ts`, `tsconfig*.json`), so a change to them is not built or tested as an image. | Filter on everything the `Dockerfile` copies. | +| `src/lib/localstack/license-checker.ts` | Calls `GET /_localstack/licenseinfo`, which is not documented and not covered by the rule 4 exception. | The documented way LocalStack names (see [Gaps in LocalStack](#gaps-in-localstack)). | | `data/sample-terraform/` | The AWS provider has only a version constraint, no lockfile, and no Dependabot entry; CI downloads it with secrets in the job's environment. | A committed `.terraform.lock.hcl` and a `terraform` Dependabot entry ("Dependencies"). | -| `playwright.config.mjs` | Passes the whole environment to the server under test, so in `mcp-evals.yml` the `OPENROUTER_API_KEY` reaches the server and the `lstk` processes it starts (REVIEW §0). | Pass the server only what it needs; the model keys stay with the eval host. | +| `playwright.config.mjs`, `src/tools/localstack-snowflake-client.ts`, `src/core/localstack-env.ts` | Child processes inherit the whole environment (REVIEW §0): the server under test gets the eval's model keys, and the `snow` CLI and `lstk terraform` / `cdk` / `sam` get every variable of the server's environment. | Pass each child only the variables it needs. | | `Dockerfile` | The `# syntax=` line floats to the newest frontend of its major version. | Pin the frontend version where Dependabot can update it. | | `src/tools-tests/`, `src/lib/**/*.logic.test.ts` | Most tools have no handler test. The existing handler tests stub `withToolAnalytics` or mock internal modules and assert on their calls, and some `*.logic.test.ts` files test tool handlers instead of a `*.logic.ts` module. | Handler tests for every tool, following "Testing", with tool handler tests in `src/tools-tests/`. A ratchet test that lists the tools still without one would replace this row (see "Maintaining this file"). | | `src/lib/iam/iam-policy.logic.ts` | Synthesizes IAM policies from denials, for the analyzer (rule 5). | Remove with the analyzer; the IAM actions present LocalStack's suggestions and summary instead. | | `src/lib/logs/log-retriever.ts`, `src/tools/localstack-logs-analysis.ts` | Regex-classifies log lines (service, operation, status, IAM), which the logs tool's `summary`, `errors` and `requests` modes present. | Raw and filtered lines only (rule 3); App Inspector for failed requests. | | `src/lib/localstack/extensions.logic.ts` | Runs a private Python module inside the container (the same one the legacy CLI runs), and names that private module and the container's internal paths in public code (rule 4). | The extensions HTTP API: `GET /_localstack/extensions/list`, `POST …/install`, `POST …/uninstall` (newline-delimited JSON progress events; LocalStack restarts itself afterwards). See [Gaps in LocalStack](#gaps-in-localstack). | -| `README.md` | Describes the analyzer's log search and policy generation, the aws-client's `awslocal`, and the logs tool's classification as features, and refers to the legacy "LocalStack CLI" for the volume path. | Update with the code; name lstk (rule 1). | +| `README.md`, `docs/` | Describe the analyzer's log search and policy generation, the aws-client's `awslocal`, and the logs tool's classification as features, and refer to the legacy "LocalStack CLI" for the volume path. | Update with the code; name lstk (rule 1). | ## Open questions diff --git a/REVIEW.md b/REVIEW.md index 3ae370d..9c7ed49 100644 --- a/REVIEW.md +++ b/REVIEW.md @@ -33,7 +33,7 @@ Each of these is a BLOCKER: add `LOCALSTACK_AUTH_TOKEN` or another secret as a Dependabot secret: it would reach the install scripts of the packages the PR just bumped. Review the bump, then run it with secrets the same way as an external PR. - **An external PR pushed unreviewed.** For a fork or Dependabot PR to be tested, a maintainer pushes its branch to this - repository, where every workflow runs with secrets. Review the whole diff under this section before pushing it, + repository and opens a PR from it, so its workflows run with secrets. Review the whole diff under this section before pushing it, and push the reviewed commit, not a branch the contributor can still change. - **Secrets printed or stored.** A step or script echoes a secret, runs `set -x`, `env` or `printenv`, logs request headers, or writes a secret to a file. GitHub masks secrets in logs only: an uploaded artifact (test results, @@ -41,8 +41,10 @@ Each of these is a BLOCKER: Check what each `upload-artifact` path can contain. - **Secrets handed to something that records or runs untrusted input.** A coding agent, a model-driven tool, a wrapped CLI or the LocalStack container inherits the whole environment when it needs one variable, or can read a - file that holds a secret. (A secret set at job level in a trusted workflow, where one step would do, is a - SHOULD-FIX.) + file that holds a secret. Or a step installs or runs third-party code (dependency install scripts, a sample + project's packages, a third-party action) while a secret is in its environment or in `.git/config` + (`actions/checkout` keeps the GitHub token there unless `persist-credentials: false`). A secret set at job level + where only first-party code runs is a SHOULD-FIX. - **Write access by default.** A new workflow, or a new job or trigger in an existing one, runs without a `permissions:` block, or a job that runs PR code gets more than it needs (`contents: write`, `packages: write`, `id-token: write`). A version bump of an action in a listed workflow is not such a change. @@ -182,7 +184,7 @@ See "Testing" in AGENTS.md. the error and analytics. - **SHOULD-FIX:** tests assert on internal calls instead of the response, the spawned argument array or the HTTP request sent to LocalStack. -- **SHOULD-FIX:** a new tool missing from `EXPECTED_TOOLS` in `tests/mcp/direct.spec.mjs`, or a new action without +- **SHOULD-FIX:** a new tool missing from `EXPECTED_TOOLS` in `tests/mcp/direct.spec.mjs` or `tests/docker/validate-image.mjs`, or a new action without a case in `tests/mcp/tools.spec.mjs` that checks its effect through another call. - **SHOULD-FIX:** a test reaches real AWS or Azure resources; creates a LocalStack platform resource without cleanup, or with a fixed name outside a `concurrency` group; or skips or passes silently when a required secret From b78d8ec140eaf942303c259c912c8698592c408a Mon Sep 17 00:00:00 2001 From: Joel Scheuner Date: Thu, 19 Nov 2026 18:04:42 +0100 Subject: [PATCH 05/13] =?UTF-8?q?docs:=20address=20review=20round=202=20(D?= =?UTF-8?q?ependabot=20under=20=C2=A70,=20endpoint=20question,=20payload?= =?UTF-8?q?=20fields,=20deviations=20intro)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 75 ++++++++++++++++++++++++++----------------------------- REVIEW.md | 9 ++++--- 2 files changed, 40 insertions(+), 44 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 83cd378..44e4e8f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -26,7 +26,7 @@ yarn format # Prettier |------|----------| | `src/tools/` | One file per MCP tool. xmcp discovers them: each exports `schema` (zod), `metadata`, and a default handler. | | `src/prompts/` | MCP prompts (workflow templates such as `infrastructure-tester`). | -| `src/core/` | Shared plumbing: `preflight.ts`, `response-builder.ts`, `analytics.ts`, `command-runner.ts`, `localstack-env.ts`, `config.ts`, `http-client.ts`. | +| `src/core/` | Shared plumbing: preflights, the response builder, analytics, the command runner, configuration. | | `src/lib//` | Tool logic. Pure, testable logic lives in `*.logic.ts` with a matching `*.logic.test.ts`. | | `src/lib/docker/` | Docker Engine API client (find, start, stop, exec, logs). | | `src/lib/localstack/` | LocalStack HTTP clients and the container spec. | @@ -34,7 +34,7 @@ yarn format # Prettier | `src/tools-tests/` | Handler tests, one file per tool. They live outside `src/tools/` because xmcp registers every file there as a tool. | | `tests/mcp/` | Playwright tests against the built server. | | `tests/docker/` | Validation of the Docker image. | -| `docs/` | Topic docs (`AZURE.md`, `DOCKER.md`). | +| `docs/` | Topic docs. | ## Writing a tool @@ -92,8 +92,8 @@ Rules for tool tests: `ci.yml` does for fork PRs, so a fork PR never looks tested. Only an opt-in scenario (one that needs its own credentials, such as the replicator's source AWS keys, or a flag such as `HARNESS_RUN_REMOTE`) or an outside service being down (the docs search) may skip a case, and the output says why. -- **Evals are not PR gates.** `test:mcp:evals` depends on a model. `mcp-evals.yml` runs it after merges to `main`, - weekly and on demand. Run them yourself when a tool's description, schema or +- **Evals are not PR gates.** `test:mcp:evals` depends on a model. `mcp-evals.yml` runs it after merges to `main` + that touch its path filter, weekly and on demand. Run them yourself when a tool's description, schema or response wording changes, and report the result in the PR. ## Dependencies @@ -171,16 +171,12 @@ without lstk. That container must be interchangeable with one lstk starts: - **Either tool can take over.** `lstk stop` / `status` / `logs` / `restart` work on an emulator the MCP server started, and the MCP server's `stop` / `status` work on one lstk started. Today this holds only through fallbacks: the server detects `localstack-aws` by name, and lstk, finding no container under its own name, matches a running - LocalStack image that serves the gateway on container port 4566. That interoperability is a known gap. Do not - widen the difference: keep both names detected, keep LocalStack's own images, and keep the gateway on container - port 4566. -- **Same endpoint, for the commands that take one.** `lstk aws`, `az`, `terraform`, `cdk`, `sam`, `status`, - `inspect`, `reset` and `snapshot` (save, load, remove, and list of an `s3://` remote) accept an endpoint. Pass - the server's target (`LOCALSTACK_HOSTNAME` / `LOCALSTACK_PORT`) to them so both talk to the same emulator. The current proposal is - `LSTK_ENDPOINT_URL`, which lstk prefers over `AWS_ENDPOINT_URL`, so a value already in the user's environment - would not win. lstk refuses an endpoint for the local lifecycle commands (`start`, - `stop`, `restart`, `logs`, `volume`), so setting one there makes the command fail. Which commands - get the endpoint is not decided yet: ask the user before wiring lstk calls (see [Open questions](#open-questions)). + LocalStack image that serves the gateway on lstk's default container port. That interoperability is a known gap. + Do not widen the difference: keep both names detected, keep LocalStack's own images, and keep the gateway on that + port. +- **Same emulator.** An lstk command the server runs must reach the emulator the server's other tools use + (`LOCALSTACK_HOSTNAME` / `LOCALSTACK_PORT`). How the server passes that target is an open question: ask the user + before wiring a new lstk call (see [Open questions](#open-questions)). ### 2. Use lstk's cloud and IaC commands @@ -274,21 +270,20 @@ The IAM actions have three jobs. Each reads a LocalStack API; without the featur what is missing instead of returning an empty answer. 1. **Why was a request denied, and what fixes it.** `list-iam-denials`, `explain-iam`, `get-iam-policy`. Source: - App Inspector IAM events (`/_localstack/appinspector/v1/traces/{t}/spans/{s}/events/iam`, `iam.policy.denied` for - denials, also when they were not enforced, with `enforcement: allow_with_warning`). The event's `payload` - attribute is a JSON string holding the decisions (`explicit_allows`, `explicit_denies`, `implicit_denies`, each with `caused_by` - policy keys and statement positions), the `policies` map, and for a denial LocalStack's `suggestion` (version 1: - `outcome`, `entries`, `hints`, `near_misses`). `explain-iam` is to render them in the section order of the App - Inspector UI's "Copy agent prompt", then the payload as sent. Do not recompute, merge or reorder the changes. - A policy stored truncated (`statement_count` without `document`) is fetched on demand with `get-iam-policy` - (`/v1/events/{event_id}/policies/{policy_key}`), never automatically. + App Inspector IAM events (`/_localstack/appinspector/v1/traces/{t}/spans/{s}/events/iam`), which include denials + that were recorded but not enforced. An event's payload holds LocalStack's decisions (which policies allowed or + denied the request, and at which statements), the policies involved, and for a denial a versioned suggestion of + what to change and whether that fixes the request. `explain-iam` is to render them in the section order of the + App Inspector UI's "Copy agent prompt", then the payload as sent. Do not recompute, merge or reorder the changes. + A policy the event stored truncated is fetched on demand with `get-iam-policy` (the event policy route), never + automatically. - Without the `localstack.appinspector/view_iam_policy_errors` entitlement, LocalStack hides IAM events and returns an empty list. Spans say so (`iam_errors_suppressed`) and the policy route answers 403 `not_entitled`, but the event lists don't, so an empty IAM result reads one span of the scope and reports suppression, or else the IAM mode. `/_localstack/licenseinfo` cannot tell: it lists plugins, not entitlements. - - A denial without `suggestion` comes from an older LocalStack: say so and suggest upgrading. A suggestion - `version` other than 1 is shown as JSON only. + - A denial without a suggestion comes from an older LocalStack: say so and suggest upgrading. A suggestion + version the server does not know is shown as JSON only. 2. **Which policies does the workload need.** `get-iam-policy-summary`, `reset-iam-policy-summary`. Source: the IAM policy summary (`GET`/`DELETE /_aws/iam/policies/summary`; a 404 means the feature is not licensed or not enabled). It is global to the emulator: there is no per-trace summary @@ -323,7 +318,7 @@ row once it ships. | Need | Status | Used by | |------|--------|---------| -| Document the extensions API (`GET /_localstack/extensions/list`, `POST …/install`, `POST …/uninstall`) in the OpenAPI spec and the docs | The LocalStack web app's Extension Manager depends on it; neither the spec nor the docs name it. lstk has no extensions commands. | `localstack-extensions` (allowed under the rule 4 exception) | +| Document the extensions API (`GET /_localstack/extensions/list`, `POST …/install`, `POST …/uninstall`) in the OpenAPI spec and the docs | The LocalStack web app's Extension Manager depends on it; neither the spec nor the docs name it. lstk has no commands for LocalStack extensions. | `localstack-extensions` (allowed under the rule 4 exception) | | Document the IAM evaluation event payload (the `payload` attribute of App Inspector's IAM events: the decisions, the `policies` map and the `suggestion`) in the App Inspector spec | The spec types the attribute as a string; its fields are not documented anywhere public. The App Inspector UI depends on them. Until they are documented, describe them in this repository's code as types and tests written from observed responses, not from LocalStack's code; this file names them only as far as the design needs. | `explain-iam`, planned (allowed under the rule 4 exception) | | Document the IAM policy summary and stream (`GET`/`DELETE /_aws/iam/policies/summary`, `GET /_aws/iam/policies/stream`) | App Inspector's Policies view and the web app's IAM Policy Stream depend on them; the spec documents only `/_aws/iam/config` and `/_aws/iam/check-actions-allowed`. | `localstack-app-inspector` IAM job 2, planned (allowed under the rule 4 exception) | | Filter the IAM policy summary by trace (or time range), so a run's policies don't mix with earlier requests | Not implemented: the summary is global and only resettable. | `get-iam-policy-summary` (planned) | @@ -336,25 +331,25 @@ row once it ships. ## Known deviations -Existing code that does not follow the rules yet. This list is for code that is already on `main`; it is not a way -to let new code through. +Code on `main` that does not follow the rules: deviations that predate them, and exceptions the user approved. A +row is never a way to let unapproved code through. -- **New code follows the rules.** A change that needs to break one raises it with the user before it is written, - and its PR records the decision. Only then does it get a row here. +- **New code follows the rules.** A change that needs to break one raises it with the user before it is written. + If the user approves, the PR records the approval and adds a row naming who approved it and when. - **A change that touches a listed deviation raises it.** Say in the PR whether the change fixes the deviation, leaves it as it is, or why it cannot fix it yet. Do not copy the pattern and do not extend it to another tool. - **Remove a row when the code is fixed.** -Approved exceptions to rule 5 are listed here too, with who approved them (by role or GitHub handle) and when; they -stay until the LocalStack API that replaces them ships. +An approved exception stays until what replaces it ships. One approved for code not on `main` yet is recorded +with its follow-up until it lands. | Where | Deviation | Target | |-------|-----------|--------| -| `src/lib/localstack/container-spec.logic.ts`, `src/lib/docker/docker.client.ts` | The header comment describes the container as replicating the legacy `localstack start` (rule 1). The container spec differs from lstk's in several ways, among them the name (`localstack-main` for every emulator), labels, volume directory, ports and environment, so emulator state does not carry over and only AWS containers are found in both directions. Compare with lstk's `internal/config/containers.go` and `internal/container/start.go` for the current differences. Interoperability rests on the server detecting lstk's AWS container name and on lstk's fallback, which matches a known LocalStack image serving the gateway on container port 4566; it breaks for other images, a gateway moved off 4566 inside the container, and the other emulators. | Converge on lstk's container spec; decide later whether to adopt its names. Until then, keep lstk's AWS name detected (rule 1). | +| `src/lib/localstack/container-spec.logic.ts`, `src/lib/docker/docker.client.ts` | The header comment describes the container as replicating the legacy `localstack start` (rule 1). The container spec differs from lstk's in several ways, among them the name (`localstack-main` for every emulator), labels, volume directory, ports and environment, so emulator state does not carry over and only AWS containers are found in both directions. Compare with lstk's `internal/config/containers.go` and `internal/container/start.go` for the current differences. Interoperability rests on the server detecting lstk's AWS container name and on lstk's fallback, which matches a known LocalStack image serving the gateway on lstk's default container port; it breaks for other images, a gateway moved off that port inside the container, and the other emulators. | Converge on lstk's container spec; decide later whether to adopt its names. Until then, keep lstk's AWS name detected (rule 1). | | `src/tools/localstack-aws-client.ts` | Runs `awslocal` inside the container via `docker exec`. | `lstk aws`, no fallback (rule 2). | | `src/tools/localstack-deployer.ts`, `src/prompts/infrastructure-tester.ts` | The deployer runs `awslocal cloudformation` through `docker exec` for CloudFormation deploys and deletes, and the prompt tells the agent to use `awslocal`. | `lstk aws cloudformation` (rule 2); the prompt names lstk. | | `src/tools/localstack-chaos-injector.ts`, `src/tools/localstack-ephemeral-instances.ts`, `src/tools/localstack-aws-replicator.ts` | Tools that skip parts of "Writing a tool": hand-built `❌` responses instead of `ResponseBuilder` (`withToolAnalytics` flags them as errors only by their prefix), a handler without `runPreflights`, and ARN parsing in the tool file (rule 5). | Follow "Writing a tool". | -| `src/core/localstack-env.ts` | Passes the server's target to `lstk terraform` / `cdk` / `sam` as `AWS_ENDPOINT_URL`, but keeps an `AWS_ENDPOINT_URL` already in the user's environment, and a `LSTK_ENDPOINT_URL` there overrides both: either sends the command to a different emulator. | Depends on the endpoint-scoping [open question](#open-questions). | +| `src/core/localstack-env.ts` | Passes the server's target to `lstk terraform` / `cdk` / `sam` as `AWS_ENDPOINT_URL`, but keeps an `AWS_ENDPOINT_URL` already in the user's environment, and a `LSTK_ENDPOINT_URL` there overrides both: either sends the command to a different emulator. | Depends on the [open question](#open-questions) on passing the target to lstk. | | `src/tools/localstack-azure-client.ts` | Configures the host `az` itself. | `lstk az`, keeping the existing guards (rule 2). | | `Dockerfile` | Bundles lstk, Terraform, CDK and SAM, but not the AWS CLI. | Bundle the AWS CLI for `lstk aws` (rule 2). This is a larger image change: ask the user first ("Docker image size"). | | `src/core/analytics.ts` | Sends data that can hold secrets (REVIEW §0): free-form arguments such as the cloud CLIs' `command` (for example `--secret-string …`), and error text. `looksSensitiveValue` only catches a value that is a whole key or token. The allowlist also sends arguments that hold user data: names, paths, queries and filters. | Only fields that can never hold secrets or user data ("Writing a tool"): for example the command's service and operation instead of the command, an error code instead of the message. | @@ -378,13 +373,13 @@ stay until the LocalStack API that replaces them ships. Decisions not settled yet. Ask the user before acting on one, and move the answer into the rules once it is made. -- **Endpoint scoping for lstk calls (rule 1).** Rule 1 passes the server's endpoint as `LSTK_ENDPOINT_URL` only to - the lstk commands that accept one (`aws`, `az`, `terraform`, `cdk`, `sam`, `status`, `inspect`, `reset`, - `snapshot`), and never sets one for `start`, `stop`, `restart`, `logs` or `volume`, because lstk refuses those when `--endpoint-url`, `LSTK_ENDPOINT_URL` or `AWS_ENDPOINT_URL` - is set. Open: - - Is this scoping the intended contract? - - If the server delegates a lifecycle command to lstk, must it remove all three variables from the child's - environment, so an endpoint inherited from the user's shell does not make lstk refuse the command? +- **How the server passes its target to lstk (rule 1).** The proposal: pass it as `LSTK_ENDPOINT_URL`, which lstk + prefers over `AWS_ENDPOINT_URL` (so a value in the user's environment does not win), to the commands that accept + an endpoint, and to none of the commands lstk's `rejectEndpointURL` (`cmd/endpoint.go`) refuses one for. Today + the deployer passes `AWS_ENDPOINT_URL` (`src/core/localstack-env.ts`). Open: + - Is this the intended contract? + - If the server runs a command lstk refuses an endpoint for, must it also remove the endpoint variables it + inherited from the user's shell? ## Follow-ups diff --git a/REVIEW.md b/REVIEW.md index 9c7ed49..be06a9e 100644 --- a/REVIEW.md +++ b/REVIEW.md @@ -43,8 +43,9 @@ Each of these is a BLOCKER: wrapped CLI or the LocalStack container inherits the whole environment when it needs one variable, or can read a file that holds a secret. Or a step installs or runs third-party code (dependency install scripts, a sample project's packages, a third-party action) while a secret is in its environment or in `.git/config` - (`actions/checkout` keeps the GitHub token there unless `persist-credentials: false`). A secret set at job level - where only first-party code runs is a SHOULD-FIX. + (`actions/checkout` keeps the GitHub token there unless `persist-credentials: false`). Where AGENTS.md already + lists a workflow under Known deviations for this, a change that only bumps a dependency or action is not a new + finding; one that adds a step, job or dependency source there is. - **Write access by default.** A new workflow, or a new job or trigger in an existing one, runs without a `permissions:` block, or a job that runs PR code gets more than it needs (`contents: write`, `packages: write`, `id-token: write`). A version bump of an action in a listed workflow is not such a change. @@ -75,8 +76,8 @@ artifact, a pushed commit), say so in the review so a maintainer rotates it; del questions for the user, not as rule violations, unless the failure is confirmed: - an lstk call can reach a different emulator than the server's other tools, for example because a `LSTK_ENDPOINT_URL` or `AWS_ENDPOINT_URL` in the user's environment wins (SHOULD-FIX if confirmed) - - an endpoint reaches `lstk start`, `stop`, `restart`, `logs` or `volume`, through the server or the inherited - environment; lstk refuses these commands when one is set (BLOCKER if confirmed: the command fails) + - an endpoint reaches an lstk command that refuses one (`rejectEndpointURL` in lstk's `cmd/endpoint.go`), + through the server or the inherited environment (BLOCKER if confirmed: the command fails) ## 2. Cloud and IaC commands From 5c8327904377f05ac09f3461ec199d8c92a74b01 Mon Sep 17 00:00:00 2001 From: Joel Scheuner Date: Thu, 19 Nov 2026 18:08:49 +0100 Subject: [PATCH 06/13] docs: address review round 3 (unescaped rendering, API detection, exceptions home) Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 17 ++++++++++------- REVIEW.md | 6 +++--- 2 files changed, 13 insertions(+), 10 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 44e4e8f..9a3dc0a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -61,7 +61,8 @@ the failing test first when fixing a bug. | Logic | `src/lib//*.logic.test.ts` | always | Parsing, validation, rendering and sanitizing, as pure functions. | | Handler | `src/tools-tests/.test.ts` | always | The tool's actions end to end inside the process, with Docker, LocalStack and spawned binaries mocked at the module boundary. No existing handler test follows every rule below yet; do not copy their mocks of internal modules. | | Tools | `tests/mcp/tools.spec.mjs` | every PR, with the auth token | Each tool over MCP stdio against a real emulator, without a model. Each case reads the state the tool left behind back through another call. | -| Image | `tests/mcp/direct.spec.mjs`, `tests/docker/validate-image.mjs` | every PR (`docker.yml` when its path filter matches; its integration test on amd64 only), with the auth token | The built server and the Docker image: tool list, prompts, setup wizard, and real calls through the image. | +| Server | `tests/mcp/direct.spec.mjs` | every PR, with the auth token | The built server: tool list, prompts, setup wizard. | +| Image | `tests/docker/validate-image.mjs` | `docker.yml`, with the auth token | Real calls through the Docker image. | Rules for tool tests: @@ -113,7 +114,7 @@ arrive as PRs instead of waiting for someone to notice: (except the LocalStack emulator images, which track the newest release on purpose), or `npx ` for a package that is not in `package.json`, in the Dockerfile, workflows or scripts. A new tool gets a lockfile in one of the places above. -- **New ecosystem or directory, new Dependabot entry.** Keep the 7-day cooldown and the minor/patch grouping. +- **New ecosystem or directory, new Dependabot entry.** Copy the cooldown and grouping of the existing entries. - **CI uses the image's versions.** Jobs that run tools on the host install them through `.github/actions/host-tools`, from the same files the image uses. `src/core/dependency-pins.test.ts` checks that the jobs running the tool tests or the evals use it, and that `ci.yml`, `docker.yml` and `mcp-evals.yml` run on @@ -227,8 +228,8 @@ public docs or in one of the OpenAPI specs a running emulator serves: Inspector Ask the user before adopting an API this server does not use yet, even a documented one, and name the endpoint and -what the tool would do with it. The APIs the server uses today are the ones its clients call -(`src/lib/localstack/*.client.ts`, `src/core/config.ts`); an endpoint missing there is new. The IAM design below +what the tool would do with it. An endpoint is new when `git grep -E '/_(localstack|aws)/' src` does not +find it. The IAM design below would add the event policy route and the IAM policy summary; adopting them still needs the user's approval, recorded in the PR that adds them. @@ -251,7 +252,8 @@ If two clients could compute the same answer differently, the answer belongs in or user gets, and LocalStack cannot offer it yet, it may be done as a documented exception, but only with the user's explicit approval. This is not a default way out of the rule: first propose the LocalStack change, then ask the user, naming the computation, what it improves, and what it would take in LocalStack. Never assume - approval. An approved exception gets a row in [Known deviations](#known-deviations), saying who approved it and + approval. An approved exception gets a row in [Known deviations](#known-deviations) (under [Follow-ups](#follow-ups) + while its code is not on `main`), saying who approved it and when, and a [Gaps in LocalStack](#gaps-in-localstack) row for the API that would replace it; delete both once that API ships. Exceptions never cover decisions about IAM (which policy denied, what fixes it), and never read logs (rule 3). @@ -324,7 +326,7 @@ row once it ships. | Filter the IAM policy summary by trace (or time range), so a run's policies don't mix with earlier requests | Not implemented: the summary is global and only resettable. | `get-iam-policy-summary` (planned) | | Say on the App Inspector events page when IAM events are hidden by the license (as spans do with `iam_errors_suppressed`) | Not implemented: the events routes return an empty list, so a tool has to probe one span. | `list-iam-denials`, `explain-iam` (planned) | | Compare suggested policies with deployed ones: permissions to add and permissions granted but unused, per ARN | Not implemented. Listed as a follow-up in localstack-appinspector-ui#341. | IAM job 3 | -| Document App Inspector's status route (`GET`/`PUT /_localstack/appinspector/status`) in its OpenAPI spec | Not in `/_localstack/appinspector/v1/openapi.yaml`. lstk's `inspect on` / `off` uses it too. | `get-status`, `set-status` (allowed under the rule 4 exception) | +| Document App Inspector's status route (`GET`/`PUT /_localstack/appinspector/status`) in its OpenAPI spec | Not in `/_localstack/appinspector/v1/openapi.yaml`. lstk's `inspect` command uses it too. | `get-status`, `set-status` (allowed under the rule 4 exception) | | Presentation fields for IAM: a display name, the services and the action count per policy summary entry; the principal kind (ARN, service, account) on suggestion hints; the statements a decision names, resolved | Not implemented. Until it ships, show the ARN, the type and the policy JSON, name statement positions without resolving them, and word hints for any kind of caller. | `explain-iam`, `get-iam-policy-summary` (planned) | | Document `GET /_localstack/licenseinfo`, or name the documented way to tell whether a licensed feature is available | In none of the emulator's specs and not in the docs. Whether a LocalStack client depends on it is unverified: lstk and the App Inspector UI don't. | `requireProFeature` in every Pro tool. The rule 4 exception does not cover it, since no LocalStack client is known to depend on it; see [Known deviations](#known-deviations). | | A failed-requests summary in App Inspector: failed spans grouped by service, operation, HTTP status and AWS error code, with counts, first and last time, and an example trace and span | Not implemented. | `lstk mcp`'s `localstack-app-inspector` `failed-requests`, planned (an approved exception, see [Follow-ups](#follow-ups)) | @@ -341,7 +343,7 @@ row is never a way to let unapproved code through. - **Remove a row when the code is fixed.** An approved exception stays until what replaces it ships. One approved for code not on `main` yet is recorded -with its follow-up until it lands. +under [Follow-ups](#follow-ups) until it lands. | Where | Deviation | Target | |-------|-----------|--------| @@ -358,6 +360,7 @@ with its follow-up until it lands. | `src/tools/localstack-iam-policy-analyzer.ts` | Finds IAM denials by parsing container logs (rule 3). | The IAM actions of `localstack-app-inspector` ("IAM policies"); then deprecate the analyzer and remove it after a notice period. | | `.github/workflows/azure-smoke.yml`, `ci.yml` (`mcp-direct-tests`), `tests/mcp/azure-offline.spec.mjs` | `azure-smoke.yml` runs on `ubuntu-latest`; it and `mcp-direct-tests` use the runner image's Azure CLI, whose version no file here pins. `azure-offline` skips its cases when `az` is missing, a skip "Testing" does not allow. | The pinned Ubuntu of the other workflows, and the Azure CLI from a pinned file, as `host-tools` does for Terraform, uv and lstk. | | `.github/workflows/` | Secrets reach third-party code in CI (REVIEW §0): jobs set `LOCALSTACK_AUTH_TOKEN` (and, in `mcp-evals.yml`, the OpenRouter key) at job level, so `yarn install` runs dependency install scripts and third-party actions run with them; `docker.yml` runs `npm ci` for `data/sample-cdk`, with install scripts, in a step that holds the token; and `actions/checkout` leaves the GitHub token in `.git/config` for later steps. | Install dependencies with no secret in the environment (or with `--ignore-scripts`), set secrets only on the steps that need them, and check out with `persist-credentials: false`. | +| `src/tools/localstack-app-inspector.ts`, `src/lib/deployment/` | Render LocalStack and workload strings (span and event names, stack outputs and descriptions) into markdown tables and text unescaped (REVIEW §7). | Escape them, and size code fences to their content (REVIEW §7). | | `.github/workflows/docker.yml` | Its path filter misses files the image build reads (`patches/`, `xmcp.config.ts`, `tsconfig*.json`), so a change to them is not built or tested as an image. | Filter on everything the `Dockerfile` copies. | | `src/lib/localstack/license-checker.ts` | Calls `GET /_localstack/licenseinfo`, which is not documented and not covered by the rule 4 exception. | The documented way LocalStack names (see [Gaps in LocalStack](#gaps-in-localstack)). | | `data/sample-terraform/` | The AWS provider has only a version constraint, no lockfile, and no Dependabot entry; CI downloads it with secrets in the job's environment. | A committed `.terraform.lock.hcl` and a `terraform` Dependabot entry ("Dependencies"). | diff --git a/REVIEW.md b/REVIEW.md index be06a9e..8aa2e5b 100644 --- a/REVIEW.md +++ b/REVIEW.md @@ -32,9 +32,9 @@ Each of these is a BLOCKER: - **Secrets for Dependabot.** Dependabot PRs get no Actions secrets either, so token-gated jobs fail on them. Do not add `LOCALSTACK_AUTH_TOKEN` or another secret as a Dependabot secret: it would reach the install scripts of the packages the PR just bumped. Review the bump, then run it with secrets the same way as an external PR. -- **An external PR pushed unreviewed.** For a fork or Dependabot PR to be tested, a maintainer pushes its branch to this - repository and opens a PR from it, so its workflows run with secrets. Review the whole diff under this section before pushing it, - and push the reviewed commit, not a branch the contributor can still change. +- **An external PR pushed unreviewed.** For a fork or Dependabot PR to be tested, a maintainer pushes the reviewed commit to a + new branch the maintainer owns and opens a PR from it, so its workflows run with secrets. Review the whole diff + under this section first, and never push a branch the contributor or Dependabot can still change. - **Secrets printed or stored.** A step or script echoes a secret, runs `set -x`, `env` or `printenv`, logs request headers, or writes a secret to a file. GitHub masks secrets in logs only: an uploaded artifact (test results, Playwright traces, eval transcripts, the server's stderr), a cache entry, or a built image layer is not masked. From 65b4ec40436f33ce8bf688653330e1b19d108563 Mon Sep 17 00:00:00 2001 From: Joel Scheuner Date: Thu, 19 Nov 2026 18:13:13 +0100 Subject: [PATCH 07/13] docs: address review round 4 (private link, existing IAM actions, dependency sources) Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 24 ++++++++++++------------ REVIEW.md | 3 ++- 2 files changed, 14 insertions(+), 13 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 9a3dc0a..fc44aab 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -59,8 +59,8 @@ the failing test first when fixing a bug. | Level | Where | Runs in CI | What it proves | |-------|-------|------------|----------------| | Logic | `src/lib//*.logic.test.ts` | always | Parsing, validation, rendering and sanitizing, as pure functions. | -| Handler | `src/tools-tests/.test.ts` | always | The tool's actions end to end inside the process, with Docker, LocalStack and spawned binaries mocked at the module boundary. No existing handler test follows every rule below yet; do not copy their mocks of internal modules. | -| Tools | `tests/mcp/tools.spec.mjs` | every PR, with the auth token | Each tool over MCP stdio against a real emulator, without a model. Each case reads the state the tool left behind back through another call. | +| Handler | `src/tools-tests/.test.ts` | always | The tool's actions end to end inside the process, with Docker, LocalStack and spawned binaries mocked at the module boundary. | +| Tools | `tests/mcp/tools.spec.mjs` | every PR, with the auth token | Each tool over MCP stdio against a real emulator, without a model. A case for an action that changes state reads that state back through another call. | | Server | `tests/mcp/direct.spec.mjs` | every PR, with the auth token | The built server: tool list, prompts, setup wizard. | | Image | `tests/docker/validate-image.mjs` | `docker.yml`, with the auth token | Real calls through the Docker image. | @@ -82,8 +82,8 @@ Rules for tool tests: in neither the response, the error, nor the analytics payload. - **Real-emulator coverage.** A new tool goes into `EXPECTED_TOOLS` in `tests/mcp/direct.spec.mjs` and in `tests/docker/validate-image.mjs`. A new action - gets a case in `tests/mcp/tools.spec.mjs` that checks its effect through another call, not just its success - message, so a tool that skips its API call but reports success fails. Add a case to + gets a case in `tests/mcp/tools.spec.mjs`; one that changes state checks its effect through another call, not + just its success message, so a tool that skips its API call but reports success fails. Add a case to `tests/docker/validate-image.mjs` when the action depends on the image (bundled binaries, mounted paths). - **Local by default.** No real AWS or Azure resources. A test that creates something on the LocalStack platform (a Cloud Pod) deletes it in a `finally` and uses a name unique to the run, as `tools.spec.mjs` does, or a fixed @@ -228,8 +228,7 @@ public docs or in one of the OpenAPI specs a running emulator serves: Inspector Ask the user before adopting an API this server does not use yet, even a documented one, and name the endpoint and -what the tool would do with it. An endpoint is new when `git grep -E '/_(localstack|aws)/' src` does not -find it. The IAM design below +what the tool would do with it. The IAM design below would add the event policy route and the IAM policy summary; adopting them still needs the user's approval, recorded in the PR that adds them. @@ -264,7 +263,8 @@ If two clients could compute the same answer differently, the answer belongs in Today `localstack-iam-policy-analyzer` handles IAM by parsing logs and synthesizing policies (known deviations), with its `set-mode` and `get-status` actions for the enforcement mode. IAM moves to `localstack-app-inspector`. None -of the actions below exist yet; this is the design they are built to. Once its actions ship, the analyzer is +of the actions below exist yet; this is the design they are built to. `localstack-app-inspector` already lists a +span's raw IAM events (`list-iam-events`, `get-event`); `list-iam-denials` and `explain-iam` replace those for IAM. Once its actions ship, the analyzer is deprecated: its behaviour stays unchanged until it is removed, but its description and every response name the app-inspector action that replaces it. @@ -291,8 +291,8 @@ what is missing instead of returning an empty answer. enabled). It is global to the emulator: there is no per-trace summary (see [Gaps in LocalStack](#gaps-in-localstack)). Each entry is one policy for one ARN: `resource`, `policy_type` (`identity` or `resource`) and `policy_document`. Present it the way App Inspector's Policies view - does ([localstack-appinspector-ui#341](https://github.com/localstack/localstack-appinspector-ui/pull/341)): - - one entry per ARN, with its type LocalStack does not send a display name, services or action count yet (see + does: + - one entry per ARN, with its type; LocalStack does not send a display name, services or action count yet (see [Gaps in LocalStack](#gaps-in-localstack)); do not derive them from the ARN or the policy - the policy JSON exactly as sent: a role's trust statement and its permissions stay in one policy - always labelled as a suggestion, derived from the requests seen since the last reset, not as a verified policy @@ -325,7 +325,7 @@ row once it ships. | Document the IAM policy summary and stream (`GET`/`DELETE /_aws/iam/policies/summary`, `GET /_aws/iam/policies/stream`) | App Inspector's Policies view and the web app's IAM Policy Stream depend on them; the spec documents only `/_aws/iam/config` and `/_aws/iam/check-actions-allowed`. | `localstack-app-inspector` IAM job 2, planned (allowed under the rule 4 exception) | | Filter the IAM policy summary by trace (or time range), so a run's policies don't mix with earlier requests | Not implemented: the summary is global and only resettable. | `get-iam-policy-summary` (planned) | | Say on the App Inspector events page when IAM events are hidden by the license (as spans do with `iam_errors_suppressed`) | Not implemented: the events routes return an empty list, so a tool has to probe one span. | `list-iam-denials`, `explain-iam` (planned) | -| Compare suggested policies with deployed ones: permissions to add and permissions granted but unused, per ARN | Not implemented. Listed as a follow-up in localstack-appinspector-ui#341. | IAM job 3 | +| Compare suggested policies with deployed ones: permissions to add and permissions granted but unused, per ARN | Not implemented. | IAM job 3 | | Document App Inspector's status route (`GET`/`PUT /_localstack/appinspector/status`) in its OpenAPI spec | Not in `/_localstack/appinspector/v1/openapi.yaml`. lstk's `inspect` command uses it too. | `get-status`, `set-status` (allowed under the rule 4 exception) | | Presentation fields for IAM: a display name, the services and the action count per policy summary entry; the principal kind (ARN, service, account) on suggestion hints; the statements a decision names, resolved | Not implemented. Until it ships, show the ARN, the type and the policy JSON, name statement positions without resolving them, and word hints for any kind of caller. | `explain-iam`, `get-iam-policy-summary` (planned) | | Document `GET /_localstack/licenseinfo`, or name the documented way to tell whether a licensed feature is available | In none of the emulator's specs and not in the docs. Whether a LocalStack client depends on it is unverified: lstk and the App Inspector UI don't. | `requireProFeature` in every Pro tool. The rule 4 exception does not cover it, since no LocalStack client is known to depend on it; see [Known deviations](#known-deviations). | @@ -360,7 +360,7 @@ under [Follow-ups](#follow-ups) until it lands. | `src/tools/localstack-iam-policy-analyzer.ts` | Finds IAM denials by parsing container logs (rule 3). | The IAM actions of `localstack-app-inspector` ("IAM policies"); then deprecate the analyzer and remove it after a notice period. | | `.github/workflows/azure-smoke.yml`, `ci.yml` (`mcp-direct-tests`), `tests/mcp/azure-offline.spec.mjs` | `azure-smoke.yml` runs on `ubuntu-latest`; it and `mcp-direct-tests` use the runner image's Azure CLI, whose version no file here pins. `azure-offline` skips its cases when `az` is missing, a skip "Testing" does not allow. | The pinned Ubuntu of the other workflows, and the Azure CLI from a pinned file, as `host-tools` does for Terraform, uv and lstk. | | `.github/workflows/` | Secrets reach third-party code in CI (REVIEW §0): jobs set `LOCALSTACK_AUTH_TOKEN` (and, in `mcp-evals.yml`, the OpenRouter key) at job level, so `yarn install` runs dependency install scripts and third-party actions run with them; `docker.yml` runs `npm ci` for `data/sample-cdk`, with install scripts, in a step that holds the token; and `actions/checkout` leaves the GitHub token in `.git/config` for later steps. | Install dependencies with no secret in the environment (or with `--ignore-scripts`), set secrets only on the steps that need them, and check out with `persist-credentials: false`. | -| `src/tools/localstack-app-inspector.ts`, `src/lib/deployment/` | Render LocalStack and workload strings (span and event names, stack outputs and descriptions) into markdown tables and text unescaped (REVIEW §7). | Escape them, and size code fences to their content (REVIEW §7). | +| `src/tools/localstack-app-inspector.ts`, `src/lib/deployment/` | Render LocalStack and workload strings (span and event names, stack outputs and descriptions) into markdown tables and text unescaped (REVIEW §7). `list-iam-events` answers "No events found." when the license hides IAM events, instead of explaining it ("IAM policies"). | Escape them, and size code fences to their content (REVIEW §7); explain hidden IAM events. | | `.github/workflows/docker.yml` | Its path filter misses files the image build reads (`patches/`, `xmcp.config.ts`, `tsconfig*.json`), so a change to them is not built or tested as an image. | Filter on everything the `Dockerfile` copies. | | `src/lib/localstack/license-checker.ts` | Calls `GET /_localstack/licenseinfo`, which is not documented and not covered by the rule 4 exception. | The documented way LocalStack names (see [Gaps in LocalStack](#gaps-in-localstack)). | | `data/sample-terraform/` | The AWS provider has only a version constraint, no lockfile, and no Dependabot entry; CI downloads it with secrets in the job's environment. | A committed `.terraform.lock.hcl` and a `terraform` Dependabot entry ("Dependencies"). | @@ -399,7 +399,7 @@ Planned work that is not on `main` yet. Ask the user before starting one. latest example. That computation is an approved exception to rule 5: approved by the project maintainer on 2026-11-17, because it replaces the log-based error summary of `localstack-logs-analysis` with one from App Inspector's data. It is recorded here because the code is not on lstk's `main` yet; when it - lands, record it where lstk's MCP server documents its design. It goes away once the failed-requests summary in + lands, it moves to [Known deviations](#known-deviations). It goes away once the failed-requests summary in [Gaps in LocalStack](#gaps-in-localstack) ships. ## Maintaining this file diff --git a/REVIEW.md b/REVIEW.md index 8aa2e5b..61456f5 100644 --- a/REVIEW.md +++ b/REVIEW.md @@ -45,7 +45,8 @@ Each of these is a BLOCKER: project's packages, a third-party action) while a secret is in its environment or in `.git/config` (`actions/checkout` keeps the GitHub token there unless `persist-credentials: false`). Where AGENTS.md already lists a workflow under Known deviations for this, a change that only bumps a dependency or action is not a new - finding; one that adds a step, job or dependency source there is. + finding; one that adds a step, a job, or a new manifest, lockfile or registry there is. A new package in an + existing manifest is checked under "New third-party action or package from an unknown publisher" instead. - **Write access by default.** A new workflow, or a new job or trigger in an existing one, runs without a `permissions:` block, or a job that runs PR code gets more than it needs (`contents: write`, `packages: write`, `id-token: write`). A version bump of an action in a listed workflow is not such a change. From b998d255c945116f10879ed061c9f7a66725808d Mon Sep 17 00:00:00 2001 From: Joel Scheuner Date: Thu, 19 Nov 2026 18:17:50 +0100 Subject: [PATCH 08/13] docs: address review round 5 (shared helpers, Debian packages, deviations by pattern) Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 15 ++++++++------- REVIEW.md | 8 +++++--- 2 files changed, 13 insertions(+), 10 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index fc44aab..d36b350 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -113,7 +113,8 @@ arrive as PRs instead of waiting for someone to notice: - **No version outside these files.** No `npm install -g `, `pip install `, `curl … | sh`, `latest` tags (except the LocalStack emulator images, which track the newest release on purpose), or `npx ` for a package that is not in `package.json`, in the Dockerfile, workflows or scripts. A new tool gets a lockfile in one of the - places above. + places above. Debian packages are the exception: they come unversioned from the base image's release, which its + pinned `FROM` line fixes. - **New ecosystem or directory, new Dependabot entry.** Copy the cooldown and grouping of the existing entries. - **CI uses the image's versions.** Jobs that run tools on the host install them through `.github/actions/host-tools`, from the same files the image uses. `src/core/dependency-pins.test.ts` checks that @@ -192,7 +193,7 @@ lstk wraps the cloud CLIs and IaC tools - **Say what ran.** The tool response names the command that ran (`lstk aws …`), so a user can reproduce it. - **Keep the guards.** Moving a tool to lstk does not remove its safety checks: the AWS CLI sanitizer, and the Azure tool's profile isolation, egress guard and command policy (`src/lib/azure/`). -- **Docker image.** The image bundles lstk and everything its commands call (the AWS CLI, Terraform, CDK, SAM, and +- **Docker image.** The image must bundle lstk and everything its commands call (the AWS CLI, Terraform, CDK, SAM, and the Azure CLI once the Azure tool moves). - **Snowflake is exempt.** The Snowflake tool uses the `snow` CLI until lstk wraps it. @@ -303,7 +304,7 @@ what is missing instead of returning an empty answer. The enforcement mode is to get separate actions (`get-iam-mode`, `set-iam-mode`, through `/_aws/iam/config`). Enforcement is not needed to record denials. With App Inspector enabled and enforcement off, `/_aws/iam/config` reports -`ENGINE_ONLY` (a documented state), App Inspector records denials with `enforcement: allow_with_warning`, and the +`ENGINE_ONLY` (a documented state), App Inspector records denials without blocking them, and the policy summary grows, but nothing is blocked. `ENFORCED` only makes denials fail the request; `SOFT_MODE` keeps them passing. Never tell a user to switch on enforcement just to see denials, since `ENFORCED` can break a workload that worked. @@ -333,8 +334,8 @@ row once it ships. ## Known deviations -Code on `main` that does not follow the rules: deviations that predate them, and exceptions the user approved. A -row is never a way to let unapproved code through. +Code on `main` (of the repository that holds it) that does not follow the rules: deviations that predate them, and +exceptions the user approved. A row is never a way to let unapproved code through. - **New code follows the rules.** A change that needs to break one raises it with the user before it is written. If the user approves, the PR records the approval and adds a row naming who approved it and when. @@ -350,7 +351,7 @@ under [Follow-ups](#follow-ups) until it lands. | `src/lib/localstack/container-spec.logic.ts`, `src/lib/docker/docker.client.ts` | The header comment describes the container as replicating the legacy `localstack start` (rule 1). The container spec differs from lstk's in several ways, among them the name (`localstack-main` for every emulator), labels, volume directory, ports and environment, so emulator state does not carry over and only AWS containers are found in both directions. Compare with lstk's `internal/config/containers.go` and `internal/container/start.go` for the current differences. Interoperability rests on the server detecting lstk's AWS container name and on lstk's fallback, which matches a known LocalStack image serving the gateway on lstk's default container port; it breaks for other images, a gateway moved off that port inside the container, and the other emulators. | Converge on lstk's container spec; decide later whether to adopt its names. Until then, keep lstk's AWS name detected (rule 1). | | `src/tools/localstack-aws-client.ts` | Runs `awslocal` inside the container via `docker exec`. | `lstk aws`, no fallback (rule 2). | | `src/tools/localstack-deployer.ts`, `src/prompts/infrastructure-tester.ts` | The deployer runs `awslocal cloudformation` through `docker exec` for CloudFormation deploys and deletes, and the prompt tells the agent to use `awslocal`. | `lstk aws cloudformation` (rule 2); the prompt names lstk. | -| `src/tools/localstack-chaos-injector.ts`, `src/tools/localstack-ephemeral-instances.ts`, `src/tools/localstack-aws-replicator.ts` | Tools that skip parts of "Writing a tool": hand-built `❌` responses instead of `ResponseBuilder` (`withToolAnalytics` flags them as errors only by their prefix), a handler without `runPreflights`, and ARN parsing in the tool file (rule 5). | Follow "Writing a tool". | +| `src/tools/` | Many tools skip parts of "Writing a tool": hand-built `❌` responses instead of `ResponseBuilder` (`withToolAnalytics` flags them as errors only by their prefix), handlers without `runPreflights`, and parsing (such as ARN splitting) in the tool file instead of a `*.logic.ts` module (rule 5). | Follow "Writing a tool". | | `src/core/localstack-env.ts` | Passes the server's target to `lstk terraform` / `cdk` / `sam` as `AWS_ENDPOINT_URL`, but keeps an `AWS_ENDPOINT_URL` already in the user's environment, and a `LSTK_ENDPOINT_URL` there overrides both: either sends the command to a different emulator. | Depends on the [open question](#open-questions) on passing the target to lstk. | | `src/tools/localstack-azure-client.ts` | Configures the host `az` itself. | `lstk az`, keeping the existing guards (rule 2). | | `Dockerfile` | Bundles lstk, Terraform, CDK and SAM, but not the AWS CLI. | Bundle the AWS CLI for `lstk aws` (rule 2). This is a larger image change: ask the user first ("Docker image size"). | @@ -360,7 +361,7 @@ under [Follow-ups](#follow-ups) until it lands. | `src/tools/localstack-iam-policy-analyzer.ts` | Finds IAM denials by parsing container logs (rule 3). | The IAM actions of `localstack-app-inspector` ("IAM policies"); then deprecate the analyzer and remove it after a notice period. | | `.github/workflows/azure-smoke.yml`, `ci.yml` (`mcp-direct-tests`), `tests/mcp/azure-offline.spec.mjs` | `azure-smoke.yml` runs on `ubuntu-latest`; it and `mcp-direct-tests` use the runner image's Azure CLI, whose version no file here pins. `azure-offline` skips its cases when `az` is missing, a skip "Testing" does not allow. | The pinned Ubuntu of the other workflows, and the Azure CLI from a pinned file, as `host-tools` does for Terraform, uv and lstk. | | `.github/workflows/` | Secrets reach third-party code in CI (REVIEW §0): jobs set `LOCALSTACK_AUTH_TOKEN` (and, in `mcp-evals.yml`, the OpenRouter key) at job level, so `yarn install` runs dependency install scripts and third-party actions run with them; `docker.yml` runs `npm ci` for `data/sample-cdk`, with install scripts, in a step that holds the token; and `actions/checkout` leaves the GitHub token in `.git/config` for later steps. | Install dependencies with no secret in the environment (or with `--ignore-scripts`), set secrets only on the steps that need them, and check out with `persist-credentials: false`. | -| `src/tools/localstack-app-inspector.ts`, `src/lib/deployment/` | Render LocalStack and workload strings (span and event names, stack outputs and descriptions) into markdown tables and text unescaped (REVIEW §7). `list-iam-events` answers "No events found." when the license hides IAM events, instead of explaining it ("IAM policies"). | Escape them, and size code fences to their content (REVIEW §7); explain hidden IAM events. | +| `src/tools/`, `src/lib/` | Most tools render LocalStack and workload strings (names, outputs, log lines, error text) into markdown tables, text and fixed code fences unescaped (REVIEW §7). `list-iam-events` answers "No events found." when the license hides IAM events, instead of explaining it ("IAM policies"). | Escape them, and size code fences to their content (REVIEW §7); explain hidden IAM events. | | `.github/workflows/docker.yml` | Its path filter misses files the image build reads (`patches/`, `xmcp.config.ts`, `tsconfig*.json`), so a change to them is not built or tested as an image. | Filter on everything the `Dockerfile` copies. | | `src/lib/localstack/license-checker.ts` | Calls `GET /_localstack/licenseinfo`, which is not documented and not covered by the rule 4 exception. | The documented way LocalStack names (see [Gaps in LocalStack](#gaps-in-localstack)). | | `data/sample-terraform/` | The AWS provider has only a version constraint, no lockfile, and no Dependabot entry; CI downloads it with secrets in the job's environment. | A committed `.terraform.lock.hcl` and a `terraform` Dependabot entry ("Dependencies"). | diff --git a/REVIEW.md b/REVIEW.md index 61456f5..f124a7e 100644 --- a/REVIEW.md +++ b/REVIEW.md @@ -9,6 +9,8 @@ state, then the wrong result). Call something a BLOCKER only after confirming it - New code that breaks a rule is a finding at the rule's severity, even if the same pattern is listed. A new row in "Known deviations" excuses it only if the PR records that the user approved the deviation. +- Calling a shared helper whose behaviour is a listed deviation (for example `requireProFeature` or + `withToolAnalytics`) is not new code; copying the pattern or changing the helper is. - A change may touch listed code without fixing it, but the PR must say so: fixed, left as is, or why it cannot be fixed yet (SHOULD-FIX if it does not). It must not extend a deviation to another tool. @@ -106,8 +108,8 @@ This repository is public, so these are BLOCKERs: - a reference to private LocalStack code: Python module paths (`localstack.pro.*`), private file paths or class names, in code, comments, tests or docs - an endpoint that is in neither the public docs nor one of the emulator's OpenAPI specs (`/_localstack/swagger`; - `/_localstack/appinspector/v1/openapi.yaml` for App Inspector), unless it is listed under "Gaps in LocalStack" in - AGENTS.md + `/_localstack/appinspector/v1/openapi.yaml` for App Inspector), unless "Gaps in LocalStack" in AGENTS.md lists it as + allowed under the rule 4 exception - a script or command run inside the LocalStack container to reach internals - an undocumented configuration variable @@ -204,7 +206,7 @@ See "Dependencies" in AGENTS.md. - **BLOCKER:** a dependency or tool installed without a pinned version in a file Dependabot updates: `npm install -g`, `pip install`, `curl … | sh`, a `latest` tag (other than the LocalStack emulator images), or `npx` of a package - not in `package.json`. + not in `package.json`. Debian packages from the base image's release are allowed (AGENTS.md, "Dependencies"). - **BLOCKER:** a known high or critical advisory is silenced (an `audit-ci` allowlist entry, a removed audit step) without the advisory ID, why it does not apply, and a date to revisit. - **SHOULD-FIX:** a new manifest, lockfile, ecosystem or Dockerfile stage without a matching entry in From d20e52f6ad24ac3650a87eb8e29b846250209b24 Mon Sep 17 00:00:00 2001 From: Joel Scheuner Date: Thu, 19 Nov 2026 18:22:24 +0100 Subject: [PATCH 09/13] docs: address review round 6 (scope of install rule, license errors, analytics allowlist) Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 3 ++- REVIEW.md | 21 +++++++++++++-------- 2 files changed, 15 insertions(+), 9 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index d36b350..58605ba 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -70,7 +70,8 @@ Rules for tool tests: `ResponseBuilder.error` with `isError: true`; assert that, not just the text. - **Test the gates.** Each preflight the tool runs gets a case: no auth token, LocalStack not running, lstk missing, feature not licensed. Keep `runPreflights` real and stub only the checks that reach Docker or the license API. - A missing license or feature must read as an explanation (see "IAM policies"), and the test should say so. + A missing license or feature returns an `isError` response that names the feature and how to get it, and the + test checks both. - **Assert on what leaves the process**: the response (text, `isError`), the argument array passed to the spawned binary, and the HTTP method, path, query and body sent to LocalStack. Do not assert on internal calls. - **Schema.** Invalid arguments are rejected by the zod schema; test a missing required field and a wrong enum diff --git a/REVIEW.md b/REVIEW.md index f124a7e..bf923b3 100644 --- a/REVIEW.md +++ b/REVIEW.md @@ -11,7 +11,8 @@ state, then the wrong result). Call something a BLOCKER only after confirming it "Known deviations" excuses it only if the PR records that the user approved the deviation. - Calling a shared helper whose behaviour is a listed deviation (for example `requireProFeature` or `withToolAnalytics`) is not new code; copying the pattern or changing the helper is. -- A change may touch listed code without fixing it, but the PR must say so: fixed, left as is, or why it cannot be +- A change may touch the deviating code itself (not just a file a row names) without fixing it, but the PR must + say so: fixed, left as is, or why it cannot be fixed yet (SHOULD-FIX if it does not). It must not extend a deviation to another tool. ## 0. Security leaks (always BLOCKER) @@ -47,7 +48,8 @@ Each of these is a BLOCKER: project's packages, a third-party action) while a secret is in its environment or in `.git/config` (`actions/checkout` keeps the GitHub token there unless `persist-credentials: false`). Where AGENTS.md already lists a workflow under Known deviations for this, a change that only bumps a dependency or action is not a new - finding; one that adds a step, a job, or a new manifest, lockfile or registry there is. A new package in an + finding; one that adds a job, a step that installs or runs third-party code not run there before, or a new + manifest, lockfile or registry there is. A new package in an existing manifest is checked under "New third-party action or package from an unknown publisher" instead. - **Write access by default.** A new workflow, or a new job or trigger in an existing one, runs without a `permissions:` block, or a job that runs PR code gets more than it needs (`contents: write`, `packages: write`, @@ -74,7 +76,8 @@ artifact, a pushed commit), say so in the review so a maintainer rotates it; del volumes, environment, auth env) or makes an existing one worse, so `lstk stop` or `lstk status` can find or manage fewer of the server's emulators, or the server fewer of lstk's; or the change drops detection of `localstack-aws`. Differences that already exist are not findings; the "Known deviations" list of them is not exhaustive. -- **SHOULD-FIX:** a change to the container spec has no test that compares it with lstk's defaults. +- **SHOULD-FIX:** a change to the container spec does not say how it compares with lstk's container + (`internal/config/containers.go`). - Endpoint handling for lstk calls is an open question in AGENTS.md. Until it is decided, report these as questions for the user, not as rule violations, unless the failure is confirmed: - an lstk call can reach a different emulator than the server's other tools, for example because a @@ -140,8 +143,8 @@ See "IAM policies" in AGENTS.md. requests seen so far; or "unused permissions" is not qualified as unused in this run. - **SHOULD-FIX:** a summary policy is reshaped, for example by splitting a role's trust statement from its permissions. -- **SHOULD-FIX:** a missing license feature (a 404 from the policy summary, or App Inspector unavailable) shows as an - error instead of an explanation of the feature and how to get it. +- **SHOULD-FIX:** a missing license feature (a 404 from the policy summary, or App Inspector unavailable) returns a + bare failure or an empty result instead of an `isError` response that names the feature and how to get it. ## 7. Tool contract @@ -151,8 +154,9 @@ See "IAM policies" in AGENTS.md. log lines) reaches an agent-facing response without escaping. A value must not be able to close a code fence, break a table, or read as an instruction. Size code fences to their content, and escape markdown in table cells and inline text. -- **SHOULD-FIX:** the handler skips `withToolAnalytics`, preflights or `ResponseBuilder`, or a new tool argument is - missing from (or wrongly added to) the analytics allowlist in `src/core/analytics.ts`. +- **SHOULD-FIX:** the handler skips `withToolAnalytics`, preflights or `ResponseBuilder`, or a new tool argument + that can never hold secrets or user data is missing from the analytics allowlist in `src/core/analytics.ts`, or + one that can is added to it. - **SHOULD-FIX:** decision or parsing logic lives in the tool file instead of a tested `*.logic.ts` module. - **SHOULD-FIX:** a tool's `schema` descriptions or `metadata.description` no longer match what the tool does. Agents choose tools from these strings. @@ -204,7 +208,8 @@ See "Testing" in AGENTS.md. See "Dependencies" in AGENTS.md. -- **BLOCKER:** a dependency or tool installed without a pinned version in a file Dependabot updates: `npm install -g`, +- **BLOCKER:** the Dockerfile, a workflow or a script installs a dependency or tool without a pinned version in a + file Dependabot updates: `npm install -g`, `pip install`, `curl … | sh`, a `latest` tag (other than the LocalStack emulator images), or `npx` of a package not in `package.json`. Debian packages from the base image's release are allowed (AGENTS.md, "Dependencies"). - **BLOCKER:** a known high or critical advisory is silenced (an `audit-ci` allowlist entry, a removed audit step) From fadb882db75744e9d075962ff0d6aeef100990d8 Mon Sep 17 00:00:00 2001 From: Joel Scheuner Date: Thu, 19 Nov 2026 18:24:49 +0100 Subject: [PATCH 10/13] docs: address review round 7 (scope bug-fix tests to src/, hand-managed pins) Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 17 +++++++++-------- REVIEW.md | 7 ++++--- 2 files changed, 13 insertions(+), 11 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 58605ba..22a0685 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -54,13 +54,13 @@ Follow the shape of an existing tool, for example `src/tools/localstack-aws-clie ## Testing Test every tool at these levels; many do not have all of them yet (see [Known deviations](#known-deviations)). Write -the failing test first when fixing a bug. +the failing test first when fixing a bug in `src/`. | Level | Where | Runs in CI | What it proves | |-------|-------|------------|----------------| | Logic | `src/lib//*.logic.test.ts` | always | Parsing, validation, rendering and sanitizing, as pure functions. | | Handler | `src/tools-tests/.test.ts` | always | The tool's actions end to end inside the process, with Docker, LocalStack and spawned binaries mocked at the module boundary. | -| Tools | `tests/mcp/tools.spec.mjs` | every PR, with the auth token | Each tool over MCP stdio against a real emulator, without a model. A case for an action that changes state reads that state back through another call. | +| Tools | `tests/mcp/tools.spec.mjs` | every PR, with the auth token | Each tool over MCP stdio against a real emulator, without a model. | | Server | `tests/mcp/direct.spec.mjs` | every PR, with the auth token | The built server: tool list, prompts, setup wizard. | | Image | `tests/docker/validate-image.mjs` | `docker.yml`, with the auth token | Real calls through the Docker image. | @@ -101,7 +101,7 @@ Rules for tool tests: ## Dependencies Every dependency version lives in exactly one file that Dependabot updates (`.github/dependabot.yml`), so CVE fixes -arrive as PRs instead of waiting for someone to notice: +arrive as PRs instead of waiting for someone to notice, or is a hand-managed pin (below): | What | Pinned in | |------|-----------| @@ -121,9 +121,9 @@ arrive as PRs instead of waiting for someone to notice: `.github/actions/host-tools`, from the same files the image uses. `src/core/dependency-pins.test.ts` checks that the jobs running the tool tests or the evals use it, and that `ci.yml`, `docker.yml` and `mcp-evals.yml` run on the same pinned Ubuntu. -- **Hand-managed pins** (Node's major line, `zod` up to xmcp's copy, `@gleanwork/mcp-server-tester` while - `patches/` patches it) are listed as `ignore` entries in `dependabot.yml` with the reason, and - `src/core/dependency-pins.test.ts` keeps the ones it can in sync. +- **Hand-managed pins** (Node's major line, the runner's Ubuntu release, `zod` up to xmcp's copy, + `@gleanwork/mcp-server-tester` while `patches/` patches it) carry their reason next to them (an `ignore` entry in + `dependabot.yml`, or a comment), and `src/core/dependency-pins.test.ts` keeps the ones it can in sync. - **Fix advisories by upgrading.** `audit-ci --high --skip-dev` gates CI. If no fixed version exists, an allowlist entry names the advisory, why it does not apply, and when to revisit it. - **Actions pinned to a commit SHA**, with the version in a comment (`uses: actions/checkout@ # v7`). @@ -211,7 +211,8 @@ format changes without notice, lines rotate out of the window, and regexes miss A tool may explain how to enable the feature that provides the data (for example App Inspector), but must not fall back to logs. - **Logs are for showing logs.** `localstack-logs-analysis` returns raw or filtered log lines (startup failures, stack - traces). It does not classify lines into services, operations, status codes or IAM denials. + traces) and must not classify them into services, operations, status codes or IAM denials (it still does; see + [Known deviations](#known-deviations)). ### 4. Use only public interfaces @@ -266,7 +267,7 @@ If two clients could compute the same answer differently, the answer belongs in Today `localstack-iam-policy-analyzer` handles IAM by parsing logs and synthesizing policies (known deviations), with its `set-mode` and `get-status` actions for the enforcement mode. IAM moves to `localstack-app-inspector`. None of the actions below exist yet; this is the design they are built to. `localstack-app-inspector` already lists a -span's raw IAM events (`list-iam-events`, `get-event`); `list-iam-denials` and `explain-iam` replace those for IAM. Once its actions ship, the analyzer is +span's raw IAM events (`list-iam-events`, `get-event`); `list-iam-denials` and `explain-iam` are to replace those for IAM. Once its actions ship, the analyzer is deprecated: its behaviour stays unchanged until it is removed, but its description and every response name the app-inspector action that replaces it. diff --git a/REVIEW.md b/REVIEW.md index bf923b3..09533dc 100644 --- a/REVIEW.md +++ b/REVIEW.md @@ -136,7 +136,7 @@ not compute it as a fallback. See "IAM policies" in AGENTS.md. - **BLOCKER:** the tool changes LocalStack's suggestion for a denied request: drops, merges or reorders its changes, - or reports a different `outcome`. What the agent applies must be what LocalStack verified. + or reports a different verdict on whether the changes fix the request. What the agent applies must be what LocalStack verified. - **BLOCKER:** the tool computes a diff between suggested and deployed policies itself, instead of presenting one that LocalStack returned. - **SHOULD-FIX:** a summary policy is presented as verified or complete, instead of as a suggestion from the @@ -182,7 +182,7 @@ mounted socket. See "Testing" in AGENTS.md. -- **BLOCKER:** a bug fix without a test that fails before the fix. +- **BLOCKER:** a bug fix in `src/` without a test that fails before the fix. - **SHOULD-FIX:** a new or changed action without a handler test in `src/tools-tests/.test.ts` covering its success path, each error path (`isError: true`) and each preflight it runs. - **SHOULD-FIX:** new `*.logic.ts` code without unit tests. @@ -197,7 +197,8 @@ See "Testing" in AGENTS.md. - **SHOULD-FIX:** a test reaches real AWS or Azure resources; creates a LocalStack platform resource without cleanup, or with a fixed name outside a `concurrency` group; or skips or passes silently when a required secret is absent (AGENTS.md "Testing" names the only skips allowed). -- **SHOULD-FIX:** a tool's description, schema or response wording changed and the PR does not report an eval run. +- **SHOULD-FIX:** a tool's description, schema or response wording changed and the PR does not report an eval run + (for an external PR, the maintainer runs it when re-pushing). - **SHOULD-FIX:** `README.md` tool or prerequisite tables are out of date. - **NIT:** a change adds a count, version, line number or growing list to AGENTS.md instead of a rule and a pointer, or a list a test could hold (AGENTS.md, "Maintaining this file"). From 9b1d45153a5e435f7daaf64204b50083a3ab6129 Mon Sep 17 00:00:00 2001 From: Joel Scheuner Date: Thu, 19 Nov 2026 18:27:22 +0100 Subject: [PATCH 11/13] docs: address review round 8 (hand-managed pins, read-only actions, image size on bumps) Co-Authored-By: Claude Opus 5.5 --- REVIEW.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/REVIEW.md b/REVIEW.md index 09533dc..e9f50a1 100644 --- a/REVIEW.md +++ b/REVIEW.md @@ -174,7 +174,8 @@ mounted socket. - **BLOCKER:** the image grows by more than about 50 MB per platform, or gains a new runtime, toolchain or base image, and the PR does not record that the user approved it (AGENTS.md "Docker image size"). - **SHOULD-FIX:** a change to `Dockerfile`, `docker/` or `.dockerignore`, or a new runtime dependency, without the - image size before and after for amd64 and arm64. + image size before and after for amd64 and arm64 (for an external or Dependabot PR, the maintainer reports it when + re-pushing). - **SHOULD-FIX:** the runtime stage keeps build tools, dev dependencies, package caches or package lists, or cleans them in a later layer than the one that created them. @@ -192,8 +193,9 @@ See "Testing" in AGENTS.md. the error and analytics. - **SHOULD-FIX:** tests assert on internal calls instead of the response, the spawned argument array or the HTTP request sent to LocalStack. -- **SHOULD-FIX:** a new tool missing from `EXPECTED_TOOLS` in `tests/mcp/direct.spec.mjs` or `tests/docker/validate-image.mjs`, or a new action without - a case in `tests/mcp/tools.spec.mjs` that checks its effect through another call. +- **SHOULD-FIX:** a new tool missing from `EXPECTED_TOOLS` in `tests/mcp/direct.spec.mjs` or + `tests/docker/validate-image.mjs`; a new action without a case in `tests/mcp/tools.spec.mjs`; or a state-changing + one whose case does not check its effect through another call. - **SHOULD-FIX:** a test reaches real AWS or Azure resources; creates a LocalStack platform resource without cleanup, or with a fixed name outside a `concurrency` group; or skips or passes silently when a required secret is absent (AGENTS.md "Testing" names the only skips allowed). @@ -210,7 +212,7 @@ See "Testing" in AGENTS.md. See "Dependencies" in AGENTS.md. - **BLOCKER:** the Dockerfile, a workflow or a script installs a dependency or tool without a pinned version in a - file Dependabot updates: `npm install -g`, + file Dependabot updates or as a hand-managed pin listed in AGENTS.md "Dependencies": `npm install -g`, `pip install`, `curl … | sh`, a `latest` tag (other than the LocalStack emulator images), or `npx` of a package not in `package.json`. Debian packages from the base image's release are allowed (AGENTS.md, "Dependencies"). - **BLOCKER:** a known high or critical advisory is silenced (an `audit-ci` allowlist entry, a removed audit step) From d0069eab026ab342151d66f7f9d4c9ace6b0d1ae Mon Sep 17 00:00:00 2001 From: Joel Scheuner Date: Thu, 19 Nov 2026 18:32:07 +0100 Subject: [PATCH 12/13] docs: address review round 9 (maintainer approvals, helper changes, audit scope) Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 21 +++++++++++++-------- REVIEW.md | 6 +++--- 2 files changed, 16 insertions(+), 11 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 22a0685..5467569 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -7,6 +7,10 @@ infrastructure, run cloud CLIs, and inspect the emulator. It is a TypeScript ser Read [REVIEW.md](REVIEW.md) before opening a PR. It turns the rules below into review checks. +Where a rule says to ask the user, ask the person you work for. An approval that lets a change break a rule, adopt +a new API or grow the image counts only if it comes from a maintainer of this repository and the PR records it; a +contributor cannot approve their own exception. + ## Commands ```bash @@ -91,9 +95,9 @@ Rules for tool tests: name only when a workflow `concurrency` group keeps runs from overlapping, as `mcp-evals.yml` does. Costly ones (ephemeral instances) stay opt-in (`HARNESS_RUN_EPHEMERAL`). - **Missing secrets fail loudly.** A job that needs a secret fails with a message naming it when it is empty, as - `ci.yml` does for fork PRs, so a fork PR never looks tested. Only an opt-in scenario (one that needs its own - credentials, such as the replicator's source AWS keys, or a flag such as `HARNESS_RUN_REMOTE`) or an outside - service being down (the docs search) may skip a case, and the output says why. + `ci.yml` does for fork PRs, so a fork PR never looks tested. A case may skip for a missing secret, credential + or tool only when it belongs to an opt-in scenario (the replicator's source AWS keys, `HARNESS_RUN_REMOTE`), or + when an outside service is down (the docs search), and the output says why. - **Evals are not PR gates.** `test:mcp:evals` depends on a model. `mcp-evals.yml` runs it after merges to `main` that touch its path filter, weekly and on demand. Run them yourself when a tool's description, schema or response wording changes, and report the result in the PR. @@ -232,7 +236,7 @@ public docs or in one of the OpenAPI specs a running emulator serves: Ask the user before adopting an API this server does not use yet, even a documented one, and name the endpoint and what the tool would do with it. The IAM design below -would add the event policy route and the IAM policy summary; adopting them still needs the user's approval, +would add the event policy route and the IAM policy summary; adopting them still needs a maintainer's approval, recorded in the PR that adds them. Exception: an endpoint listed under [Gaps in LocalStack](#gaps-in-localstack) may be used before it is documented. @@ -278,8 +282,8 @@ what is missing instead of returning an empty answer. App Inspector IAM events (`/_localstack/appinspector/v1/traces/{t}/spans/{s}/events/iam`), which include denials that were recorded but not enforced. An event's payload holds LocalStack's decisions (which policies allowed or denied the request, and at which statements), the policies involved, and for a denial a versioned suggestion of - what to change and whether that fixes the request. `explain-iam` is to render them in the section order of the - App Inspector UI's "Copy agent prompt", then the payload as sent. Do not recompute, merge or reorder the changes. + what to change and whether that fixes the request. `explain-iam` is to render, in this order: the request and its + result, the deciding policies, the suggested fix, then the payload as sent. Do not recompute, merge or reorder the changes. A policy the event stored truncated is fetched on demand with `get-iam-policy` (the event policy route), never automatically. - Without the `localstack.appinspector/view_iam_policy_errors` entitlement, LocalStack hides IAM events and @@ -337,10 +341,10 @@ row once it ships. ## Known deviations Code on `main` (of the repository that holds it) that does not follow the rules: deviations that predate them, and -exceptions the user approved. A row is never a way to let unapproved code through. +exceptions a maintainer approved. A row is never a way to let unapproved code through. - **New code follows the rules.** A change that needs to break one raises it with the user before it is written. - If the user approves, the PR records the approval and adds a row naming who approved it and when. + If a maintainer approves, the PR records the approval and adds a row naming who approved it and when. - **A change that touches a listed deviation raises it.** Say in the PR whether the change fixes the deviation, leaves it as it is, or why it cannot fix it yet. Do not copy the pattern and do not extend it to another tool. - **Remove a row when the code is fixed.** @@ -364,6 +368,7 @@ under [Follow-ups](#follow-ups) until it lands. | `.github/workflows/azure-smoke.yml`, `ci.yml` (`mcp-direct-tests`), `tests/mcp/azure-offline.spec.mjs` | `azure-smoke.yml` runs on `ubuntu-latest`; it and `mcp-direct-tests` use the runner image's Azure CLI, whose version no file here pins. `azure-offline` skips its cases when `az` is missing, a skip "Testing" does not allow. | The pinned Ubuntu of the other workflows, and the Azure CLI from a pinned file, as `host-tools` does for Terraform, uv and lstk. | | `.github/workflows/` | Secrets reach third-party code in CI (REVIEW §0): jobs set `LOCALSTACK_AUTH_TOKEN` (and, in `mcp-evals.yml`, the OpenRouter key) at job level, so `yarn install` runs dependency install scripts and third-party actions run with them; `docker.yml` runs `npm ci` for `data/sample-cdk`, with install scripts, in a step that holds the token; and `actions/checkout` leaves the GitHub token in `.git/config` for later steps. | Install dependencies with no secret in the environment (or with `--ignore-scripts`), set secrets only on the steps that need them, and check out with `persist-credentials: false`. | | `src/tools/`, `src/lib/` | Most tools render LocalStack and workload strings (names, outputs, log lines, error text) into markdown tables, text and fixed code fences unescaped (REVIEW §7). `list-iam-events` answers "No events found." when the license hides IAM events, instead of explaining it ("IAM policies"). | Escape them, and size code fences to their content (REVIEW §7); explain hidden IAM events. | +| `.github/workflows/ci.yml` | `audit-ci --skip-dev` skips the devDependencies that are bundled into `dist/` and run in production, and nothing audits the `docker/tools/` lockfiles. | Audit what ships: the bundled packages and the image's lockfiles. | | `.github/workflows/docker.yml` | Its path filter misses files the image build reads (`patches/`, `xmcp.config.ts`, `tsconfig*.json`), so a change to them is not built or tested as an image. | Filter on everything the `Dockerfile` copies. | | `src/lib/localstack/license-checker.ts` | Calls `GET /_localstack/licenseinfo`, which is not documented and not covered by the rule 4 exception. | The documented way LocalStack names (see [Gaps in LocalStack](#gaps-in-localstack)). | | `data/sample-terraform/` | The AWS provider has only a version constraint, no lockfile, and no Dependabot entry; CI downloads it with secrets in the job's environment. | A committed `.terraform.lock.hcl` and a `terraform` Dependabot entry ("Dependencies"). | diff --git a/REVIEW.md b/REVIEW.md index e9f50a1..f8bdec6 100644 --- a/REVIEW.md +++ b/REVIEW.md @@ -8,9 +8,9 @@ state, then the wrong result). Call something a BLOCKER only after confirming it "Known deviations" in AGENTS.md lists existing code on `main` only. For a change under review: - New code that breaks a rule is a finding at the rule's severity, even if the same pattern is listed. A new row in - "Known deviations" excuses it only if the PR records that the user approved the deviation. + "Known deviations" excuses it only if the PR records that a maintainer approved the deviation. - Calling a shared helper whose behaviour is a listed deviation (for example `requireProFeature` or - `withToolAnalytics`) is not new code; copying the pattern or changing the helper is. + `withToolAnalytics`) is not new code; copying the pattern is. Changing the helper falls under the next bullet. - A change may touch the deviating code itself (not just a file a row names) without fixing it, but the PR must say so: fixed, left as is, or why it cannot be fixed yet (SHOULD-FIX if it does not). It must not extend a deviation to another tool. @@ -172,7 +172,7 @@ mounted socket. (`localhost` versus `host.docker.internal`) and binaries the image must bundle. - **SHOULD-FIX:** a new host binary is needed but neither README prerequisites nor the `Dockerfile` are updated. - **BLOCKER:** the image grows by more than about 50 MB per platform, or gains a new runtime, toolchain or base - image, and the PR does not record that the user approved it (AGENTS.md "Docker image size"). + image, and the PR does not record that a maintainer approved it (AGENTS.md "Docker image size"). - **SHOULD-FIX:** a change to `Dockerfile`, `docker/` or `.dockerignore`, or a new runtime dependency, without the image size before and after for amd64 and arm64 (for an external or Dependabot PR, the maintainer reports it when re-pushing). From ae67d2eef17eeb344ca39df42927bc7bb4be0dea Mon Sep 17 00:00:00 2001 From: Joel Scheuner Date: Thu, 19 Nov 2026 18:36:46 +0100 Subject: [PATCH 13/13] =?UTF-8?q?docs:=20address=20review=20round=2010=20(?= =?UTF-8?q?maintainer=20approval=20in=20=C2=A74,=20rules=20apply=20to=20th?= =?UTF-8?q?e=20change,=20testable=20escaping)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 6 +++--- REVIEW.md | 8 +++++--- 2 files changed, 8 insertions(+), 6 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 5467569..eeaa3b0 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -81,7 +81,7 @@ Rules for tool tests: - **Schema.** Invalid arguments are rejected by the zod schema; test a missing required field and a wrong enum value for each action that takes them. - **Adversarial input.** For a tool that renders LocalStack or workload strings into markdown, add a case whose - payload contains a code fence, a table pipe and an instruction-like line, and assert it cannot escape. For a tool + payload contains a code fence, a table pipe and a markdown heading, and assert they stay inside their fence or cell. For a tool that builds a command line, add cases that try shell chaining, option injection and path traversal. - **No secrets in output.** Run the handler with a recognizable fake token and credentials, and assert they appear in neither the response, the error, nor the analytics payload. @@ -316,7 +316,7 @@ request; `SOFT_MODE` keeps them passing. Never tell a user to switch on enforcem `ENFORCED` can break a workload that worked. Render every payload string escaped (REVIEW §7): an ARN, a resource name or an error message must not be able to -close a code fence, break a table, or read as an instruction. +close a code fence, break a table, or leave its fence or cell. An agentic evaluation of these actions is a follow-up (see [Follow-ups](#follow-ups)). @@ -372,7 +372,7 @@ under [Follow-ups](#follow-ups) until it lands. | `.github/workflows/docker.yml` | Its path filter misses files the image build reads (`patches/`, `xmcp.config.ts`, `tsconfig*.json`), so a change to them is not built or tested as an image. | Filter on everything the `Dockerfile` copies. | | `src/lib/localstack/license-checker.ts` | Calls `GET /_localstack/licenseinfo`, which is not documented and not covered by the rule 4 exception. | The documented way LocalStack names (see [Gaps in LocalStack](#gaps-in-localstack)). | | `data/sample-terraform/` | The AWS provider has only a version constraint, no lockfile, and no Dependabot entry; CI downloads it with secrets in the job's environment. | A committed `.terraform.lock.hcl` and a `terraform` Dependabot entry ("Dependencies"). | -| `playwright.config.mjs`, `src/tools/localstack-snowflake-client.ts`, `src/core/localstack-env.ts` | Child processes inherit the whole environment (REVIEW §0): the server under test gets the eval's model keys, and the `snow` CLI and `lstk terraform` / `cdk` / `sam` get every variable of the server's environment. | Pass each child only the variables it needs. | +| `src/core/command-runner.ts`, `src/tools/localstack-snowflake-client.ts`, `src/core/localstack-env.ts`, `playwright.config.mjs` | Child processes inherit the whole environment (REVIEW §0): `runCommand` spawns without an `env` of its own, the `snow` CLI and `lstk terraform` / `cdk` / `sam` get every variable of the server's environment, and the server under test gets the eval's model keys. | Pass each child only the variables it needs. New callers of `runCommand` pass an explicit `env` (REVIEW §0). | | `Dockerfile` | The `# syntax=` line floats to the newest frontend of its major version. | Pin the frontend version where Dependabot can update it. | | `src/tools-tests/`, `src/lib/**/*.logic.test.ts` | Most tools have no handler test. The existing handler tests stub `withToolAnalytics` or mock internal modules and assert on their calls, and some `*.logic.test.ts` files test tool handlers instead of a `*.logic.ts` module. | Handler tests for every tool, following "Testing", with tool handler tests in `src/tools-tests/`. A ratchet test that lists the tools still without one would replace this row (see "Maintaining this file"). | | `src/lib/iam/iam-policy.logic.ts` | Synthesizes IAM policies from denials, for the analyzer (rule 5). | Remove with the analyzer; the IAM actions present LocalStack's suggestions and summary instead. | diff --git a/REVIEW.md b/REVIEW.md index f8bdec6..66feaf3 100644 --- a/REVIEW.md +++ b/REVIEW.md @@ -5,6 +5,8 @@ Rules for reviewing a change to this repository. The design rules they enforce a Report each finding as BLOCKER, SHOULD-FIX or NIT, with file and line and a concrete failure scenario (inputs and state, then the wrong result). Call something a BLOCKER only after confirming it in the code. +Each rule applies to what the change adds or alters, not to gaps the code already has. + "Known deviations" in AGENTS.md lists existing code on `main` only. For a change under review: - New code that breaks a rule is a finding at the rule's severity, even if the same pattern is listed. A new row in @@ -116,8 +118,8 @@ This repository is public, so these are BLOCKERs: - a script or command run inside the LocalStack container to reach internals - an undocumented configuration variable -Also a BLOCKER: the change adopts an API the server did not use before, and the PR does not record that the user -approved it (AGENTS.md rule 4). +Also a BLOCKER: the change adopts an API the server did not use before, and the PR does not record that a +maintainer approved it (AGENTS.md rule 4). ## 5. Compute in LocalStack, render here @@ -152,7 +154,7 @@ See "IAM policies" in AGENTS.md. bypasses its sanitizer. - **BLOCKER:** a string from LocalStack or the user's workload (resource names, ARNs, error messages, policy JSON, log lines) reaches an agent-facing response without escaping. A value must not be able to close a code fence, - break a table, or read as an instruction. Size code fences to their content, and escape markdown in table cells + break a table, or leave its fence or cell. Size code fences to their content, and escape markdown in table cells and inline text. - **SHOULD-FIX:** the handler skips `withToolAnalytics`, preflights or `ResponseBuilder`, or a new tool argument that can never hold secrets or user data is missing from the analytics allowlist in `src/core/analytics.ts`, or