From 41fb685b824b2b5f754b864b62edf6bc632f9dea Mon Sep 17 00:00:00 2001 From: Ayla Croft Date: Tue, 8 Sep 2026 13:31:30 -0400 Subject: [PATCH 1/6] Replace BeamMCP.ToolCatalog with BeamMCP.Catalog, carrying resources and prompts BREAKING, to a host contract rather than to the wire. Taken now, before the resources and prompts slices exist, so it is taken once. (a) MIGRATION PATH: a clean break, no deprecated delegate. The decisive argument is not taste, it is the version policy already in place. Breaks land at the minor while this package is 0.x, and the README recommends `~> 0.3.0`, which EXCLUDES 0.4.0 -- measured in slice 003's version table. No consumer is carried across by a routine mix deps.update; that tight pin exists for exactly this. And a delegate would cost the thing the contract is for: keeping ToolCatalog as a shim means two entry points into "which tools exist", which is the second reader that decision (b) forbids. (b) THE SINGLE-LOOKUP GUARANTEE: preserved, and strengthened. tool_catalog.ex promised it in prose and left two call sites -- server.ex:185 called catalog.all() directly, Catalog.fetch/2 called it again. Now both go through Catalog.tools/1, one function, so advertise and call cannot drift apart without deleting it. Slice 002 fixed a real defect of exactly this shape: tools/list honoured an injected catalog and tools/call ignored it. Pinned by effect, not by prose: catalog_test.exs compares what tools/list advertises against what tools/call accepts, for a catalog whose tool exists in no other catalog in the suite -- so neither path can pass by the coincidence that made slice 002's original test green. (c) THE fetch/2 @spec: documented honestly, not caught. Catching would turn a host bug into :error, which is indistinguishable from "no such tool" -- the advertise-versus-call confusion this behaviour exists to prevent, reintroduced by the error handling meant to be defensive. The raise conditions are listed in the @doc and one is pinned by a test. THE COUNT IN PR #10's LEDGER IS WRONG AND IS CORRECTED. It claims three shapes raise. Measured on this tree (slices/008-catalog-generalization/logs/probe-fetch-spec.txt): host spec.name is a binary ArgumentError capabilities/0 returns a non-list map BadMapError <- not the capabilities/0 returns nil Protocol.UndefinedError one error module not loaded UndefinedFunctionError Four, not three: "returns a non-list" is two different raises depending on the shape. And a FIFTH case the ledger missed entirely, which is worse than a raise because nothing fails: a bare map entry returns {:ok, %{name: :echo}} SUCCESSFULLY, violating the @spec silently. That one is closed by init-time validation rather than by documentation. CALLBACK RENAMED, not just re-typed. Keeping all/0 while changing its return from a list to a map compiles against every existing host and fails at the first request with a BadMapError. capabilities/0 makes the break arrive at compile time as an unimplemented callback. The option is :catalog, not :tool_catalog, for the same reason: a catalog carrying resources and prompts is not a tool catalog. A malformed catalog is refused at Server.new/1 -- runtime, so calling the host is safe. Transport.HTTP.init/1 checks only the export, deliberately: under Plug's default init_mode it is the host's COMPILE time, and a correct catalog reading config would fail there. The call-site list I was given missed several: server.ex's alias, moduledoc, typespec and new/1 key; stdio.ex's two moduledoc mentions; http.ex's alias, moduledoc, opts lookup and the ArgumentError text naming all/0. server.ex:311 (dispatch) is NOT affected -- it consumes a ToolSpec, which is unchanged. 187 tests, 0 failures -- quoted from the run, and corrected: the first version of this message typed 185, which no command had printed. Gate OK. GATE_EXIT=0 Signed-off-by: Ayla Croft --- CHANGELOG.md | 39 ++++ README.md | 27 ++- lib/beam_mcp/catalog.ex | 154 ++++++++++++++ lib/beam_mcp/server.ex | 41 +++- lib/beam_mcp/tool_catalog.ex | 35 ---- lib/beam_mcp/transport/http.ex | 37 ++-- lib/beam_mcp/transport/stdio.ex | 4 +- .../logs/green-malformed-catalog.txt | 10 + .../logs/probe-fetch-spec.txt | 11 + .../logs/red-malformed-catalog.txt | 55 +++++ test/beam_mcp/catalog_test.exs | 198 ++++++++++++++++++ test/beam_mcp/error_payload_test.exs | 8 +- test/beam_mcp/injection_test.exs | 10 +- test/beam_mcp/negotiation_test.exs | 8 +- test/beam_mcp/readme_claims_test.exs | 52 +++-- test/beam_mcp/server_test.exs | 14 +- test/beam_mcp/tool_spec_schema_test.exs | 8 +- test/beam_mcp/transport/http_bandit_test.exs | 8 +- test/beam_mcp/transport/http_test.exs | 111 ++++++---- test/beam_mcp/transport/stdio_test.exs | 8 +- 20 files changed, 687 insertions(+), 151 deletions(-) create mode 100644 lib/beam_mcp/catalog.ex delete mode 100644 lib/beam_mcp/tool_catalog.ex create mode 100644 slices/008-catalog-generalization/logs/green-malformed-catalog.txt create mode 100644 slices/008-catalog-generalization/logs/probe-fetch-spec.txt create mode 100644 slices/008-catalog-generalization/logs/red-malformed-catalog.txt create mode 100644 test/beam_mcp/catalog_test.exs diff --git a/CHANGELOG.md b/CHANGELOG.md index 1952be5..db535cd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,45 @@ All notable changes to this project are documented here. The format follows ## [Unreleased] +### Changed — BREAKING, and it breaks a host contract rather than the wire + +- **`BeamMCP.ToolCatalog` is replaced by `BeamMCP.Catalog`, and `all/0` by `capabilities/0`.** + Every host implementing a catalog must change. While this package is `0.x` a break lands at + the **minor** position, so this is `0.4.0` and `~> 0.3.0` — the requirement the README + recommends — already excludes it. No consumer is carried across by a routine + `mix deps.update`; that is what the tight pin is for. + + `capabilities/0` returns `%{tools: [ToolSpec.t()], resources: [], prompts: []}`. `resources` + and `prompts` are **required and may be empty**. Nothing reads them yet. They exist so that + serving resources and prompts later adds a reader rather than changing this contract a second + time — the break is taken once, now, before those slices exist. + + **The callback is renamed, not just re-typed.** Keeping `all/0` while changing its return from + a list to a map would compile against every existing host and fail at the first request with a + `BadMapError`. Renaming makes the break arrive at compile time as an unimplemented callback. + + The option is `:catalog`, not `:tool_catalog` — a catalog carrying resources and prompts is + not a tool catalog, and renaming it in the same break costs less than a second one later. + + **Migration:** + + ```elixir + # before + @behaviour BeamMCP.ToolCatalog + def all, do: [%BeamMCP.ToolSpec{...}] + + # after + @behaviour BeamMCP.Catalog + def capabilities, do: %{tools: [%BeamMCP.ToolSpec{...}], resources: [], prompts: []} + ``` + +- **A malformed catalog is refused by `BeamMCP.Server.new/1`**, at startup, with a message naming + what is wrong — an absent key, a non-list `:tools`, an entry that is not a `%ToolSpec{}`, a + `capabilities/0` that does not return a map, or a module that does not export it. + `Transport.HTTP.init/1` checks only that the callback is exported, deliberately: under Plug's + default initialisation it runs at the host's **compile** time, where calling a catalog that + reads config would fail for a correct host. + ### Added - **`:authorize_body`, an optional post-read authorization hook.** `authorize/1` runs before the diff --git a/README.md b/README.md index 8484e0a..b095974 100644 --- a/README.md +++ b/README.md @@ -29,15 +29,23 @@ not. The tighter form is deliberate and is not an over-pin to be tidied away. Injection without a specification is a claim with nothing behind it, so both are declared. -**`BeamMCP.ToolCatalog`** — the host names the tools. +**`BeamMCP.Catalog`** — the host names what it offers. + +`capabilities/0` returns a map with three required keys. `resources` and `prompts` may be empty +and nothing reads them yet; they are required so that serving them later adds a reader rather +than changing this contract a second time. **An absent key is a malformed catalog, not an empty +one**, and `BeamMCP.Server.new/1` refuses it at startup rather than at the first request. ```elixir defmodule MyApp.Catalog do - @behaviour BeamMCP.ToolCatalog + @behaviour BeamMCP.Catalog @impl true - def all do - [ + def capabilities do + %{ + resources: [], + prompts: [], + tools: [ %BeamMCP.ToolSpec{ name: :get_weather, command_class: :observe, @@ -50,7 +58,8 @@ defmodule MyApp.Catalog do "additionalProperties" => false } } - ] + ] + } end end ``` @@ -65,13 +74,13 @@ end ```elixir BeamMCP.Transport.Stdio.run( - tool_catalog: MyApp.Catalog, + catalog: MyApp.Catalog, dispatch: &MyApp.Dispatch.call/3, server_name: "my-app" ) ``` -`:tool_catalog` is required. `:dispatch` is required for `tools/call`. `:server_name` defaults +`:catalog` is required. `:dispatch` is required for `tools/call`. `:server_name` defaults to `beam_mcp`, and a host that wants its own name in `initialize` says so. ## One schema, one source @@ -110,7 +119,7 @@ no such step. ```elixir Bandit.child_spec( plug: {BeamMCP.Transport.HTTP, - tool_catalog: MyApp.Catalog, + catalog: MyApp.Catalog, dispatch: &MyApp.Dispatch.call/3, authorize: &MyApp.Auth.check/1, allowed_origins: ["https://app.example.com"]}, @@ -131,7 +140,7 @@ defmodule MyApp.Router do forward "/mcp", to: BeamMCP.Transport.HTTP, init_opts: [ - tool_catalog: MyApp.Catalog, + catalog: MyApp.Catalog, dispatch: &MyApp.Dispatch.call/3, authorize: &MyApp.Auth.check/1, allowed_origins: ["https://app.example.com"] diff --git a/lib/beam_mcp/catalog.ex b/lib/beam_mcp/catalog.ex new file mode 100644 index 0000000..b2b9288 --- /dev/null +++ b/lib/beam_mcp/catalog.ex @@ -0,0 +1,154 @@ +# SPDX-FileCopyrightText: 2026 Sudo Apt Holdings LLC +# SPDX-License-Identifier: Apache-2.0 + +defmodule BeamMCP.Catalog do + @moduledoc """ + The contract a host implements to tell a `BeamMCP.Server` what it offers. + + The server holds no catalog of its own. It advertises what `capabilities/0` returns and + accepts a `tools/call` only for a tool `capabilities/0` names, so one implementation governs + both — a tool advertised by `tools/list` and refused by `tools/call` is the defect this + behaviour exists to make impossible. + + ## The shape carries keys this package does not yet serve + + %{tools: [BeamMCP.ToolSpec.t()], resources: [], prompts: []} + + `resources` and `prompts` are **required and may be empty**. Nothing reads them today. They + are here so that serving them later adds a reader rather than changing this contract a second + time, and every host that has already written `capabilities/0` keeps working when they do. + + A key that is absent is a malformed catalog, not an empty one — the two are different claims + and only one of them is checkable. + + ## Why `capabilities/0` and not `all/0` + + This behaviour replaces `BeamMCP.ToolCatalog`, whose callback was `all/0` returning a list. + Keeping the name while changing the return from a list to a map would compile against every + existing host and fail at the first request with a `BadMapError` — a silent shape change, + which is the defect class this repository keeps finding. Renaming makes the break arrive at + compile time as an unimplemented callback, which is the loudest place it can arrive. + """ + + @typedoc """ + What a host offers. Every key is required; `resources` and `prompts` are typed as generic + lists because nothing in this package reads them yet, and typing them precisely now would be + a claim about a shape no code enforces. + """ + @type t :: %{ + required(:tools) => [BeamMCP.ToolSpec.t()], + required(:resources) => list(), + required(:prompts) => list() + } + + @callback capabilities() :: t() + + @required_keys [:tools, :resources, :prompts] + + @doc """ + The tools a catalog offers. + + **This is the single reader, and that is the point rather than a convenience.** Both paths go + through it: `tools/list` advertises what it returns, and `fetch/2` decides callability from + the same call. Slice 002 fixed a real defect where advertising honoured an injected catalog + and calling ignored it — two readers, two answers. One function is how that stays fixed, and + a mutant that gives the two paths different sources is scored in + `slices/008-catalog-generalization/`. + """ + @spec tools(module()) :: [BeamMCP.ToolSpec.t()] + def tools(catalog), do: catalog.capabilities().tools + + @doc """ + Finds the spec a tool name refers to, or `:error`. + + One lookup, used by every caller that has to answer "which tool does this name mean". The + core uses it to decide whether a `tools/call` is callable at all; the HTTP transport uses it + to read the `x-mcp-header` annotations it must validate against. Two implementations of this + question would be two answers, which is precisely the header-versus-body disagreement the + transport's validation exists to prevent. + + ## This function raises on a malformed catalog, and the `@spec` does not say so + + Stated rather than caught, and the reason is the guarantee above. A host bug turned into + `:error` is indistinguishable from "no such tool" — so a catalog that is broken would present + exactly as a catalog that is working and simply does not have that tool. That is the + advertise-versus-call disagreement this behaviour exists to prevent, reintroduced by the + error handling meant to be defensive. + + Measured on this tree rather than inherited from a prior note + (`slices/008-catalog-generalization/logs/probe-fetch-spec.txt`): + + host spec.name is a binary, not an atom ArgumentError + capabilities/0 returns a non-list map BadMapError + capabilities/0 returns nil Protocol.UndefinedError + module is not loaded / does not exist UndefinedFunctionError + + Use `validate/1` to refuse these at startup instead, which is what `BeamMCP.Server.new/1` + does. + """ + @spec fetch(module(), String.t() | atom()) :: {:ok, BeamMCP.ToolSpec.t()} | :error + def fetch(catalog, name) when is_atom(name), do: fetch(catalog, Atom.to_string(name)) + + def fetch(catalog, name) when is_atom(catalog) and is_binary(name) do + catalog + |> tools() + |> Enum.find_value(:error, fn spec -> + if Atom.to_string(spec.name) == name, do: {:ok, spec} + end) + end + + def fetch(_catalog, _name), do: :error + + @doc """ + Checks that a module is a usable catalog, returning `:ok` or `{:error, reason}`. + + Calls `capabilities/0`, so it belongs where host code may safely run. `BeamMCP.Server.new/1` + calls it; the HTTP transport's `Plug` callback `init/1` deliberately does not, because Plug's + default initialisation is the host's **compile** time and a catalog reading config or ETS + there would fail for a correct host. That transport checks the export and leaves the shape to + `new/1`. (`init/1` is not linked here: it is a hidden callback, and a doc reference to it is a + broken link rather than a useful one.) + """ + @spec validate(module()) :: :ok | {:error, String.t()} + def validate(catalog) when is_atom(catalog) and not is_nil(catalog) do + with {:module, _} <- Code.ensure_compiled(catalog), + true <- function_exported?(catalog, :capabilities, 0) do + validate_shape(catalog) + else + _ -> {:error, "#{inspect(catalog)} does not export capabilities/0"} + end + end + + def validate(other), do: {:error, "expected a module, got: #{inspect(other)}"} + + defp validate_shape(catalog) do + case catalog.capabilities() do + %{} = caps -> + validate_keys(catalog, caps) + + other -> + {:error, "#{inspect(catalog)}.capabilities/0 must return a map, got: #{inspect(other)}"} + end + rescue + e -> {:error, "#{inspect(catalog)}.capabilities/0 raised #{inspect(e.__struct__)}"} + end + + defp validate_keys(catalog, caps) do + missing = Enum.reject(@required_keys, &Map.has_key?(caps, &1)) + + cond do + missing != [] -> + {:error, + "#{inspect(catalog)}.capabilities/0 is missing required key(s): #{inspect(missing)}"} + + not is_list(caps.tools) -> + {:error, "#{inspect(catalog)}.capabilities/0's :tools must be a list"} + + not Enum.all?(caps.tools, &match?(%BeamMCP.ToolSpec{}, &1)) -> + {:error, "#{inspect(catalog)}.capabilities/0's :tools must all be %BeamMCP.ToolSpec{}"} + + true -> + :ok + end + end +end diff --git a/lib/beam_mcp/server.ex b/lib/beam_mcp/server.ex index a2b80c8..7c84bb3 100644 --- a/lib/beam_mcp/server.ex +++ b/lib/beam_mcp/server.ex @@ -2,8 +2,8 @@ # SPDX-License-Identifier: Apache-2.0 defmodule BeamMCP.Server do + alias BeamMCP.Catalog alias BeamMCP.Schema - alias BeamMCP.ToolCatalog @moduledoc """ The protocol core: one message in, one response out, no process and no state of its own. @@ -30,7 +30,7 @@ defmodule BeamMCP.Server do ## What the host supplies BeamMCP.Server.new( - tool_catalog: MyApp.Catalog, # required, a BeamMCP.ToolCatalog + catalog: MyApp.Catalog, # required, a BeamMCP.Catalog dispatch: &MyApp.Dispatch.call/3, # required for tools/call server_name: "my-app" # optional, defaults to "beam_mcp" ) @@ -70,7 +70,7 @@ defmodule BeamMCP.Server do initialized?: boolean(), server_name: String.t(), shutdown?: boolean(), - tool_catalog: module(), + catalog: module(), tools_cache_scope: String.t(), tools_ttl_ms: non_neg_integer() } @@ -83,7 +83,7 @@ defmodule BeamMCP.Server do initialized?: false, server_name: Keyword.get(opts, :server_name, @default_server_name), shutdown?: false, - tool_catalog: Keyword.fetch!(opts, :tool_catalog), + catalog: fetch_catalog!(opts), # 2026-07-28 requires ttlMs and cacheScope on tools/list results. Neither is the # package's to invent: ttlMs is a freshness hint about a catalog the host owns, and # cacheScope is a disclosure decision -- "public" lets shared intermediaries cache a @@ -183,7 +183,7 @@ defmodule BeamMCP.Server do end def handle_message(state, %{"jsonrpc" => "2.0", "id" => id, "method" => "tools/list"}) do - tools = Enum.map(state.tool_catalog.all(), &tool_definition/1) + tools = state.catalog |> Catalog.tools() |> Enum.map(&tool_definition/1) # CacheableResult: 2026-07-28 requires both fields on tools/list. They are carried at # both eras rather than only the modern one -- 2025-11-25 permits any result structure, @@ -299,8 +299,35 @@ defmodule BeamMCP.Server do defp stringify(value), do: Jason.encode!(value) # One lookup governs both paths: a tool is callable exactly when the injected catalog names - # it, and the spec it returns carries the schema that will be enforced. - defp find_tool(state, name), do: ToolCatalog.fetch(state.tool_catalog, name) + # it, and the spec it returns carries the schema that will be enforced. Both this and the + # advertisement above go through `Catalog.tools/1`, so there is one reader rather than two + # call sites that could drift apart. + defp find_tool(state, name), do: Catalog.fetch(state.catalog, name) + + # THE SHAPE IS REFUSED HERE, not at the first request. `new/1` is runtime, so calling the + # host's `capabilities/0` is safe -- unlike `Transport.HTTP.init/1`, which under Plug's + # default init_mode is the host's COMPILE time. Same pattern as `:authorize`: a host that + # mis-wires this learns when it starts, not from a BadMapError in a request path. + defp fetch_catalog!(opts) do + catalog = Keyword.fetch!(opts, :catalog) + + case Catalog.validate(catalog) do + :ok -> + catalog + + {:error, reason} -> + raise ArgumentError, """ + BeamMCP.Server requires a :catalog implementing the BeamMCP.Catalog behaviour. + + #{reason} + + capabilities/0 must return a map with all three keys; resources and prompts may be + empty, but an absent key is a malformed catalog rather than an empty one: + + %{tools: [%BeamMCP.ToolSpec{}], resources: [], prompts: []} + """ + end + end # The advertised schema is the contract. Validate the wire form -- string keys, as the # client sent them -- before normalising, so `required` and `additionalProperties` diff --git a/lib/beam_mcp/tool_catalog.ex b/lib/beam_mcp/tool_catalog.ex deleted file mode 100644 index 1ea0f8b..0000000 --- a/lib/beam_mcp/tool_catalog.ex +++ /dev/null @@ -1,35 +0,0 @@ -# SPDX-FileCopyrightText: 2026 Sudo Apt Holdings LLC -# SPDX-License-Identifier: Apache-2.0 - -defmodule BeamMCP.ToolCatalog do - @moduledoc """ - The contract a host implements to tell a `BeamMCP.Server` which tools exist. - - The server holds no catalog of its own. It advertises what `all/0` returns and accepts a - `tools/call` only for a tool `all/0` names, so one implementation governs both — a tool - advertised by `tools/list` and refused by `tools/call` is the defect this behaviour exists - to make impossible. - """ - - @callback all() :: [BeamMCP.ToolSpec.t()] - - @doc """ - Finds the spec a tool name refers to, or `:error`. - - One lookup, used by every caller that has to answer "which tool does this name mean". The - core uses it to decide whether a `tools/call` is callable at all; the HTTP transport uses it - to read the `x-mcp-header` annotations it must validate against. Two implementations of this - question would be two answers, which is precisely the header-versus-body disagreement the - transport's validation exists to prevent. - """ - @spec fetch(module(), String.t() | atom()) :: {:ok, BeamMCP.ToolSpec.t()} | :error - def fetch(catalog, name) when is_atom(name), do: fetch(catalog, Atom.to_string(name)) - - def fetch(catalog, name) when is_atom(catalog) and is_binary(name) do - Enum.find_value(catalog.all(), :error, fn spec -> - if Atom.to_string(spec.name) == name, do: {:ok, spec} - end) - end - - def fetch(_catalog, _name), do: :error -end diff --git a/lib/beam_mcp/transport/http.ex b/lib/beam_mcp/transport/http.ex index 4c9db0f..181f739 100644 --- a/lib/beam_mcp/transport/http.ex +++ b/lib/beam_mcp/transport/http.ex @@ -92,7 +92,7 @@ if Code.ensure_loaded?(Plug) do Bandit.child_spec( plug: {BeamMCP.Transport.HTTP, - tool_catalog: MyApp.Catalog, + catalog: MyApp.Catalog, dispatch: &MyApp.Dispatch.call/3, authorize: &MyApp.Auth.check/1, allowed_origins: ["https://app.example.com"]}, @@ -110,8 +110,8 @@ if Code.ensure_loaded?(Plug) do require Logger + alias BeamMCP.Catalog alias BeamMCP.Server - alias BeamMCP.ToolCatalog @modern_version "2026-07-28" @version_meta_key "io.modelcontextprotocol/protocolVersion" @@ -202,7 +202,7 @@ if Code.ensure_loaded?(Plug) do """ end - catalog = Keyword.get(opts, :tool_catalog) + catalog = Keyword.get(opts, :catalog) # Checked for the behaviour, not for truthiness: `tool_catalog: true` passed the old check # and failed later, at the first tools/list, as an UndefinedFunctionError from inside the @@ -213,12 +213,19 @@ if Code.ensure_loaded?(Plug) do # `ensure_loaded?` raised ArgumentError at build time naming a perfectly valid catalog. # An option contract that rejects correct configurations is worse than the truthiness # check it replaced. + # STRUCTURAL ONLY, DELIBERATELY. This checks that the module exports `capabilities/0` + # and does NOT call it. Under Plug's default initialisation this runs at the host's + # COMPILE time, and a correct catalog that reads config or ETS would fail there -- the + # same class as the `ensure_loaded?` defect this check already carries a comment about. + # The map's SHAPE is refused by `BeamMCP.Server.new/1`, which is runtime. Recorded in + # slices/008-catalog-generalization/FINDINGS.md as the one place "refuse at init" is + # answered with "as much as is safe here". unless is_atom(catalog) and catalog != nil and match?({:module, _}, Code.ensure_compiled(catalog)) and - function_exported?(catalog, :all, 0) do + function_exported?(catalog, :capabilities, 0) do raise ArgumentError, """ - BeamMCP.Transport.HTTP requires a :tool_catalog option: a module implementing the - BeamMCP.ToolCatalog behaviour, that is, exporting all/0. + BeamMCP.Transport.HTTP requires a :catalog option: a module implementing the + BeamMCP.Catalog behaviour, that is, exporting capabilities/0. Got: #{inspect(catalog)} """ @@ -769,13 +776,13 @@ if Code.ensure_loaded?(Plug) do end) end - # One lookup, `BeamMCP.ToolCatalog.fetch/2`, is what the core uses to decide whether a tool + # One lookup, `BeamMCP.Catalog.fetch/2`, is what the core uses to decide whether a tool # is callable. The transport asks the same question of the same function: two lookups would # be two answers to "which tool does this name mean", which is the disagreement this whole # header mechanism exists to prevent. defp mirrored_params(%{"method" => "tools/call"} = message, opts) do with name when is_binary(name) <- param(message, "name"), - catalog when not is_nil(catalog) <- opts.server_opts[:tool_catalog], + catalog when not is_nil(catalog) <- opts.server_opts[:catalog], {:ok, entries} <- host_call(fn -> tool_annotations(catalog, name) end) do entries else @@ -798,14 +805,14 @@ if Code.ensure_loaded?(Plug) do # EVERYTHING THE HOST SUPPLIES IS READ INSIDE `host_call/1`, and the boundary is the whole # point of this function existing rather than the three steps sitting in the `with` above. # - # `ToolCatalog.fetch/2` was already wrapped; `spec.input_schema` and the schema walk were + # `Catalog.fetch/2` was already wrapped; `spec.input_schema` and the schema walk were # not. So a host catalog that RAISED kept the request's id, and the same host catalog # returning MALFORMED DATA -- a map where a `%ToolSpec{}` was promised -- raised `KeyError` # one line later, escaped to `call/2`'s rescue and answered `id: null`. Measured in # `slices/002-streamable-http/logs/probe-fault-ids.txt`: # - # host tool_catalog RAISES (header validation) 500 -32603 id=4242 - # host tool_catalog returns a malformed spec 500 -32603 id=nil + # host catalog RAISES (header validation) 500 -32603 id=4242 + # host catalog returns a malformed spec 500 -32603 id=nil # # One host bug, two envelopes, decided by which line it landed on. The line is not the # boundary; the host is. @@ -822,11 +829,11 @@ if Code.ensure_loaded?(Plug) do # `annotation_detail/1` interpolate it. No JSON-derived schema can produce that. It is # recorded as a survivor with this argument rather than pinned by a contrived test. # - # `ToolCatalog.fetch/2` returning anything but `{:ok, spec}` still falls through this + # `Catalog.fetch/2` returning anything but `{:ok, spec}` still falls through this # `with` unchanged, to the caller's `_ -> []`: a tool this catalog does not have mirrors # no parameters, which is not a fault. defp tool_annotations(catalog, name) do - with {:ok, spec} <- ToolCatalog.fetch(catalog, name), + with {:ok, spec} <- Catalog.fetch(catalog, name), entries = annotations(spec.input_schema), :ok <- check_annotations(name, entries) do {:ok, entries} @@ -1103,10 +1110,10 @@ if Code.ensure_loaded?(Plug) do # "every exception, throw and exit out of the host". Two round-5 lanes independently # derived the real population with one command -- # - # grep -n 'authorize_fun\.(\|ToolCatalog.fetch\|Server.handle_message' http.ex + # grep -n 'authorize_fun\.(\|Catalog.fetch\|Server.handle_message' http.ex # # -- and found three sites, of which one was inside that rescue. A host `authorize/1` or - # `tool_catalog` raising `Plug.BadRequestError` still handed its HTTP status to the adapter + # `catalog` raising `Plug.BadRequestError` still handed its HTTP status to the adapter # and dropped the envelope, on a path the docs above call possibly unauthenticated. One host # function raising one exception had two behaviours depending on which catalog lookup fired # first, which is the disagreement the single-lookup discipline exists to prevent. diff --git a/lib/beam_mcp/transport/stdio.ex b/lib/beam_mcp/transport/stdio.ex index 24b1909..24df847 100644 --- a/lib/beam_mcp/transport/stdio.ex +++ b/lib/beam_mcp/transport/stdio.ex @@ -19,12 +19,12 @@ defmodule BeamMCP.Transport.Stdio do option with no default. BeamMCP.Transport.Stdio.run( - tool_catalog: MyApp.Catalog, + catalog: MyApp.Catalog, dispatch: &MyApp.Dispatch.call/3, server_name: "my-app" ) - Options are `BeamMCP.Server.new/1`'s; `:tool_catalog` is required. + Options are `BeamMCP.Server.new/1`'s; `:catalog` is required. """ alias BeamMCP.Server diff --git a/slices/008-catalog-generalization/logs/green-malformed-catalog.txt b/slices/008-catalog-generalization/logs/green-malformed-catalog.txt new file mode 100644 index 0000000..5550ffc --- /dev/null +++ b/slices/008-catalog-generalization/logs/green-malformed-catalog.txt @@ -0,0 +1,10 @@ +=== GREEN: the init-time shape refusal restored === +$ mix test test/beam_mcp/catalog_test.exs +Compiling 1 file (.ex) +Generated beam_mcp app +Running ExUnit with seed: 985007, max_cases: 64 + +............. +Finished in 0.09 seconds (0.09s async, 0.00s sync) +13 tests, 0 failures +TEST_EXIT=0 diff --git a/slices/008-catalog-generalization/logs/probe-fetch-spec.txt b/slices/008-catalog-generalization/logs/probe-fetch-spec.txt new file mode 100644 index 0000000..332f358 --- /dev/null +++ b/slices/008-catalog-generalization/logs/probe-fetch-spec.txt @@ -0,0 +1,11 @@ +=== REPRODUCING the fetch/2 @spec claim on the current tree === +PR #10's ledger claims THREE host shapes raise. Verified here rather than inherited. + +@spec says: {:ok, BeamMCP.ToolSpec.t()} | :error + +host spec.name is a binary, not an atom {:RAISED, ArgumentError} +all/0 returns a non-list map {:RAISED, BadMapError} +all/0 returns nil {:RAISED, Protocol.UndefinedError} +all/0 returns a bare map, not a %ToolSpec{} {:returned, {:ok, %{name: :echo}}} +module is not loaded / does not exist {:RAISED, UndefinedFunctionError} +CONTROL: a well-formed catalog {:returned, {:ok, %BeamMCP.ToolSpec{name: :echo, command_class: :observe, mode: :read_only, description: "d", input_schema: %{"additionalProperties" => true, "properties" => %{}, "type" => "object"}}}} diff --git a/slices/008-catalog-generalization/logs/red-malformed-catalog.txt b/slices/008-catalog-generalization/logs/red-malformed-catalog.txt new file mode 100644 index 0000000..c388b96 --- /dev/null +++ b/slices/008-catalog-generalization/logs/red-malformed-catalog.txt @@ -0,0 +1,55 @@ +=== RED: the init-time shape refusal removed from Server.new/1 === +$ mix test test/beam_mcp/catalog_test.exs +Compiling 1 file (.ex) +Generated beam_mcp app +Running ExUnit with seed: 393777, max_cases: 64 + + + + 1) test a malformed catalog is refused at new/1, not at the first request an absent key is a malformed catalog, not an empty one (BeamMCP.CatalogTest) + test/beam_mcp/catalog_test.exs:118 + Expected exception ArgumentError but nothing was raised + code: assert_raise ArgumentError, ~r/missing required key\(s\): \[:prompts\]/, fn -> + stacktrace: + test/beam_mcp/catalog_test.exs:119: (test) + +... + + 2) test a malformed catalog is refused at new/1, not at the first request :tools must be a list (BeamMCP.CatalogTest) + test/beam_mcp/catalog_test.exs:124 + Expected exception ArgumentError but nothing was raised + code: assert_raise ArgumentError, ~r/:tools must be a list/, fn -> + stacktrace: + test/beam_mcp/catalog_test.exs:125: (test) + +.... + + 3) test a malformed catalog is refused at new/1, not at the first request :tools entries must be %ToolSpec{} (BeamMCP.CatalogTest) + test/beam_mcp/catalog_test.exs:130 + Expected exception ArgumentError but nothing was raised + code: assert_raise ArgumentError, ~r/must all be %BeamMCP.ToolSpec\{\}/, fn -> + stacktrace: + test/beam_mcp/catalog_test.exs:134: (test) + + + + 4) test a malformed catalog is refused at new/1, not at the first request a module that does not export capabilities/0 is refused (BeamMCP.CatalogTest) + test/beam_mcp/catalog_test.exs:145 + Expected exception ArgumentError but nothing was raised + code: assert_raise ArgumentError, ~r/does not export capabilities\/0/, fn -> + stacktrace: + test/beam_mcp/catalog_test.exs:146: (test) + +. + + 5) test a malformed catalog is refused at new/1, not at the first request capabilities/0 must return a map (BeamMCP.CatalogTest) + test/beam_mcp/catalog_test.exs:139 + Expected exception ArgumentError but nothing was raised + code: assert_raise ArgumentError, ~r/must return a map/, fn -> + stacktrace: + test/beam_mcp/catalog_test.exs:140: (test) + + +Finished in 0.09 seconds (0.09s async, 0.00s sync) +13 tests, 5 failures +TEST_EXIT=2 diff --git a/test/beam_mcp/catalog_test.exs b/test/beam_mcp/catalog_test.exs new file mode 100644 index 0000000..9503bf9 --- /dev/null +++ b/test/beam_mcp/catalog_test.exs @@ -0,0 +1,198 @@ +# SPDX-FileCopyrightText: 2026 Sudo Apt Holdings LLC +# SPDX-License-Identifier: Apache-2.0 + +defmodule BeamMCP.CatalogTest do + use ExUnit.Case, async: true + + alias BeamMCP.{Catalog, Server, ToolSpec} + + defp spec(name) do + %ToolSpec{name: name, command_class: :observe, mode: :read_only, description: "d"} + end + + defmodule OnlyHere do + @behaviour BeamMCP.Catalog + @impl true + def capabilities do + %{ + tools: [ + %BeamMCP.ToolSpec{ + name: :only_in_this_catalog, + command_class: :observe, + mode: :read_only, + description: "Exists in no other catalog." + } + ], + resources: [], + prompts: [] + } + end + end + + describe "the single-lookup guarantee" do + # Slice 002 fixed a real defect: `tools/list` honoured the injected catalog and + # `tools/call` consulted a different source, so a tool could be advertised and refused. + # `Catalog.tools/1` is now the one reader and both paths go through it. This pins that by + # EFFECT -- the two answers are compared against each other, not against a constant -- and + # the catalog's tool exists in no other catalog in this suite, so neither path can pass by + # coincidence the way slice 002's original test did. + test "what tools/list advertises is exactly what tools/call will accept" do + state = Server.new(catalog: OnlyHere, dispatch: fn _n, a, _o -> {:ok, a} end) + + {_state, listed} = + Server.handle_message(state, %{ + "jsonrpc" => "2.0", + "id" => 1, + "method" => "tools/list" + }) + + advertised = listed["result"]["tools"] |> Enum.map(& &1["name"]) |> Enum.sort() + + callable = + for name <- advertised do + {_s, r} = + Server.handle_message(state, %{ + "jsonrpc" => "2.0", + "id" => 2, + "method" => "tools/call", + "params" => %{"name" => name, "arguments" => %{}} + }) + + if Map.has_key?(r, "result"), do: name + end + |> Enum.reject(&is_nil/1) + |> Enum.sort() + + assert advertised == ["only_in_this_catalog"] + assert callable == advertised + end + + test "a name the catalog does not advertise is not callable either" do + state = Server.new(catalog: OnlyHere, dispatch: fn _n, a, _o -> {:ok, a} end) + + {_s, r} = + Server.handle_message(state, %{ + "jsonrpc" => "2.0", + "id" => 3, + "method" => "tools/call", + "params" => %{"name" => "not_advertised", "arguments" => %{}} + }) + + assert r["error"]["code"] == -32_601 + end + + test "Catalog.tools/1 and Catalog.fetch/2 read the same source" do + # The structural half of the same guarantee: fetch resolves exactly the specs tools/1 + # returns, so a mutant giving them different sources has something to break. + names = OnlyHere |> Catalog.tools() |> Enum.map(& &1.name) + + for n <- names do + assert {:ok, %ToolSpec{name: ^n}} = Catalog.fetch(OnlyHere, n) + end + + assert Catalog.fetch(OnlyHere, "absent") == :error + end + end + + describe "a malformed catalog is refused at new/1, not at the first request" do + defmodule MissingPrompts do + def capabilities, do: %{tools: [], resources: []} + end + + defmodule ToolsNotAList do + def capabilities, do: %{tools: %{}, resources: [], prompts: []} + end + + defmodule NotSpecs do + def capabilities, do: %{tools: [%{name: :echo}], resources: [], prompts: []} + end + + defmodule NotAMap do + def capabilities, do: [:tools] + end + + defmodule NoCallback do + def unrelated, do: :ok + end + + test "an absent key is a malformed catalog, not an empty one" do + assert_raise ArgumentError, ~r/missing required key\(s\): \[:prompts\]/, fn -> + Server.new(catalog: MissingPrompts) + end + end + + test ":tools must be a list" do + assert_raise ArgumentError, ~r/:tools must be a list/, fn -> + Server.new(catalog: ToolsNotAList) + end + end + + test ":tools entries must be %ToolSpec{}" do + # This one is the silent failure the @spec probe found: fetch/2 RETURNS a bare map + # happily, so without this check a malformed entry reaches dispatch as a struct-shaped + # thing that is not a struct. + assert_raise ArgumentError, ~r/must all be %BeamMCP.ToolSpec\{\}/, fn -> + Server.new(catalog: NotSpecs) + end + end + + test "capabilities/0 must return a map" do + assert_raise ArgumentError, ~r/must return a map/, fn -> + Server.new(catalog: NotAMap) + end + end + + test "a module that does not export capabilities/0 is refused" do + assert_raise ArgumentError, ~r/does not export capabilities\/0/, fn -> + Server.new(catalog: NoCallback) + end + end + end + + describe "Catalog.validate/1 reports rather than raises" do + test "a good catalog is :ok" do + assert Catalog.validate(OnlyHere) == :ok + end + + test "a non-module is refused with the value it got" do + assert {:error, msg} = Catalog.validate("not a module") + assert msg =~ "expected a module" + end + end + + describe "fetch/2's documented raise conditions, held to what was measured" do + defmodule BinaryName do + def capabilities do + %{ + tools: [ + %BeamMCP.ToolSpec{ + name: "echo", + command_class: :observe, + mode: :read_only, + description: "d" + } + ], + resources: [], + prompts: [] + } + end + end + + test "a binary spec.name raises rather than returning :error" do + # The @spec says {:ok, t} | :error and this raises. That is DOCUMENTED rather than + # caught, because catching would make a host bug indistinguishable from "no such tool" -- + # the advertise-versus-call confusion this behaviour exists to prevent. The moduledoc + # lists the measured conditions; this pins one of them so the list cannot go stale + # silently. + assert_raise ArgumentError, fn -> Catalog.fetch(BinaryName, "echo") end + end + + test "an unloaded module raises" do + assert_raise UndefinedFunctionError, fn -> Catalog.fetch(NoSuchCatalogAnywhere, "echo") end + end + end + + test "the spec/1 helper is unused elsewhere and exists only to keep this file readable" do + assert %ToolSpec{name: :x} = spec(:x) + end +end diff --git a/test/beam_mcp/error_payload_test.exs b/test/beam_mcp/error_payload_test.exs index 9cecece..013fc9d 100644 --- a/test/beam_mcp/error_payload_test.exs +++ b/test/beam_mcp/error_payload_test.exs @@ -14,10 +14,12 @@ defmodule BeamMCP.ErrorPayloadTest do alias BeamMCP.Server defmodule Catalog do - @behaviour BeamMCP.ToolCatalog + @behaviour BeamMCP.Catalog @impl true - def all do + def capabilities, do: %{tools: all_tools(), resources: [], prompts: []} + + defp all_tools do [ %BeamMCP.ToolSpec{ name: :widget, @@ -35,7 +37,7 @@ defmodule BeamMCP.ErrorPayloadTest do end defp call(dispatch, args) do - Server.new(tool_catalog: Catalog, dispatch: dispatch) + Server.new(catalog: Catalog, dispatch: dispatch) |> Server.handle_message(%{ "jsonrpc" => "2.0", "id" => 1, diff --git a/test/beam_mcp/injection_test.exs b/test/beam_mcp/injection_test.exs index d54b6fe..9d4f2e0 100644 --- a/test/beam_mcp/injection_test.exs +++ b/test/beam_mcp/injection_test.exs @@ -14,10 +14,12 @@ defmodule BeamMCP.InjectionTest do alias BeamMCP.Server defmodule CatalogWithNovelTool do - @behaviour BeamMCP.ToolCatalog + @behaviour BeamMCP.Catalog @impl true - def all do + def capabilities, do: %{tools: all_tools(), resources: [], prompts: []} + + defp all_tools do [ %BeamMCP.ToolSpec{ name: :novel_tool, @@ -30,7 +32,7 @@ defmodule BeamMCP.InjectionTest do end test "an injected catalog's tool is advertised by tools/list" do - state = Server.new(tool_catalog: CatalogWithNovelTool) + state = Server.new(catalog: CatalogWithNovelTool) {_s, resp} = Server.handle_message(state, %{"jsonrpc" => "2.0", "id" => 1, "method" => "tools/list"}) @@ -46,7 +48,7 @@ defmodule BeamMCP.InjectionTest do {:ok, %{}} end - state = Server.new(dispatch: dispatch, tool_catalog: CatalogWithNovelTool) + state = Server.new(dispatch: dispatch, catalog: CatalogWithNovelTool) {_s, resp} = Server.handle_message(state, %{ diff --git a/test/beam_mcp/negotiation_test.exs b/test/beam_mcp/negotiation_test.exs index 5aa6992..c035d8f 100644 --- a/test/beam_mcp/negotiation_test.exs +++ b/test/beam_mcp/negotiation_test.exs @@ -23,10 +23,12 @@ defmodule BeamMCP.NegotiationTest do @legacy "2025-11-25" defmodule Catalog do - @behaviour BeamMCP.ToolCatalog + @behaviour BeamMCP.Catalog @impl true - def all do + def capabilities, do: %{tools: all_tools(), resources: [], prompts: []} + + defp all_tools do [ %BeamMCP.ToolSpec{ name: :echo, @@ -38,7 +40,7 @@ defmodule BeamMCP.NegotiationTest do end end - defp state, do: Server.new(tool_catalog: Catalog, dispatch: fn _n, a, _o -> {:ok, a} end) + defp state, do: Server.new(catalog: Catalog, dispatch: fn _n, a, _o -> {:ok, a} end) defp send_msg(msg), do: state() |> Server.handle_message(msg) |> elem(1) diff --git a/test/beam_mcp/readme_claims_test.exs b/test/beam_mcp/readme_claims_test.exs index 81c3419..ef1a63b 100644 --- a/test/beam_mcp/readme_claims_test.exs +++ b/test/beam_mcp/readme_claims_test.exs @@ -42,10 +42,12 @@ defmodule BeamMCP.ReadmeClaimsTest do @vkey "io.modelcontextprotocol/protocolVersion" defmodule Catalog do - @behaviour BeamMCP.ToolCatalog + @behaviour BeamMCP.Catalog @impl true - def all do + def capabilities, do: %{tools: all_tools(), resources: [], prompts: []} + + defp all_tools do [ %BeamMCP.ToolSpec{ name: :echo, @@ -62,7 +64,7 @@ defmodule BeamMCP.ReadmeClaimsTest do me = self() Server.new( - tool_catalog: Catalog, + catalog: Catalog, dispatch: fn name, args, _opts -> send(me, {:dispatched, name}) {:ok, args} @@ -342,6 +344,28 @@ defmodule BeamMCP.ReadmeClaimsTest do end end + describe "the BeamMCP.Catalog README claims" do + test "the catalog contract's behavioural sentences are pinned to the sentences that state them" do + # Fragments chosen to sit within one wrapped line, checked with grep against the file + # first -- slice 007 lost three fragments to the README's hard wrap. + claims("the host names what it offers") + claims("An absent key is a malformed catalog, not an empty") + claims("refuses it at startup rather than at the first request") + end + + test "the refusal the README promises is the refusal the code performs" do + # Held to the code, not only to itself. The README says an absent key is refused at + # startup; this is that startup, and it is the sentence's only evidence. + defmodule ReadmeMissingKey do + def capabilities, do: %{tools: [], resources: []} + end + + assert_raise ArgumentError, ~r/missing required key\(s\)/, fn -> + BeamMCP.Server.new(catalog: ReadmeMissingKey) + end + end + end + describe "the :authorize_body README claims" do test "every behavioural sentence about the hook is pinned to the sentence that states it" do # FRAGMENTS ARE CHOSEN TO SIT WITHIN ONE WRAPPED LINE. The README is hard-wrapped, and @@ -449,9 +473,11 @@ defmodule BeamMCP.ReadmeClaimsTest do end defmodule WeatherCatalog do - @behaviour BeamMCP.ToolCatalog + @behaviour BeamMCP.Catalog @impl true - def all do + def capabilities, do: %{tools: all_tools(), resources: [], prompts: []} + + defp all_tools do Process.get(:catalog_fun).() end end @@ -460,7 +486,7 @@ defmodule BeamMCP.ReadmeClaimsTest do state = BeamMCP.Server.new( - tool_catalog: WeatherCatalog, + catalog: WeatherCatalog, dispatch: fn _name, args, _opts -> send(me, {:dispatched, args}) {:ok, args} @@ -536,7 +562,7 @@ defmodule BeamMCP.ReadmeClaimsTest do # The core is dual-era and still serves these on stdio -- that is the README's point, and # the reason the refusal lives in the transport. Asserted so the sentence explaining the # split is pinned to a core that actually still answers them. - state = Server.new(tool_catalog: Catalog, dispatch: fn _, a, _ -> {:ok, a} end) + state = Server.new(catalog: Catalog, dispatch: fn _, a, _ -> {:ok, a} end) {_state, initialize} = Server.handle_message(state, %{ @@ -572,10 +598,12 @@ defmodule BeamMCP.ReadmeClaimsTest do claims("until the schema is corrected") defmodule ForbiddenAnnotationCatalog do - @behaviour BeamMCP.ToolCatalog + @behaviour BeamMCP.Catalog @impl true - def all do + def capabilities, do: %{tools: all_tools(), resources: [], prompts: []} + + defp all_tools do [ %BeamMCP.ToolSpec{ name: :echo, @@ -594,7 +622,7 @@ defmodule BeamMCP.ReadmeClaimsTest do end end - conn = http_request(tool_catalog: ForbiddenAnnotationCatalog) + conn = http_request(catalog: ForbiddenAnnotationCatalog) assert conn.status == 500 assert Jason.decode!(conn.resp_body)["error"]["code"] == -32_603 @@ -653,7 +681,7 @@ defmodule BeamMCP.ReadmeClaimsTest do |> Plug.Conn.put_req_header("mcp-name", "echo") |> HTTP.call( HTTP.init( - tool_catalog: Catalog, + catalog: Catalog, dispatch: fn _n, a, _o -> {:ok, a} end, authorize: fn _conn -> :ok end, allowed_origins: :any @@ -681,7 +709,7 @@ defmodule BeamMCP.ReadmeClaimsTest do opts = Keyword.merge( [ - tool_catalog: Catalog, + catalog: Catalog, dispatch: fn _n, a, _o -> {:ok, a} end, authorize: fn _conn -> :ok end, allowed_origins: :any diff --git a/test/beam_mcp/server_test.exs b/test/beam_mcp/server_test.exs index cb8e940..aa0ff9d 100644 --- a/test/beam_mcp/server_test.exs +++ b/test/beam_mcp/server_test.exs @@ -6,8 +6,10 @@ defmodule BeamMCP.ServerTest do alias BeamMCP.Server - defmodule FakeToolCatalog do - def all do + defmodule FakeCatalog do + def capabilities, do: %{tools: all_tools(), resources: [], prompts: []} + + defp all_tools do [ %BeamMCP.ToolSpec{ name: :get_latest_alerts, @@ -36,7 +38,7 @@ defmodule BeamMCP.ServerTest do end test "initialize advertises MCP tool capability" do - state = Server.new(tool_catalog: FakeToolCatalog) + state = Server.new(catalog: FakeCatalog) {next_state, response} = Server.handle_message(state, %{"jsonrpc" => "2.0", "id" => 1, "method" => "initialize"}) @@ -50,7 +52,7 @@ defmodule BeamMCP.ServerTest do end test "tools/list exposes MCP-compatible tool metadata" do - state = Server.new(tool_catalog: FakeToolCatalog) + state = Server.new(catalog: FakeCatalog) {_next_state, response} = Server.handle_message(state, %{"jsonrpc" => "2.0", "id" => 2, "method" => "tools/list"}) @@ -71,7 +73,7 @@ defmodule BeamMCP.ServerTest do {:ok, %{received: args[:action_class], case_id: args[:case_id], target: args[:target]}} end - state = Server.new(dispatch: dispatch, tool_catalog: FakeToolCatalog) + state = Server.new(dispatch: dispatch, catalog: FakeCatalog) {_next_state, response} = Server.handle_message(state, %{ @@ -105,7 +107,7 @@ defmodule BeamMCP.ServerTest do end test "unknown tools return a JSON-RPC error" do - state = Server.new(tool_catalog: FakeToolCatalog) + state = Server.new(catalog: FakeCatalog) {_next_state, response} = Server.handle_message(state, %{ diff --git a/test/beam_mcp/tool_spec_schema_test.exs b/test/beam_mcp/tool_spec_schema_test.exs index 309ee9a..da140a3 100644 --- a/test/beam_mcp/tool_spec_schema_test.exs +++ b/test/beam_mcp/tool_spec_schema_test.exs @@ -26,10 +26,12 @@ defmodule BeamMCP.ToolSpecSchemaTest do } defmodule CatalogWithSchema do - @behaviour BeamMCP.ToolCatalog + @behaviour BeamMCP.Catalog @impl true - def all do + def capabilities, do: %{tools: all_tools(), resources: [], prompts: []} + + defp all_tools do [ %BeamMCP.ToolSpec{ name: :inspect_widget, @@ -48,7 +50,7 @@ defmodule BeamMCP.ToolSpecSchemaTest do end defp state(dispatch \\ fn _n, _a, _o -> {:ok, %{}} end) do - Server.new(dispatch: dispatch, tool_catalog: CatalogWithSchema) + Server.new(dispatch: dispatch, catalog: CatalogWithSchema) end defp call(state, args) do diff --git a/test/beam_mcp/transport/http_bandit_test.exs b/test/beam_mcp/transport/http_bandit_test.exs index 89d271c..79a93a4 100644 --- a/test/beam_mcp/transport/http_bandit_test.exs +++ b/test/beam_mcp/transport/http_bandit_test.exs @@ -59,9 +59,11 @@ defmodule BeamMCP.Transport.HTTPBanditTest do @declared_but_unsent 5_000_000 defmodule Catalog do - @behaviour BeamMCP.ToolCatalog + @behaviour BeamMCP.Catalog @impl true - def all do + def capabilities, do: %{tools: all_tools(), resources: [], prompts: []} + + defp all_tools do [ %BeamMCP.ToolSpec{ name: :echo, @@ -80,7 +82,7 @@ defmodule BeamMCP.Transport.HTTPBanditTest do plug_opts = Keyword.merge( [ - tool_catalog: Catalog, + catalog: Catalog, dispatch: fn _n, a, _o -> {:ok, a} end, authorize: fn _conn -> :ok end, allowed_origins: :any diff --git a/test/beam_mcp/transport/http_test.exs b/test/beam_mcp/transport/http_test.exs index 2acf70c..9a2e4a0 100644 --- a/test/beam_mcp/transport/http_test.exs +++ b/test/beam_mcp/transport/http_test.exs @@ -18,9 +18,11 @@ defmodule BeamMCP.Transport.HTTPTest do @hdr "mcp-protocol-version" defmodule Catalog do - @behaviour BeamMCP.ToolCatalog + @behaviour BeamMCP.Catalog @impl true - def all do + def capabilities, do: %{tools: all_tools(), resources: [], prompts: []} + + defp all_tools do [ %BeamMCP.ToolSpec{ name: :echo, @@ -57,7 +59,7 @@ defmodule BeamMCP.Transport.HTTPTest do defp opts(extra \\ []) do Keyword.merge( [ - tool_catalog: Catalog, + catalog: Catalog, dispatch: fn _n, a, _o -> {:ok, a} end, authorize: fn _conn -> :ok end, allowed_origins: :any @@ -106,20 +108,20 @@ defmodule BeamMCP.Transport.HTTPTest do describe "the two options the package refuses to default" do test "init/1 raises without :authorize, naming what the host must decide" do assert_raise ArgumentError, ~r/requires an :authorize option, and it has no default/, fn -> - HTTP.init(tool_catalog: Catalog, allowed_origins: :any) + HTTP.init(catalog: Catalog, allowed_origins: :any) end end test "init/1 raises without :allowed_origins" do assert_raise ArgumentError, ~r/requires an :allowed_origins option/, fn -> - HTTP.init(tool_catalog: Catalog, authorize: fn _ -> :ok end) + HTTP.init(catalog: Catalog, authorize: fn _ -> :ok end) end end test "the failure is at init, not at request time" do # The distinction is the whole contract: a host that forgets cannot start, rather than # serving unauthorised requests until someone notices. - assert_raise ArgumentError, fn -> HTTP.init(tool_catalog: Catalog) end + assert_raise ArgumentError, fn -> HTTP.init(catalog: Catalog) end end test "an authorize that refuses stops the request before any message is handled" do @@ -697,7 +699,7 @@ defmodule BeamMCP.Transport.HTTPTest do # The transport matches the core's `Method not found:` prefix. That coupling is real, so # it is asserted: a reword fails this test instead of silently turning every 404 into a # 200 with no test noticing. - state = BeamMCP.Server.new(tool_catalog: Catalog, dispatch: fn _, a, _ -> {:ok, a} end) + state = BeamMCP.Server.new(catalog: Catalog, dispatch: fn _, a, _ -> {:ok, a} end) {_, unimplemented} = BeamMCP.Server.handle_message(state, %{ @@ -747,12 +749,12 @@ defmodule BeamMCP.Transport.HTTPTest do end end - describe ":tool_catalog is checked for the behaviour, not for truthiness" do - test "a value that is not a module exporting all/0 raises at init, not at the first request" do + describe ":catalog is checked for the behaviour, not for truthiness" do + test "a value that is not a module exporting capabilities/0 raises at init, not at the first request" do for bad <- [true, "MyApp.Catalog", Enum, :not_a_module] do - assert_raise ArgumentError, ~r/BeamMCP.ToolCatalog behaviour/, fn -> + assert_raise ArgumentError, ~r/BeamMCP.Catalog behaviour/, fn -> HTTP.init( - tool_catalog: bad, + catalog: bad, dispatch: fn _, a, _ -> {:ok, a} end, authorize: fn _ -> :ok end, allowed_origins: :any @@ -1010,7 +1012,7 @@ defmodule BeamMCP.Transport.HTTPTest do # ROUND 5, FOUND BY TWO LANES INDEPENDENTLY. The rule was applied to dispatch/3 and the # comment beside it claimed "every exception, throw and exit out of the host". The # population was LISTED, not derived. Derived by command, host-supplied code runs at three - # sites in the request path -- authorize/1, ToolCatalog.fetch/2 and Server.handle_message/2 + # sites in the request path -- authorize/1, Catalog.fetch/2 and Server.handle_message/2 # -- and only the third was inside the rescue that was fixed. # # authorize/1 is the branch this module's own docs call possibly unauthenticated. @@ -1023,7 +1025,7 @@ defmodule BeamMCP.Transport.HTTPTest do assert body!(conn)["error"]["code"] == -32_603 end - test "a host tool_catalog's exception does not choose the HTTP status either" do + test "a host catalog's exception does not choose the HTTP status either" do # The catalog raises ONCE, and that is the whole design of this test. # # The first version raised on every call, so `Server.handle_message/2`'s own lookup raised @@ -1037,9 +1039,11 @@ defmodule BeamMCP.Transport.HTTPTest do # succeeds. So if the fault were swallowed, this request would be served rather than # refused -- which is exactly the mutant. defmodule FlakyCatalog do - @behaviour BeamMCP.ToolCatalog + @behaviour BeamMCP.Catalog @impl true - def all do + def capabilities, do: %{tools: all_tools(), resources: [], prompts: []} + + defp all_tools do case Process.put(:flaky_called, true) do nil -> raise Plug.BadRequestError @@ -1065,7 +1069,7 @@ defmodule BeamMCP.Transport.HTTPTest do end Process.delete(:flaky_called) - o = opts(tool_catalog: FlakyCatalog) + o = opts(catalog: FlakyCatalog) # A header that LIES about the body value: 99 against a body of 42. If mirroring is # silently disabled by the swallowed fault, this is dispatched. @@ -1081,14 +1085,16 @@ defmodule BeamMCP.Transport.HTTPTest do assert body!(conn)["id"] == 92 end - test "a host tool_catalog that THROWS is answered with the id, not just rescued" do + test "a host catalog that THROWS is answered with the id, not just rescued" do # host_call/1 has a `catch` as well as a `rescue`, and a mutant dropping the `catch` # survived the suite: the throw propagated to call/2's own catch, which answers with # `id: null`. Status and code were identical, so only the id moves under that mutation. defmodule ThrowingCatalog do - @behaviour BeamMCP.ToolCatalog + @behaviour BeamMCP.Catalog @impl true - def all do + def capabilities, do: %{tools: all_tools(), resources: [], prompts: []} + + defp all_tools do case Process.put(:throwing_called, true) do nil -> throw(:catalog_unavailable) @@ -1120,7 +1126,7 @@ defmodule BeamMCP.Transport.HTTPTest do post( body, call_headers([{"mcp-param-maxrows", "99"}]), - opts(tool_catalog: ThrowingCatalog) + opts(catalog: ThrowingCatalog) ) assert conn.status == 500 @@ -1159,28 +1165,29 @@ defmodule BeamMCP.Transport.HTTPTest do end test "a host catalog returning a malformed spec answers with the request's id" do - # ONE HOST BUG, ONE ENVELOPE. A host `tool_catalog` that RAISES is answered by + # ONE HOST BUG, ONE ENVELOPE. A host `catalog` that RAISES is answered by # `check_param_headers/4`'s fault branch and keeps the request's id; the same host # catalog returning MALFORMED DATA -- a spec-shaped map that is not a `%ToolSpec{}` -- # used to raise `KeyError` on the `spec.input_schema` read one line later, escape to # `call/2`'s rescue and answer `id: null`. Measured before the fix, in # `slices/002-streamable-http/logs/probe-fault-ids.txt`: # - # host tool_catalog RAISES (header validation) 500 -32603 id=4242 - # host tool_catalog returns a malformed spec 500 -32603 id=nil + # host catalog RAISES (header validation) 500 -32603 id=4242 + # host catalog returns a malformed spec 500 -32603 id=nil # # Which envelope a host bug got was decided by which line it landed on. The id is the # ONLY thing that distinguishes the two routes, which is why it is what this asserts: # status and code are identical on both, so asserting those alone is an anchor that # cannot move. defmodule BadSpecCatalog do - @behaviour BeamMCP.ToolCatalog + @behaviour BeamMCP.Catalog @impl true - def all, do: [%{name: :echo}] + def capabilities, do: %{tools: all_tools(), resources: [], prompts: []} + defp all_tools, do: [%{name: :echo}] end body = call_body(%{}) |> Map.put("id", 4242) - conn = post(body, call_headers([]), opts(tool_catalog: BadSpecCatalog)) + conn = post(body, call_headers([]), opts(catalog: BadSpecCatalog)) assert conn.status == 500 assert conn.resp_body != "" @@ -1195,9 +1202,11 @@ defmodule BeamMCP.Transport.HTTPTest do end defmodule InvalidUtf8HeaderNameCatalog do - @behaviour BeamMCP.ToolCatalog + @behaviour BeamMCP.Catalog @impl true - def all do + def capabilities, do: %{tools: all_tools(), resources: [], prompts: []} + + defp all_tools do [ %BeamMCP.ToolSpec{ name: :echo, @@ -1243,7 +1252,7 @@ defmodule BeamMCP.Transport.HTTPTest do # A property with an unencodable annotation name is still a recorded gap -- the tool is # advertised and uncallable -- and it is filed in this slice's FINDINGS.md rather than # described here as filed. - o = opts(tool_catalog: InvalidUtf8HeaderNameCatalog) + o = opts(catalog: InvalidUtf8HeaderNameCatalog) body = call_body(%{"value" => "x"}) |> Map.put("id", 4243) conn = post(body, call_headers([]), o) @@ -1623,9 +1632,11 @@ defmodule BeamMCP.Transport.HTTPTest do # and the 400 blames the caller for the host's schema. defmodule FloatAnnotationCatalog do - @behaviour BeamMCP.ToolCatalog + @behaviour BeamMCP.Catalog @impl true - def all do + def capabilities, do: %{tools: all_tools(), resources: [], prompts: []} + + defp all_tools do [ %BeamMCP.ToolSpec{ name: :echo, @@ -1642,9 +1653,11 @@ defmodule BeamMCP.Transport.HTTPTest do end defmodule ObjectAnnotationCatalog do - @behaviour BeamMCP.ToolCatalog + @behaviour BeamMCP.Catalog @impl true - def all do + def capabilities, do: %{tools: all_tools(), resources: [], prompts: []} + + defp all_tools do [ %BeamMCP.ToolSpec{ name: :echo, @@ -1671,7 +1684,7 @@ defmodule BeamMCP.Transport.HTTPTest do o = opts( - tool_catalog: FloatAnnotationCatalog, + catalog: FloatAnnotationCatalog, dispatch: fn n, a, _ -> send(me, {:dispatched, n, a}) && {:ok, a} end ) @@ -1706,7 +1719,7 @@ defmodule BeamMCP.Transport.HTTPTest do test "the refusal does not depend on the caller sending the header" do # Both directions were closed before, and both must now land on the same answer: the # verdict is a property of the SCHEMA, so it cannot depend on what the caller sent. - o = opts(tool_catalog: FloatAnnotationCatalog) + o = opts(catalog: FloatAnnotationCatalog) ExUnit.CaptureLog.capture_log(fn -> assert post(call_body(%{"ratio" => 1.5}), call_headers([]), o).status == 500 @@ -1714,7 +1727,7 @@ defmodule BeamMCP.Transport.HTTPTest do end test "an annotated `object` property is the same fault" do - o = opts(tool_catalog: ObjectAnnotationCatalog) + o = opts(catalog: ObjectAnnotationCatalog) ExUnit.CaptureLog.capture_log(fn -> conn = post(call_body(%{"obj" => %{"k" => "v"}}), call_headers([]), o) @@ -1730,9 +1743,11 @@ defmodule BeamMCP.Transport.HTTPTest do # would turn a caller sending the wrong shape into a host fault -- the wrong side of the # boundary, and the mistake this fix exists to stop making in the other direction. defmodule UntypedAnnotationCatalog do - @behaviour BeamMCP.ToolCatalog + @behaviour BeamMCP.Catalog @impl true - def all do + def capabilities, do: %{tools: all_tools(), resources: [], prompts: []} + + defp all_tools do [ %BeamMCP.ToolSpec{ name: :echo, @@ -1748,7 +1763,7 @@ defmodule BeamMCP.Transport.HTTPTest do end end - o = opts(tool_catalog: UntypedAnnotationCatalog) + o = opts(catalog: UntypedAnnotationCatalog) body = call_body(%{"loose" => "v"}) assert post(body, call_headers([{"mcp-param-loose", "v"}]), o).status == 200 @@ -1773,9 +1788,11 @@ defmodule BeamMCP.Transport.HTTPTest do # schema. defmodule CollidingCatalog do - @behaviour BeamMCP.ToolCatalog + @behaviour BeamMCP.Catalog @impl true - def all do + def capabilities, do: %{tools: all_tools(), resources: [], prompts: []} + + defp all_tools do [ %BeamMCP.ToolSpec{ name: :collide, @@ -1795,9 +1812,11 @@ defmodule BeamMCP.Transport.HTTPTest do end defmodule NestedCollisionCatalog do - @behaviour BeamMCP.ToolCatalog + @behaviour BeamMCP.Catalog @impl true - def all do + def capabilities, do: %{tools: all_tools(), resources: [], prompts: []} + + defp all_tools do [ %BeamMCP.ToolSpec{ name: :collide, @@ -1834,7 +1853,7 @@ defmodule BeamMCP.Transport.HTTPTest do o = opts( - tool_catalog: CollidingCatalog, + catalog: CollidingCatalog, dispatch: fn n, a, _ -> send(me, {:dispatched, n, a}) && {:ok, a} end ) @@ -1869,7 +1888,7 @@ defmodule BeamMCP.Transport.HTTPTest do test "the collision is refused however the caller writes the header, or omits it" do # The verdict is a property of the schema. A caller who supplies both spellings, or # neither, gets the same answer -- otherwise the check is being decided by the request. - o = opts(tool_catalog: CollidingCatalog) + o = opts(catalog: CollidingCatalog) body = collide_body(%{"alpha" => "A", "beta" => "B"}) ExUnit.CaptureLog.capture_log(fn -> @@ -1884,7 +1903,7 @@ defmodule BeamMCP.Transport.HTTPTest do test "a nested annotation colliding with an outer one is the same fault" do # This is the `Map.merge` half rather than the `Map.put` half: the nested walk's result # was merged over the accumulator, so a nested annotation silently won. - o = opts(tool_catalog: NestedCollisionCatalog) + o = opts(catalog: NestedCollisionCatalog) ExUnit.CaptureLog.capture_log(fn -> conn = diff --git a/test/beam_mcp/transport/stdio_test.exs b/test/beam_mcp/transport/stdio_test.exs index b6ae272..6f97520 100644 --- a/test/beam_mcp/transport/stdio_test.exs +++ b/test/beam_mcp/transport/stdio_test.exs @@ -19,10 +19,12 @@ defmodule BeamMCP.Transport.StdioTest do alias BeamMCP.Transport.Stdio defmodule Catalog do - @behaviour BeamMCP.ToolCatalog + @behaviour BeamMCP.Catalog @impl true - def all do + def capabilities, do: %{tools: all_tools(), resources: [], prompts: []} + + defp all_tools do [ %BeamMCP.ToolSpec{ name: :echo, @@ -34,7 +36,7 @@ defmodule BeamMCP.Transport.StdioTest do end end - defp opts, do: [tool_catalog: Catalog, dispatch: fn _n, a, _o -> {:ok, a} end] + defp opts, do: [catalog: Catalog, dispatch: fn _n, a, _o -> {:ok, a} end] # Drive the real loop over a StringIO standing in for stdio. The group leader is what # `IO.binread(:stdio, _)` and `IO.binwrite(:stdio, _)` resolve to, so this exercises the From e9a9952a1eea22a767e96be37ea560189214c295 Mon Sep 17 00:00:00 2001 From: Ayla Croft Date: Tue, 8 Sep 2026 13:40:51 -0400 Subject: [PATCH 2/6] Add the two mutants that hold the catalog contract's shape and its one reader Mcat1 drops :prompts from @required_keys -- one key out of three, no message change, no behaviour change for a correct host. If the smallest relaxation of the check survives, the check is decoration. It dies twice over: at the startup refusal, and independently at the README claim test, because the README promises that refusal. Mcat2 leaves tools/list reading the catalog and makes tools/call answer "which tool does this name mean" from the request instead -- slice 002's defect in the direction it actually occurred, advertise and call on different sources. Mcat2's first version returned {:ok, spec} unconditionally. Elixir narrowed find_tool/2's return to that shape, called the caller's `:error ->` clause unreachable, and --warnings-as-errors failed the build: the score line came back with no test count at all. That is a compiler kill, which records a kill that never happened. It keeps an :error branch now so the suite scores it rather than the compiler -- corrected so it could be scored, not weakened until it killed. Every name a client can actually send is still callable under it. Both targets differ from the harness default, so each is scored by its own invocation with TARGET set; the mutant headers say which. Signed-off-by: Ayla Croft --- tools/mutants/Mcat1.py | 33 ++++++++++++++++++++++++++ tools/mutants/Mcat2.py | 53 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 86 insertions(+) create mode 100644 tools/mutants/Mcat1.py create mode 100644 tools/mutants/Mcat2.py diff --git a/tools/mutants/Mcat1.py b/tools/mutants/Mcat1.py new file mode 100644 index 0000000..d5a742b --- /dev/null +++ b/tools/mutants/Mcat1.py @@ -0,0 +1,33 @@ +# SPDX-FileCopyrightText: 2026 Sudo Apt Holdings LLC +# SPDX-License-Identifier: Apache-2.0 +# +# Run by tools/mutate.sh, which passes the file to mutate as argv[1]. Never hard-code +# a path here: a mutant that names one machine's worktree is the unresolvable +# `$S/mut.sh` that slice 003's record cites, in another file. +# +# TARGET=lib/beam_mcp/catalog.ex tools/mutate.sh score Mcat1 +# +# Mcat1 -- DROP A REQUIRED KEY FROM THE SHAPE. `@required_keys` loses `:prompts`, so a host +# whose capabilities/0 omits it is accepted as well formed. +# +# This is the mutation the whole slice is about. The reason `resources` and `prompts` are +# required-and-may-be-empty is that an absent key and an empty one are different claims: absent +# means the host never wrote the key, empty means it wrote it and has none. Relaxing the check +# by one key makes the two indistinguishable again, and the only place that difference is +# visible is startup -- by the first request the map has already been read. +# +# It is deliberately the smallest possible relaxation: one key out of three, no message change, +# no behaviour change for a correct host. A checker that only rejects the malformed catalogs +# nobody writes is not a checker. +import sys + +p = sys.argv[1] +s = open(p).read() + +old = " @required_keys [:tools, :resources, :prompts]" +new = " @required_keys [:tools, :resources]" + +if s.count(old) != 1: + sys.exit("Mcat1: anchor found %d times" % s.count(old)) + +open(p, "w").write(s.replace(old, new, 1)) diff --git a/tools/mutants/Mcat2.py b/tools/mutants/Mcat2.py new file mode 100644 index 0000000..1002963 --- /dev/null +++ b/tools/mutants/Mcat2.py @@ -0,0 +1,53 @@ +# SPDX-FileCopyrightText: 2026 Sudo Apt Holdings LLC +# SPDX-License-Identifier: Apache-2.0 +# +# Run by tools/mutate.sh, which passes the file to mutate as argv[1]. Never hard-code +# a path here. +# +# TARGET=lib/beam_mcp/server.ex tools/mutate.sh score Mcat2 +# +# Mcat2 -- GIVE ADVERTISE AND CALL DIFFERENT SOURCES. `tools/list` keeps reading the injected +# catalog through `Catalog.tools/1`; `tools/call` stops reading it and answers "which tool does +# this name mean" from the request itself, synthesising a spec for whatever was asked for. +# +# This is slice 002's defect restored in the direction it actually occurred: one path honoured +# the injected catalog and the other consulted something else, so the set a server advertises +# and the set it accepts are no longer the same set. The single-lookup guarantee in +# `BeamMCP.Catalog.tools/1` exists to make that unrepresentable, and a guarantee is only worth +# the mutation that has to break to violate it. +# +# It is also the shape CONVENTIONS.md names directly: deriving a population from +# attacker-controlled input is not deriving a population. Here the population of callable tools +# becomes the name in the request. +# +# WHY THE EMPTY-NAME BRANCH IS HERE. The first version of this mutant returned {:ok, spec} +# unconditionally. Elixir's type checker then narrowed find_tool/2's return to that one shape, +# declared the caller's `:error ->` clause unreachable, and --warnings-as-errors failed the +# build: the score line came back with no test count at all. That is a compiler kill, which +# CONVENTIONS.md says records a kill that never happened -- no test ever ran. The branch below +# keeps the return a union so the mutation is scored by the suite instead. It costs the mutant +# nothing: every name a client can actually send is still callable. +import sys + +p = sys.argv[1] +s = open(p).read() + +old = " defp find_tool(state, name), do: Catalog.fetch(state.catalog, name)" +new = """ defp find_tool(_state, name) do + if name == "" do + :error + else + {:ok, + %BeamMCP.ToolSpec{ + name: String.to_atom(name), + command_class: :observe, + mode: :read_only, + description: "synthesised from the request rather than read from the catalog" + }} + end + end""" + +if s.count(old) != 1: + sys.exit("Mcat2: anchor found %d times" % s.count(old)) + +open(p, "w").write(s.replace(old, new, 1)) From 52ca971008e16619afa44b30908237f42d47681a Mon Sep 17 00:00:00 2001 From: Ayla Croft Date: Tue, 8 Sep 2026 13:40:51 -0400 Subject: [PATCH 3/6] Record slice 008: the three decisions, the corrected ledger, and both scores PLAN and FINDINGS in slices/008-catalog-generalization/, plus the scored mutation archive. The PLAN specifies no review rounds, and says so rather than leaving the absence to be inferred: the brief was a single pass, nobody is scoring this slice, and a round boundary with no reviewer behind it is a heading. The mutants and the gate stand in. FINDINGS records four things that were measured rather than assumed: - PR #10's ledger says three host shapes make fetch/2 raise. Four do, and a fifth case the ledger missed is worse than any of them: a bare map entry RETURNS {:ok, %{name: :echo}} successfully, violating the @spec silently. All five are refused at Server.new/1 now. - The brief's call-site list was wrong in both directions -- two readers missing, one listed that is not affected because it consumes a ToolSpec. - Mcat2's first version was a compiler kill, left in the log above the corrected run. - 4a2d2e9's message typed "185 after" where the run printed 187. The commit was amended before anything was pushed; recorded here too, because an amended commit leaves no trace and the defect is the habit, not the digit. Also stated plainly: the Linear issue was filed at record time, not before the code, which is a departure from slice 007 and costs something real -- nothing external witnessed the acceptance criteria before they were met. Signed-off-by: Ayla Croft --- slices/008-catalog-generalization/FINDINGS.md | 169 ++++++++++++++++++ slices/008-catalog-generalization/PLAN.md | 85 +++++++++ .../logs/mutation-catalog.txt | 43 +++++ 3 files changed, 297 insertions(+) create mode 100644 slices/008-catalog-generalization/FINDINGS.md create mode 100644 slices/008-catalog-generalization/PLAN.md create mode 100644 slices/008-catalog-generalization/logs/mutation-catalog.txt diff --git a/slices/008-catalog-generalization/FINDINGS.md b/slices/008-catalog-generalization/FINDINGS.md new file mode 100644 index 0000000..c4cfcd3 --- /dev/null +++ b/slices/008-catalog-generalization/FINDINGS.md @@ -0,0 +1,169 @@ + + +# Slice 008 — findings + +## The three decisions, made deliberately + +### (a) Migration path: a clean break, no deprecated `ToolCatalog` delegate + +Two reasons, and the second is the load-bearing one. + +The 0.x policy already protects consumers. The README recommends `~> 0.3.0`, which **excludes** +`0.4.0` — so no existing consumer is upgraded into this break by resolution. A host that wants it +asks for it, which is exactly what a break at minor is for. + +And a delegate would create **a second reader**, which decision (b) forbids. `ToolCatalog.fetch/2` +delegating to `Catalog.fetch/2` is harmless; `ToolCatalog.all/0` living beside +`Catalog.capabilities/0` is not, because a host that implements both can make them disagree and +nothing would catch it. The compatibility shim would reintroduce the defect the contract exists +to prevent, in order to soften a break the version policy already handles. + +The callback is also **renamed**, `all/0` → `capabilities/0`, and that is part of the same +decision. Keeping the name while changing the return from a list to a map compiles against every +existing host and fails at the first request with a `BadMapError` — a silent shape change, which +is the defect class this repository keeps finding. Renaming makes the break arrive at compile +time as an unimplemented callback, which is the loudest place it can arrive. + +### (b) The single-lookup guarantee: preserved, and structurally stronger than before + +Before this slice, `tools/list` called `catalog.all()` and `ToolCatalog.fetch/2` called +`catalog.all()` — **two call sites of the host's function**, agreeing by convention. Now there is +one reader: + + def tools(catalog), do: catalog.capabilities().tools + +`server.ex:185` advertises through it; `Catalog.fetch/2` resolves through it. The host's function +is called from one place in `lib/`, so the two paths cannot be given different sources without +editing that line — which is what mutant `Mcat2` does, and it dies. + +Pinned **by effect**, not by inspection: `catalog_test.exs` compares the names `tools/list` +advertises against the names `tools/call` accepts, for `OnlyHere`, a catalog whose tool +(`only_in_this_catalog`) exists in no other catalog in the suite. Slice 002's original test could +pass by coincidence because both sides could find the same tool via a shared default; this one +cannot. + +### (c) `fetch/2`'s `@spec`: documented honestly, not caught — and the ledger it came from was wrong + +The `@spec` says `{:ok, t} | :error`. A malformed host catalog makes it raise. That is now stated +in the moduledoc rather than caught, because catching would make a host bug indistinguishable +from "no such tool": a broken catalog would present exactly as a working catalog that does not +have that tool. That is the advertise-versus-call disagreement this behaviour exists to prevent, +reintroduced by the error handling meant to be defensive. + +**PR #10's ledger says three shapes raise. Reproduced on this tree, four do** — and a fifth case +the ledger missed is worse than any of them (`logs/probe-fetch-spec.txt`): + + host spec.name is a binary, not an atom {:RAISED, ArgumentError} + all/0 returns a non-list map {:RAISED, BadMapError} + all/0 returns nil {:RAISED, Protocol.UndefinedError} + all/0 returns a bare map, not a %ToolSpec{} {:returned, {:ok, %{name: :echo}}} + module is not loaded / does not exist {:RAISED, UndefinedFunctionError} + CONTROL: a well-formed catalog {:returned, {:ok, %BeamMCP.ToolSpec{...}}} + +The fourth line is the one that matters. It does **not** raise: it returns `{:ok, %{name: :echo}}` +successfully, which violates the `@spec` **silently** and hands a struct-shaped thing that is not +a struct to dispatch. A raise is a loud wrong answer; this is a quiet one. + +Both are closed the same way, and not by catching: `Catalog.validate/1` at `Server.new/1` refuses +all five at **startup**, including the bare-map case. `fetch/2` keeps its raise conditions, and +they are now unreachable for a server that started. + +## The malformed-catalog refusal, demonstrated red before it passed + +A test never observed failing is not evidence. The init-time refusal was removed from +`Server.new/1` and the five tests quoted their failures — `logs/red-malformed-catalog.txt`: + + 1) an absent key is a malformed catalog, not an empty one + Expected exception ArgumentError but nothing was raised + 2) :tools must be a list ... nothing was raised + 3) :tools entries must be %ToolSpec{} ... nothing was raised + 4) a module that does not export capabilities/0 is refused ... nothing was raised + 5) capabilities/0 must return a map ... nothing was raised + + 13 tests, 5 failures + TEST_EXIT=2 + +Restored — `logs/green-malformed-catalog.txt`: **`13 tests, 0 failures`, `TEST_EXIT=0`**. + +## Mutation scores — both KILL, zero variance across passes + +`logs/mutation-catalog.txt`, tree `8f47e1f80c4dc356b3d84d72eb1d468cfa461c49`, 2 passes each. +Two invocations, because the two mutants have different targets: + + TARGET=lib/beam_mcp/catalog.ex tools/mutate.sh score Mcat1 + Mcat1 | 187 tests, 2 failures TEST_EXIT=2 | 187 tests, 2 failures TEST_EXIT=2 + failed: ... an absent key is a malformed catalog, not an empty one (BeamMCP.CatalogTest) + failed: ... the refusal the README promises is the refusal the code performs (BeamMCP.ReadmeClaimsTest) + + TARGET=lib/beam_mcp/server.ex tools/mutate.sh score Mcat2 + Mcat2 | 187 tests, 12 failures TEST_EXIT=2 | 187 tests, 12 failures TEST_EXIT=2 + failed: the single-lookup guarantee a name the catalog does not advertise is not callable either (BeamMCP.CatalogTest) + ... 11 others, across HTTPTest, ToolSpecSchemaTest, ErrorPayloadTest, ServerTest, ReadmeClaimsTest + +**Mcat1** drops `:prompts` from `@required_keys` — the smallest possible relaxation, one key out +of three, no message change, no behaviour change for a correct host. It is killed by the startup +refusal *and*, independently, by the README claim test: the README promises that refusal, so +weakening it makes the package stop doing what the README says. + +**Mcat2** leaves `tools/list` reading the catalog and makes `tools/call` answer "which tool does +this name mean" from the request instead. That is slice 002's defect in the direction it actually +occurred. The single-lookup test kills it directly, and eleven others fall with it because +`find_tool/2`'s result is what carries the schema every later check reads. + +### Mcat2's first version was a compiler kill, and that is recorded rather than tidied away + +It returned `{:ok, spec}` unconditionally. Elixir's type checker narrowed `find_tool/2`'s return +to that one shape, declared the caller's `:error ->` clause unreachable, and +`--warnings-as-errors` failed the build — the score line came back with **no test count at all**: + + Mcat2 | TEST_EXIT=1 | TEST_EXIT=1 + + warning: the following clause will never match: :error + ... typing violation found at: lib/beam_mcp/server.ex:223 + Compilation failed due to warnings while using the --warnings-as-errors option + +`CONVENTIONS.md`: a compiler kill records a kill that never happened. No test ran. The mutant was +rewritten to keep the return a union — an `:error` for the empty name only — so the suite scores +it instead of the compiler. **The mutant was corrected so it could be scored, not weakened until +it killed**; every name a client can actually send is still callable under it, which is the whole +defect intact. The failed run is left in the log above the corrected one. + +## Two things found while doing this, neither predicted + +**The brief's call-site list was incomplete, and checking it was not optional.** It named four +sites; the tree has five readers of the contract in `lib/`. `server.ex:313` (`new/1`'s validation) +and `http.ex:225` (`init/1`'s export check) were not on the list. In the other direction, +`server.ex:311` **is** on the list and is *not* affected — it consumes a `ToolSpec`, not the +catalog. A list of call sites that is accepted rather than derived is a claim about the tree, and +this one was wrong in both directions. + +**The two init paths cannot both check the same thing, and the asymmetry is deliberate.** +`Server.new/1` calls `Catalog.validate/1`, which calls `capabilities/0` — host code, at runtime. +`Transport.HTTP.init/1` deliberately does **not**: Plug's default `init_mode` is `:compile`, so +`init/1` runs at the *host's compile time*, and a correct catalog that reads config or ETS would +crash there. So the transport checks only `function_exported?(catalog, :capabilities, 0)` — a +structural check that needs no call — and leaves the shape to `new/1`. This is one mechanism +reading one kind of input in two places with two different checks, which normally reads as the +defect `CONVENTIONS.md` names. It is not: the two places are at different *times*, and the check +that cannot run at compile time is the one omitted there. Recorded because it looks wrong until +you know why. + +## Departures from slice 007's shape, stated rather than smoothed over + +- **The issue was filed at record time, not before the code.** Slice 007's PLAN opens "Written + before the code" and this one cannot. The plan itself did precede the implementation; the + Linear issue (SCR-294) did not, because the brief ordered every tree change first and the + record last. What that costs is real: nothing external witnessed the acceptance criteria before + they were met, so they are only as good as this file. +- **No review rounds**, and the PLAN says so explicitly rather than leaving the absence to be + inferred. Single pass by instruction. The mutants and the gate stand in for the reviewer. + +## The count in the commit message was wrong, and the fix is here rather than only in git + +`4a2d2e9`'s message read "172 tests before, 185 after". **185 was typed, not measured** — the run +printed `187 tests, 0 failures`. `CONVENTIONS.md` forbids exactly that, so the commit was amended +before anything was pushed and it is now `8f47e1f`. Recorded here as well because the defect is +the habit, not the digit, and an amended commit leaves no trace of the original. diff --git a/slices/008-catalog-generalization/PLAN.md b/slices/008-catalog-generalization/PLAN.md new file mode 100644 index 0000000..1c1d3a7 --- /dev/null +++ b/slices/008-catalog-generalization/PLAN.md @@ -0,0 +1,85 @@ + + +# Slice 008 — generalize the catalog contract, so resources and prompts cost no second break + +**Issue:** SCR-294 — filed at record time rather than before the code, which is a departure from +slice 007 and is recorded as one in FINDINGS. + +**Number verified free in both places before claiming it**, the check a directory-versus-issue +collision made routine. `ls slices/` on `main`: `001`, `001b`, `002`, `003`, `004`, `006`, `007` +— no `008`. `git branch -a`: no `slice/008-*` and no `origin/slice/008-*` before this branch was +cut. (`005` has a branch and an issue but no directory; it is PR #15's, still unmerged.) + +**This plan specifies no review rounds, and that is deliberate rather than an omission.** The +brief was a single pass — plan, implement, verify, commit, do not wait for review between steps. +Slices 002, 003 and 006 ran numbered rounds because a reviewer was scoring them; nobody is +scoring this one, so a round boundary here would be a heading with nothing behind it. The +mutants and the gate are what stands in for the reviewer, which is why both are acceptance +criteria below rather than nice-to-haves. + +**Honest about its own order.** The decisions and criteria below were settled before the code was +written; the file is written at record time, after it, because the brief put every tree change +first and the record last. Two things in it were revised *by measurement* rather than by +preference, and both revisions are named where they occur rather than smoothed over. + +## The defect + +`BeamMCP.ToolCatalog` names one thing and returns one thing: + + @callback all() :: [BeamMCP.ToolSpec.t()] + +MCP has three catalog-shaped concepts — tools, resources, prompts. This package serves the first. +Adding either of the others later means changing this callback's return type, which is a second +break for every host that has written one. The point of doing it now is that 0.x is where a break +is cheap and 1.0 is where it is not. + +## The shape + + %{tools: [BeamMCP.ToolSpec.t()], resources: [], prompts: []} + +Every key **required**, `resources` and `prompts` permitted to be empty and read by nothing. An +absent key is a malformed catalog, not an empty one: the two are different claims and only one of +them is checkable. + +## Three decisions this forces, to be answered in FINDINGS + +- **(a)** The migration path: clean break, or a deprecated `ToolCatalog` delegating to the new + behaviour. +- **(b)** What happens to the single-lookup guarantee at `tool_catalog.ex:19-23` — the property + that `tools/list` and `tools/call` cannot disagree. +- **(c)** `fetch/2`'s `@spec`, which claims `{:ok, t} | :error` and is violated by a malformed + host catalog. + +## Call sites to verify, all of them, not the ones a brief remembers + +The brief named four. The list was checked against the tree instead of accepted, and it was +**incomplete** — the finding is in FINDINGS. Every reader of the contract in `lib/`: + + lib/beam_mcp/server.ex:185 tools/list advertises + lib/beam_mcp/server.ex:304 find_tool/2, the tools/call gate + lib/beam_mcp/server.ex:313 new/1's init-time validation + lib/beam_mcp/transport/http.ex:225 init/1's export check + lib/beam_mcp/transport/http.ex:836 the x-mcp-header annotation lookup + +## Acceptance criteria, as measurements + +1. `tool_catalog.ex` is gone and nothing in `lib/` reads it. +2. A malformed catalog is refused at `Server.new/1`, **demonstrated red first** — the refusal + removed, the failures quoted, the refusal restored, quoted green. +3. The single-lookup guarantee is pinned **by effect**: what `tools/list` advertises compared + against what `tools/call` accepts, for a catalog whose tool exists in no other catalog in the + suite, so neither path can agree by coincidence. +4. `fetch/2`'s raise conditions are **reproduced on this tree**, not inherited from PR #10's + ledger. +5. Two mutants in `tools/mutants/`, scored by `tools/mutate.sh`, both of which must **KILL**: + one dropping a required key from the shape, one giving advertise and call different sources. + A survivor is reported, not fixed by adjusting the mutant. +6. `./tools/gate.sh` green, all eight lines, exit 0. + +## Out of scope + +Serving resources or prompts. Any opinion about what belongs in them. `authorize/1`, +`authorize_body/2`, and their tests. diff --git a/slices/008-catalog-generalization/logs/mutation-catalog.txt b/slices/008-catalog-generalization/logs/mutation-catalog.txt new file mode 100644 index 0000000..f432aad --- /dev/null +++ b/slices/008-catalog-generalization/logs/mutation-catalog.txt @@ -0,0 +1,43 @@ +=== dirty-target refusal, run before scoring === +$ printf '\n' >> lib/beam_mcp/catalog.ex && TARGET=lib/beam_mcp/catalog.ex tools/mutate.sh score Mcat1 +REFUSING: lib/beam_mcp/catalog.ex has uncommitted changes. (exit 1) + +=== Mcat1 : TARGET=lib/beam_mcp/catalog.ex === +target: lib/beam_mcp/catalog.ex +passes: 2 +tree: 8f47e1f80c4dc356b3d84d72eb1d468cfa461c49 dirty=3 +Mcat1 | 187 tests, 2 failures TEST_EXIT=2 | 187 tests, 2 failures TEST_EXIT=2 + failed: a malformed catalog is refused at new/1, not at the first request an absent key is a malformed catalog, not an empty one (BeamMCP.CatalogTest) + failed: the BeamMCP.Catalog README claims the refusal the README promises is the refusal the code performs (BeamMCP.ReadmeClaimsTest) +SCORE_EXIT=0 + +=== Mcat2 : TARGET=lib/beam_mcp/server.ex === +target: lib/beam_mcp/server.ex +passes: 2 +tree: 8f47e1f80c4dc356b3d84d72eb1d468cfa461c49 dirty=3 +Mcat2 | TEST_EXIT=1 | TEST_EXIT=1 +SCORE_EXIT=0 + ^ NO TEST COUNT: a compiler kill, not a kill. Elixir narrowed find_tool/2's return to + {:ok, spec}, called the caller's ':error ->' clause unreachable, and + --warnings-as-errors failed the build before a test ran. Mcat2 was rewritten to keep + the return a union (see its header); rescored below. The mutant was not weakened to + make it kill -- it was corrected so that it could be scored at all. + +=== Mcat2 (corrected) : TARGET=lib/beam_mcp/server.ex === +target: lib/beam_mcp/server.ex +passes: 2 +tree: 8f47e1f80c4dc356b3d84d72eb1d468cfa461c49 dirty=3 +Mcat2 | 187 tests, 12 failures TEST_EXIT=2 | 187 tests, 12 failures TEST_EXIT=2 + failed: 404 is the method-not-found case, not the -32601 code an unknown tool is 200 with a JSON-RPC error, not 404 (BeamMCP.Transport.HTTPTest) + failed: 404 is the method-not-found case, not the -32601 code the core's wording that 404 keys on is pinned here (BeamMCP.Transport.HTTPTest) + failed: a call carrying a property the catalog's schema forbids is refused (BeamMCP.ToolSpecSchemaTest) + failed: a call missing a required property the catalog declared is refused (BeamMCP.ToolSpecSchemaTest) + failed: a validation failure carries structured fields, not an inspected map (BeamMCP.ErrorPayloadTest) + failed: a valid call reaches dispatch with keys from the catalog's schema (BeamMCP.ToolSpecSchemaTest) + failed: every error result is still flagged isError (BeamMCP.ErrorPayloadTest) + failed: the HTTP transport's README claims arguments reach dispatch as atoms, as the README now says they do (BeamMCP.ReadmeClaimsTest) + failed: the human-readable content carries no Elixir syntax either (BeamMCP.ErrorPayloadTest) + failed: the single-lookup guarantee a name the catalog does not advertise is not callable either (BeamMCP.CatalogTest) + failed: tools/call normalizes JSON arguments before dispatching (BeamMCP.ServerTest) + failed: unknown tools return a JSON-RPC error (BeamMCP.ServerTest) +SCORE_EXIT=0 From eb4de1fae0ba88665ff43376e4ed5768ba58ce6f Mon Sep 17 00:00:00 2001 From: Ayla Croft Date: Tue, 8 Sep 2026 13:42:27 -0400 Subject: [PATCH 4/6] Carry the four tracked probes onto the new catalog contract Found after the records were written, by grepping for ToolCatalog across the whole tree rather than across lib/. Four tracked scripts still declared @behaviour BeamMCP.ToolCatalog, defined all/0, and passed tool_catalog: -- three separate breaks each, none of which the gate can see, because it never compiles tools/*.exs. They are not archives. tools/probe_ping.exs and the three network probes are instruments a later slice re-runs, and an instrument that no longer runs is the same defect as an archive nobody can refetch. Proven by running them, not by reading them: mix run tools/probe_ping.exs -> tools/list serves echo, exit 0 N=1 MIX_ENV=test mix run tools/probes/write_shape.exs -> PROBE_DONE, exit 0 N=1 MIX_ENV=test mix run tools/probes/drain_mechanism.exs -> PROBE_DONE, exit 0 N=1 MIX_ENV=test mix run tools/probes/loss_site.exs -> PROBE_DONE, exit 0 All three network probes still report {297, :econnreset, ..., true}, the shape slice 006 measured, so the contract change moved nothing they observe. Signed-off-by: Ayla Croft --- tools/probe_ping.exs | 10 ++++++---- tools/probes/drain_mechanism.exs | 10 ++++++---- tools/probes/loss_site.exs | 10 ++++++---- tools/probes/write_shape.exs | 10 ++++++---- 4 files changed, 24 insertions(+), 16 deletions(-) diff --git a/tools/probe_ping.exs b/tools/probe_ping.exs index 89b0497..8b517f3 100644 --- a/tools/probe_ping.exs +++ b/tools/probe_ping.exs @@ -7,10 +7,10 @@ # mix run tools/probe_ping.exs > slices/001b-ping-guard/logs/probe-after.txt 2>&1 defmodule ProbeCatalog do - @behaviour BeamMCP.ToolCatalog + @behaviour BeamMCP.Catalog @impl true - def all do - [ + def capabilities do + tools = [ %BeamMCP.ToolSpec{ name: :echo, command_class: :observe, @@ -18,10 +18,12 @@ defmodule ProbeCatalog do description: "Echo." } ] + + %{tools: tools, resources: [], prompts: []} end end -state = BeamMCP.Server.new(tool_catalog: ProbeCatalog, dispatch: fn _n, a, _o -> {:ok, a} end) +state = BeamMCP.Server.new(catalog: ProbeCatalog, dispatch: fn _n, a, _o -> {:ok, a} end) key = "io.modelcontextprotocol/protocolVersion" send_one = fn label, msg -> diff --git a/tools/probes/drain_mechanism.exs b/tools/probes/drain_mechanism.exs index c5369a2..e6044e0 100644 --- a/tools/probes/drain_mechanism.exs +++ b/tools/probes/drain_mechanism.exs @@ -17,10 +17,10 @@ alias BeamMCP.Transport.HTTP defmodule Cat do - @behaviour BeamMCP.ToolCatalog + @behaviour BeamMCP.Catalog @impl true - def all do - [ + def capabilities do + tools = [ %BeamMCP.ToolSpec{ name: :echo, command_class: :observe, @@ -29,6 +29,8 @@ defmodule Cat do input_schema: %{"type" => "object", "properties" => %{}, "additionalProperties" => true} } ] + + %{tools: tools, resources: [], prompts: []} end end @@ -38,7 +40,7 @@ modern = "2026-07-28" vkey = "io.modelcontextprotocol/protocolVersion" opts = [ - tool_catalog: Cat, + catalog: Cat, dispatch: fn _n, a, _o -> {:ok, a} end, authorize: fn _conn -> {:error, :nope} end, allowed_origins: :any diff --git a/tools/probes/loss_site.exs b/tools/probes/loss_site.exs index 67117af..9724f31 100644 --- a/tools/probes/loss_site.exs +++ b/tools/probes/loss_site.exs @@ -12,10 +12,10 @@ alias BeamMCP.Transport.HTTP defmodule Cat do - @behaviour BeamMCP.ToolCatalog + @behaviour BeamMCP.Catalog @impl true - def all do - [ + def capabilities do + tools = [ %BeamMCP.ToolSpec{ name: :echo, command_class: :observe, @@ -24,6 +24,8 @@ defmodule Cat do input_schema: %{"type" => "object", "properties" => %{}, "additionalProperties" => true} } ] + + %{tools: tools, resources: [], prompts: []} end end @@ -34,7 +36,7 @@ vkey = "io.modelcontextprotocol/protocolVersion" owner = self() opts = [ - tool_catalog: Cat, + catalog: Cat, dispatch: fn _n, a, _o -> {:ok, a} end, authorize: fn _conn -> send(owner, :authorize_ran) diff --git a/tools/probes/write_shape.exs b/tools/probes/write_shape.exs index 516aab3..cba01f2 100644 --- a/tools/probes/write_shape.exs +++ b/tools/probes/write_shape.exs @@ -12,10 +12,10 @@ alias BeamMCP.Transport.HTTP defmodule Cat do - @behaviour BeamMCP.ToolCatalog + @behaviour BeamMCP.Catalog @impl true - def all do - [ + def capabilities do + tools = [ %BeamMCP.ToolSpec{ name: :echo, command_class: :observe, @@ -24,6 +24,8 @@ defmodule Cat do input_schema: %{"type" => "object", "properties" => %{}, "additionalProperties" => true} } ] + + %{tools: tools, resources: [], prompts: []} end end @@ -34,7 +36,7 @@ vkey = "io.modelcontextprotocol/protocolVersion" owner = self() opts = [ - tool_catalog: Cat, + catalog: Cat, dispatch: fn _n, a, _o -> {:ok, a} end, authorize: fn _conn -> send(owner, :authorize_ran) From 2f1f22f45eccb64fa0e5709ea0c52109718f0848 Mon Sep 17 00:00:00 2001 From: Ayla Croft Date: Tue, 8 Sep 2026 13:42:28 -0400 Subject: [PATCH 5/6] Record the four probes, and that lib/ was the wrong population to check Criterion 1 was written as "nothing in lib/ reads tool_catalog.ex" and was true while four tracked instruments outside lib/ were broken by the same change. The finding is the scope of the grep, not the four files. Signed-off-by: Ayla Croft --- slices/008-catalog-generalization/FINDINGS.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/slices/008-catalog-generalization/FINDINGS.md b/slices/008-catalog-generalization/FINDINGS.md index c4cfcd3..cb3a7de 100644 --- a/slices/008-catalog-generalization/FINDINGS.md +++ b/slices/008-catalog-generalization/FINDINGS.md @@ -151,6 +151,18 @@ defect `CONVENTIONS.md` names. It is not: the two places are at different *times that cannot run at compile time is the one omitted there. Recorded because it looks wrong until you know why. +**Four tracked probes were still on the old contract, and `lib/` is the wrong population to +check.** Acceptance criterion 1 says "nothing in `lib/` reads `tool_catalog.ex`", and that was +true while `tools/probe_ping.exs` and the three probes in `tools/probes/` each carried three +separate breaks: `@behaviour BeamMCP.ToolCatalog`, `def all`, and `catalog: ` spelled +`tool_catalog: `. The gate cannot see them — it never compiles `tools/*.exs` — so they would have +sat broken until the next slice tried to re-run one. They are instruments, not archives, and an +instrument that no longer runs is the same defect as an archive nobody can refetch. Fixed and +**proven by running them**: `mix run tools/probe_ping.exs` serves `echo` from `tools/list`, and +each network probe prints `PROBE_DONE` at `N=1` with the same `{297, :econnreset, ..., true}` +slice 006 measured. The lesson is the population, not the four files: the grep that found this +was the one run over the whole tree, and the one that missed it was scoped to `lib/`. + ## Departures from slice 007's shape, stated rather than smoothed over - **The issue was filed at record time, not before the code.** Slice 007's PLAN opens "Written From d0cbb8b5c089ab52eebdffacb32bd99959119c1f Mon Sep 17 00:00:00 2001 From: Ayla Croft Date: Sun, 13 Sep 2026 12:42:35 -0400 Subject: [PATCH 6/6] Move slice 008's records to the internal repository The publication step added in 009 refuses new tracked paths under slices/; these six were written before the records moved out of the tree. Copied to the internal repository and verified identical by diff -r before untracking, then removed. The allowlist is unchanged: it grandfathers published history, and this branch is not merged, so there is no published history here to grandfather. Signed-off-by: Ayla Croft --- slices/008-catalog-generalization/FINDINGS.md | 181 ------------------ slices/008-catalog-generalization/PLAN.md | 85 -------- .../logs/green-malformed-catalog.txt | 10 - .../logs/mutation-catalog.txt | 43 ----- .../logs/probe-fetch-spec.txt | 11 -- .../logs/red-malformed-catalog.txt | 55 ------ 6 files changed, 385 deletions(-) delete mode 100644 slices/008-catalog-generalization/FINDINGS.md delete mode 100644 slices/008-catalog-generalization/PLAN.md delete mode 100644 slices/008-catalog-generalization/logs/green-malformed-catalog.txt delete mode 100644 slices/008-catalog-generalization/logs/mutation-catalog.txt delete mode 100644 slices/008-catalog-generalization/logs/probe-fetch-spec.txt delete mode 100644 slices/008-catalog-generalization/logs/red-malformed-catalog.txt diff --git a/slices/008-catalog-generalization/FINDINGS.md b/slices/008-catalog-generalization/FINDINGS.md deleted file mode 100644 index cb3a7de..0000000 --- a/slices/008-catalog-generalization/FINDINGS.md +++ /dev/null @@ -1,181 +0,0 @@ - - -# Slice 008 — findings - -## The three decisions, made deliberately - -### (a) Migration path: a clean break, no deprecated `ToolCatalog` delegate - -Two reasons, and the second is the load-bearing one. - -The 0.x policy already protects consumers. The README recommends `~> 0.3.0`, which **excludes** -`0.4.0` — so no existing consumer is upgraded into this break by resolution. A host that wants it -asks for it, which is exactly what a break at minor is for. - -And a delegate would create **a second reader**, which decision (b) forbids. `ToolCatalog.fetch/2` -delegating to `Catalog.fetch/2` is harmless; `ToolCatalog.all/0` living beside -`Catalog.capabilities/0` is not, because a host that implements both can make them disagree and -nothing would catch it. The compatibility shim would reintroduce the defect the contract exists -to prevent, in order to soften a break the version policy already handles. - -The callback is also **renamed**, `all/0` → `capabilities/0`, and that is part of the same -decision. Keeping the name while changing the return from a list to a map compiles against every -existing host and fails at the first request with a `BadMapError` — a silent shape change, which -is the defect class this repository keeps finding. Renaming makes the break arrive at compile -time as an unimplemented callback, which is the loudest place it can arrive. - -### (b) The single-lookup guarantee: preserved, and structurally stronger than before - -Before this slice, `tools/list` called `catalog.all()` and `ToolCatalog.fetch/2` called -`catalog.all()` — **two call sites of the host's function**, agreeing by convention. Now there is -one reader: - - def tools(catalog), do: catalog.capabilities().tools - -`server.ex:185` advertises through it; `Catalog.fetch/2` resolves through it. The host's function -is called from one place in `lib/`, so the two paths cannot be given different sources without -editing that line — which is what mutant `Mcat2` does, and it dies. - -Pinned **by effect**, not by inspection: `catalog_test.exs` compares the names `tools/list` -advertises against the names `tools/call` accepts, for `OnlyHere`, a catalog whose tool -(`only_in_this_catalog`) exists in no other catalog in the suite. Slice 002's original test could -pass by coincidence because both sides could find the same tool via a shared default; this one -cannot. - -### (c) `fetch/2`'s `@spec`: documented honestly, not caught — and the ledger it came from was wrong - -The `@spec` says `{:ok, t} | :error`. A malformed host catalog makes it raise. That is now stated -in the moduledoc rather than caught, because catching would make a host bug indistinguishable -from "no such tool": a broken catalog would present exactly as a working catalog that does not -have that tool. That is the advertise-versus-call disagreement this behaviour exists to prevent, -reintroduced by the error handling meant to be defensive. - -**PR #10's ledger says three shapes raise. Reproduced on this tree, four do** — and a fifth case -the ledger missed is worse than any of them (`logs/probe-fetch-spec.txt`): - - host spec.name is a binary, not an atom {:RAISED, ArgumentError} - all/0 returns a non-list map {:RAISED, BadMapError} - all/0 returns nil {:RAISED, Protocol.UndefinedError} - all/0 returns a bare map, not a %ToolSpec{} {:returned, {:ok, %{name: :echo}}} - module is not loaded / does not exist {:RAISED, UndefinedFunctionError} - CONTROL: a well-formed catalog {:returned, {:ok, %BeamMCP.ToolSpec{...}}} - -The fourth line is the one that matters. It does **not** raise: it returns `{:ok, %{name: :echo}}` -successfully, which violates the `@spec` **silently** and hands a struct-shaped thing that is not -a struct to dispatch. A raise is a loud wrong answer; this is a quiet one. - -Both are closed the same way, and not by catching: `Catalog.validate/1` at `Server.new/1` refuses -all five at **startup**, including the bare-map case. `fetch/2` keeps its raise conditions, and -they are now unreachable for a server that started. - -## The malformed-catalog refusal, demonstrated red before it passed - -A test never observed failing is not evidence. The init-time refusal was removed from -`Server.new/1` and the five tests quoted their failures — `logs/red-malformed-catalog.txt`: - - 1) an absent key is a malformed catalog, not an empty one - Expected exception ArgumentError but nothing was raised - 2) :tools must be a list ... nothing was raised - 3) :tools entries must be %ToolSpec{} ... nothing was raised - 4) a module that does not export capabilities/0 is refused ... nothing was raised - 5) capabilities/0 must return a map ... nothing was raised - - 13 tests, 5 failures - TEST_EXIT=2 - -Restored — `logs/green-malformed-catalog.txt`: **`13 tests, 0 failures`, `TEST_EXIT=0`**. - -## Mutation scores — both KILL, zero variance across passes - -`logs/mutation-catalog.txt`, tree `8f47e1f80c4dc356b3d84d72eb1d468cfa461c49`, 2 passes each. -Two invocations, because the two mutants have different targets: - - TARGET=lib/beam_mcp/catalog.ex tools/mutate.sh score Mcat1 - Mcat1 | 187 tests, 2 failures TEST_EXIT=2 | 187 tests, 2 failures TEST_EXIT=2 - failed: ... an absent key is a malformed catalog, not an empty one (BeamMCP.CatalogTest) - failed: ... the refusal the README promises is the refusal the code performs (BeamMCP.ReadmeClaimsTest) - - TARGET=lib/beam_mcp/server.ex tools/mutate.sh score Mcat2 - Mcat2 | 187 tests, 12 failures TEST_EXIT=2 | 187 tests, 12 failures TEST_EXIT=2 - failed: the single-lookup guarantee a name the catalog does not advertise is not callable either (BeamMCP.CatalogTest) - ... 11 others, across HTTPTest, ToolSpecSchemaTest, ErrorPayloadTest, ServerTest, ReadmeClaimsTest - -**Mcat1** drops `:prompts` from `@required_keys` — the smallest possible relaxation, one key out -of three, no message change, no behaviour change for a correct host. It is killed by the startup -refusal *and*, independently, by the README claim test: the README promises that refusal, so -weakening it makes the package stop doing what the README says. - -**Mcat2** leaves `tools/list` reading the catalog and makes `tools/call` answer "which tool does -this name mean" from the request instead. That is slice 002's defect in the direction it actually -occurred. The single-lookup test kills it directly, and eleven others fall with it because -`find_tool/2`'s result is what carries the schema every later check reads. - -### Mcat2's first version was a compiler kill, and that is recorded rather than tidied away - -It returned `{:ok, spec}` unconditionally. Elixir's type checker narrowed `find_tool/2`'s return -to that one shape, declared the caller's `:error ->` clause unreachable, and -`--warnings-as-errors` failed the build — the score line came back with **no test count at all**: - - Mcat2 | TEST_EXIT=1 | TEST_EXIT=1 - - warning: the following clause will never match: :error - ... typing violation found at: lib/beam_mcp/server.ex:223 - Compilation failed due to warnings while using the --warnings-as-errors option - -`CONVENTIONS.md`: a compiler kill records a kill that never happened. No test ran. The mutant was -rewritten to keep the return a union — an `:error` for the empty name only — so the suite scores -it instead of the compiler. **The mutant was corrected so it could be scored, not weakened until -it killed**; every name a client can actually send is still callable under it, which is the whole -defect intact. The failed run is left in the log above the corrected one. - -## Two things found while doing this, neither predicted - -**The brief's call-site list was incomplete, and checking it was not optional.** It named four -sites; the tree has five readers of the contract in `lib/`. `server.ex:313` (`new/1`'s validation) -and `http.ex:225` (`init/1`'s export check) were not on the list. In the other direction, -`server.ex:311` **is** on the list and is *not* affected — it consumes a `ToolSpec`, not the -catalog. A list of call sites that is accepted rather than derived is a claim about the tree, and -this one was wrong in both directions. - -**The two init paths cannot both check the same thing, and the asymmetry is deliberate.** -`Server.new/1` calls `Catalog.validate/1`, which calls `capabilities/0` — host code, at runtime. -`Transport.HTTP.init/1` deliberately does **not**: Plug's default `init_mode` is `:compile`, so -`init/1` runs at the *host's compile time*, and a correct catalog that reads config or ETS would -crash there. So the transport checks only `function_exported?(catalog, :capabilities, 0)` — a -structural check that needs no call — and leaves the shape to `new/1`. This is one mechanism -reading one kind of input in two places with two different checks, which normally reads as the -defect `CONVENTIONS.md` names. It is not: the two places are at different *times*, and the check -that cannot run at compile time is the one omitted there. Recorded because it looks wrong until -you know why. - -**Four tracked probes were still on the old contract, and `lib/` is the wrong population to -check.** Acceptance criterion 1 says "nothing in `lib/` reads `tool_catalog.ex`", and that was -true while `tools/probe_ping.exs` and the three probes in `tools/probes/` each carried three -separate breaks: `@behaviour BeamMCP.ToolCatalog`, `def all`, and `catalog: ` spelled -`tool_catalog: `. The gate cannot see them — it never compiles `tools/*.exs` — so they would have -sat broken until the next slice tried to re-run one. They are instruments, not archives, and an -instrument that no longer runs is the same defect as an archive nobody can refetch. Fixed and -**proven by running them**: `mix run tools/probe_ping.exs` serves `echo` from `tools/list`, and -each network probe prints `PROBE_DONE` at `N=1` with the same `{297, :econnreset, ..., true}` -slice 006 measured. The lesson is the population, not the four files: the grep that found this -was the one run over the whole tree, and the one that missed it was scoped to `lib/`. - -## Departures from slice 007's shape, stated rather than smoothed over - -- **The issue was filed at record time, not before the code.** Slice 007's PLAN opens "Written - before the code" and this one cannot. The plan itself did precede the implementation; the - Linear issue (SCR-294) did not, because the brief ordered every tree change first and the - record last. What that costs is real: nothing external witnessed the acceptance criteria before - they were met, so they are only as good as this file. -- **No review rounds**, and the PLAN says so explicitly rather than leaving the absence to be - inferred. Single pass by instruction. The mutants and the gate stand in for the reviewer. - -## The count in the commit message was wrong, and the fix is here rather than only in git - -`4a2d2e9`'s message read "172 tests before, 185 after". **185 was typed, not measured** — the run -printed `187 tests, 0 failures`. `CONVENTIONS.md` forbids exactly that, so the commit was amended -before anything was pushed and it is now `8f47e1f`. Recorded here as well because the defect is -the habit, not the digit, and an amended commit leaves no trace of the original. diff --git a/slices/008-catalog-generalization/PLAN.md b/slices/008-catalog-generalization/PLAN.md deleted file mode 100644 index 1c1d3a7..0000000 --- a/slices/008-catalog-generalization/PLAN.md +++ /dev/null @@ -1,85 +0,0 @@ - - -# Slice 008 — generalize the catalog contract, so resources and prompts cost no second break - -**Issue:** SCR-294 — filed at record time rather than before the code, which is a departure from -slice 007 and is recorded as one in FINDINGS. - -**Number verified free in both places before claiming it**, the check a directory-versus-issue -collision made routine. `ls slices/` on `main`: `001`, `001b`, `002`, `003`, `004`, `006`, `007` -— no `008`. `git branch -a`: no `slice/008-*` and no `origin/slice/008-*` before this branch was -cut. (`005` has a branch and an issue but no directory; it is PR #15's, still unmerged.) - -**This plan specifies no review rounds, and that is deliberate rather than an omission.** The -brief was a single pass — plan, implement, verify, commit, do not wait for review between steps. -Slices 002, 003 and 006 ran numbered rounds because a reviewer was scoring them; nobody is -scoring this one, so a round boundary here would be a heading with nothing behind it. The -mutants and the gate are what stands in for the reviewer, which is why both are acceptance -criteria below rather than nice-to-haves. - -**Honest about its own order.** The decisions and criteria below were settled before the code was -written; the file is written at record time, after it, because the brief put every tree change -first and the record last. Two things in it were revised *by measurement* rather than by -preference, and both revisions are named where they occur rather than smoothed over. - -## The defect - -`BeamMCP.ToolCatalog` names one thing and returns one thing: - - @callback all() :: [BeamMCP.ToolSpec.t()] - -MCP has three catalog-shaped concepts — tools, resources, prompts. This package serves the first. -Adding either of the others later means changing this callback's return type, which is a second -break for every host that has written one. The point of doing it now is that 0.x is where a break -is cheap and 1.0 is where it is not. - -## The shape - - %{tools: [BeamMCP.ToolSpec.t()], resources: [], prompts: []} - -Every key **required**, `resources` and `prompts` permitted to be empty and read by nothing. An -absent key is a malformed catalog, not an empty one: the two are different claims and only one of -them is checkable. - -## Three decisions this forces, to be answered in FINDINGS - -- **(a)** The migration path: clean break, or a deprecated `ToolCatalog` delegating to the new - behaviour. -- **(b)** What happens to the single-lookup guarantee at `tool_catalog.ex:19-23` — the property - that `tools/list` and `tools/call` cannot disagree. -- **(c)** `fetch/2`'s `@spec`, which claims `{:ok, t} | :error` and is violated by a malformed - host catalog. - -## Call sites to verify, all of them, not the ones a brief remembers - -The brief named four. The list was checked against the tree instead of accepted, and it was -**incomplete** — the finding is in FINDINGS. Every reader of the contract in `lib/`: - - lib/beam_mcp/server.ex:185 tools/list advertises - lib/beam_mcp/server.ex:304 find_tool/2, the tools/call gate - lib/beam_mcp/server.ex:313 new/1's init-time validation - lib/beam_mcp/transport/http.ex:225 init/1's export check - lib/beam_mcp/transport/http.ex:836 the x-mcp-header annotation lookup - -## Acceptance criteria, as measurements - -1. `tool_catalog.ex` is gone and nothing in `lib/` reads it. -2. A malformed catalog is refused at `Server.new/1`, **demonstrated red first** — the refusal - removed, the failures quoted, the refusal restored, quoted green. -3. The single-lookup guarantee is pinned **by effect**: what `tools/list` advertises compared - against what `tools/call` accepts, for a catalog whose tool exists in no other catalog in the - suite, so neither path can agree by coincidence. -4. `fetch/2`'s raise conditions are **reproduced on this tree**, not inherited from PR #10's - ledger. -5. Two mutants in `tools/mutants/`, scored by `tools/mutate.sh`, both of which must **KILL**: - one dropping a required key from the shape, one giving advertise and call different sources. - A survivor is reported, not fixed by adjusting the mutant. -6. `./tools/gate.sh` green, all eight lines, exit 0. - -## Out of scope - -Serving resources or prompts. Any opinion about what belongs in them. `authorize/1`, -`authorize_body/2`, and their tests. diff --git a/slices/008-catalog-generalization/logs/green-malformed-catalog.txt b/slices/008-catalog-generalization/logs/green-malformed-catalog.txt deleted file mode 100644 index 5550ffc..0000000 --- a/slices/008-catalog-generalization/logs/green-malformed-catalog.txt +++ /dev/null @@ -1,10 +0,0 @@ -=== GREEN: the init-time shape refusal restored === -$ mix test test/beam_mcp/catalog_test.exs -Compiling 1 file (.ex) -Generated beam_mcp app -Running ExUnit with seed: 985007, max_cases: 64 - -............. -Finished in 0.09 seconds (0.09s async, 0.00s sync) -13 tests, 0 failures -TEST_EXIT=0 diff --git a/slices/008-catalog-generalization/logs/mutation-catalog.txt b/slices/008-catalog-generalization/logs/mutation-catalog.txt deleted file mode 100644 index f432aad..0000000 --- a/slices/008-catalog-generalization/logs/mutation-catalog.txt +++ /dev/null @@ -1,43 +0,0 @@ -=== dirty-target refusal, run before scoring === -$ printf '\n' >> lib/beam_mcp/catalog.ex && TARGET=lib/beam_mcp/catalog.ex tools/mutate.sh score Mcat1 -REFUSING: lib/beam_mcp/catalog.ex has uncommitted changes. (exit 1) - -=== Mcat1 : TARGET=lib/beam_mcp/catalog.ex === -target: lib/beam_mcp/catalog.ex -passes: 2 -tree: 8f47e1f80c4dc356b3d84d72eb1d468cfa461c49 dirty=3 -Mcat1 | 187 tests, 2 failures TEST_EXIT=2 | 187 tests, 2 failures TEST_EXIT=2 - failed: a malformed catalog is refused at new/1, not at the first request an absent key is a malformed catalog, not an empty one (BeamMCP.CatalogTest) - failed: the BeamMCP.Catalog README claims the refusal the README promises is the refusal the code performs (BeamMCP.ReadmeClaimsTest) -SCORE_EXIT=0 - -=== Mcat2 : TARGET=lib/beam_mcp/server.ex === -target: lib/beam_mcp/server.ex -passes: 2 -tree: 8f47e1f80c4dc356b3d84d72eb1d468cfa461c49 dirty=3 -Mcat2 | TEST_EXIT=1 | TEST_EXIT=1 -SCORE_EXIT=0 - ^ NO TEST COUNT: a compiler kill, not a kill. Elixir narrowed find_tool/2's return to - {:ok, spec}, called the caller's ':error ->' clause unreachable, and - --warnings-as-errors failed the build before a test ran. Mcat2 was rewritten to keep - the return a union (see its header); rescored below. The mutant was not weakened to - make it kill -- it was corrected so that it could be scored at all. - -=== Mcat2 (corrected) : TARGET=lib/beam_mcp/server.ex === -target: lib/beam_mcp/server.ex -passes: 2 -tree: 8f47e1f80c4dc356b3d84d72eb1d468cfa461c49 dirty=3 -Mcat2 | 187 tests, 12 failures TEST_EXIT=2 | 187 tests, 12 failures TEST_EXIT=2 - failed: 404 is the method-not-found case, not the -32601 code an unknown tool is 200 with a JSON-RPC error, not 404 (BeamMCP.Transport.HTTPTest) - failed: 404 is the method-not-found case, not the -32601 code the core's wording that 404 keys on is pinned here (BeamMCP.Transport.HTTPTest) - failed: a call carrying a property the catalog's schema forbids is refused (BeamMCP.ToolSpecSchemaTest) - failed: a call missing a required property the catalog declared is refused (BeamMCP.ToolSpecSchemaTest) - failed: a validation failure carries structured fields, not an inspected map (BeamMCP.ErrorPayloadTest) - failed: a valid call reaches dispatch with keys from the catalog's schema (BeamMCP.ToolSpecSchemaTest) - failed: every error result is still flagged isError (BeamMCP.ErrorPayloadTest) - failed: the HTTP transport's README claims arguments reach dispatch as atoms, as the README now says they do (BeamMCP.ReadmeClaimsTest) - failed: the human-readable content carries no Elixir syntax either (BeamMCP.ErrorPayloadTest) - failed: the single-lookup guarantee a name the catalog does not advertise is not callable either (BeamMCP.CatalogTest) - failed: tools/call normalizes JSON arguments before dispatching (BeamMCP.ServerTest) - failed: unknown tools return a JSON-RPC error (BeamMCP.ServerTest) -SCORE_EXIT=0 diff --git a/slices/008-catalog-generalization/logs/probe-fetch-spec.txt b/slices/008-catalog-generalization/logs/probe-fetch-spec.txt deleted file mode 100644 index 332f358..0000000 --- a/slices/008-catalog-generalization/logs/probe-fetch-spec.txt +++ /dev/null @@ -1,11 +0,0 @@ -=== REPRODUCING the fetch/2 @spec claim on the current tree === -PR #10's ledger claims THREE host shapes raise. Verified here rather than inherited. - -@spec says: {:ok, BeamMCP.ToolSpec.t()} | :error - -host spec.name is a binary, not an atom {:RAISED, ArgumentError} -all/0 returns a non-list map {:RAISED, BadMapError} -all/0 returns nil {:RAISED, Protocol.UndefinedError} -all/0 returns a bare map, not a %ToolSpec{} {:returned, {:ok, %{name: :echo}}} -module is not loaded / does not exist {:RAISED, UndefinedFunctionError} -CONTROL: a well-formed catalog {:returned, {:ok, %BeamMCP.ToolSpec{name: :echo, command_class: :observe, mode: :read_only, description: "d", input_schema: %{"additionalProperties" => true, "properties" => %{}, "type" => "object"}}}} diff --git a/slices/008-catalog-generalization/logs/red-malformed-catalog.txt b/slices/008-catalog-generalization/logs/red-malformed-catalog.txt deleted file mode 100644 index c388b96..0000000 --- a/slices/008-catalog-generalization/logs/red-malformed-catalog.txt +++ /dev/null @@ -1,55 +0,0 @@ -=== RED: the init-time shape refusal removed from Server.new/1 === -$ mix test test/beam_mcp/catalog_test.exs -Compiling 1 file (.ex) -Generated beam_mcp app -Running ExUnit with seed: 393777, max_cases: 64 - - - - 1) test a malformed catalog is refused at new/1, not at the first request an absent key is a malformed catalog, not an empty one (BeamMCP.CatalogTest) - test/beam_mcp/catalog_test.exs:118 - Expected exception ArgumentError but nothing was raised - code: assert_raise ArgumentError, ~r/missing required key\(s\): \[:prompts\]/, fn -> - stacktrace: - test/beam_mcp/catalog_test.exs:119: (test) - -... - - 2) test a malformed catalog is refused at new/1, not at the first request :tools must be a list (BeamMCP.CatalogTest) - test/beam_mcp/catalog_test.exs:124 - Expected exception ArgumentError but nothing was raised - code: assert_raise ArgumentError, ~r/:tools must be a list/, fn -> - stacktrace: - test/beam_mcp/catalog_test.exs:125: (test) - -.... - - 3) test a malformed catalog is refused at new/1, not at the first request :tools entries must be %ToolSpec{} (BeamMCP.CatalogTest) - test/beam_mcp/catalog_test.exs:130 - Expected exception ArgumentError but nothing was raised - code: assert_raise ArgumentError, ~r/must all be %BeamMCP.ToolSpec\{\}/, fn -> - stacktrace: - test/beam_mcp/catalog_test.exs:134: (test) - - - - 4) test a malformed catalog is refused at new/1, not at the first request a module that does not export capabilities/0 is refused (BeamMCP.CatalogTest) - test/beam_mcp/catalog_test.exs:145 - Expected exception ArgumentError but nothing was raised - code: assert_raise ArgumentError, ~r/does not export capabilities\/0/, fn -> - stacktrace: - test/beam_mcp/catalog_test.exs:146: (test) - -. - - 5) test a malformed catalog is refused at new/1, not at the first request capabilities/0 must return a map (BeamMCP.CatalogTest) - test/beam_mcp/catalog_test.exs:139 - Expected exception ArgumentError but nothing was raised - code: assert_raise ArgumentError, ~r/must return a map/, fn -> - stacktrace: - test/beam_mcp/catalog_test.exs:140: (test) - - -Finished in 0.09 seconds (0.09s async, 0.00s sync) -13 tests, 5 failures -TEST_EXIT=2