Skip to content

feat: ship a working test suite in generated SDK projects - #32

Merged
IBJunior merged 1 commit into
mainfrom
feat/generated-tests
Sep 19, 2026
Merged

IBJunior merged 1 commit into
mainfrom
feat/generated-tests

Conversation

@IBJunior

@IBJunior IBJunior commented Sep 19, 2026

Copy link
Copy Markdown
Member

Generated SDK projects now ship a working test suite. npm test passes immediately, with no config to add.

Nine tests, deliberately. The suite is a template people copy, so its size sets an expectation. Every test pins something a tool's description promises and nothing else checks; none exercise the example's own data structures. A test proving a Map stores things would teach nothing about tool design while implying a scaffolded project owes one test per function. The emitted code is ~370 lines against ~200 of tests — a ratio that reads as "here is the pattern", not "here is the bar".

Fourth of the sequence, and the point of the previous one: PR #31 replaced greet with a notes domain chosen so its contract has something to assert. This emits the tests that assert it.

Two layers

Neither subsumes the other, so both are emitted:

  • notes-store.test.ts covers the two store guarantees a tool contract depends on — nothing else. Fast, no transport.
  • server.test.ts drives a real Client over InMemoryTransport.createLinkedPair() and asserts on the parsed tool payload, which is the surface an agent actually reads. A tool can be internally correct and still return something misleading.

What the payload tests pin:

Contract Assertion
Truncation is signalled returned: 5, matched: 30, truncated: true, hint contains "5 of 30"
A complete result doesn't lie truncated: false, hint undefined
Unknown tag ≠ zero rows unknown tag is isError and names the tags that exist
Errors give a next move unknown id is isError and mentions list_notes
Results chain create_note returns an id that get_note accepts
Both resource halves wired list enumerates and read resolves the same uri
Everything is documented every tool and every parameter has a description

That last one is deliberately generic — it keeps holding after a user replaces the example with their own tools, and an undescribed parameter is a common reason an agent calls a tool wrongly.

Three things worth a reviewer's attention

No vitest.config.ts is emitted, and none is needed. Vitest discovers src/**/*.test.ts and handles TypeScript unaided. Verified by running the generated suite, not assumed.

tsconfig gains src/**/*.test.ts in exclude. Without it the tests compile into dist/ and ship in the production Docker image. getTsconfigTemplate now takes withTests, on for SDK projects only. Confirmed dist/ contains 12 files and 0 test files after a build.

The emitted test code uses string concatenation, not template literals. Nesting a template literal inside the template literal that generates it needs triple-escaped backticks, and getting it wrong emits a stray backslash that fails to parse — which is what happened on the first attempt here, caught by running the generated suite. A test now asserts no escaped template-literal syntax reaches the output, and I verified that guard actually fires by injecting the real bug shape (a single-escaped backtick emits correctly, so only the triple form is a defect).

Relatedly, the template source files are named store-test.ts and server-test.ts rather than *.test.ts, so this repo's own vitest run doesn't try to execute them as suites.

Dependency tracking

vitest and @modelcontextprotocol/client are added to TEMPLATE_PACKAGES. The client must track @modelcontextprotocol/server's major — the script can't enforce that pairing, so it's called out where the list is documented.

Verification

Every variant generated, installed, tested and built:

v-http    Tests  9 passed | build:ok | dist-tests:0
v-stdio   Tests  9 passed | build:ok | dist-tests:0
v-oauth   Tests  9 passed | build:ok | dist-tests:0
v-fast    tests:none      | build:ok | dist-tests:0

FastMCP is correctly untouched: no test script, no vitest dependency, no test exclusion in its tsconfig.

  • npm test in this repo — 422 tests across 16 files (up from 410)
  • npm run lint clean; all changed files pass prettier --check
  • .dockerignore already excludes *.test.ts, so tests never enter the image context

Not verified: the Docker image build. The Docker CLI is installed here but the engine isn't running, so I couldn't execute docker build. The .dockerignore exclusion and the clean dist/ make it very likely fine, but it's untested and worth a check if you have an engine running.

Docs

AGENTS.md and the repo README still described the greet example and the pre-split single-file server, so both are refreshed. AGENTS.md also gains a section on the generated test setup covering the three gotchas above, plus the rule that the emitted suite stays small and exemplary — adding a test should mean retiring one.

FastMCP still has no tests and keeps its own example. Worth a tracked follow-up so the gap doesn't go quiet.

The previous PR replaced the greet example with a notes domain chosen so its
contract has something to assert. This emits the tests that assert it, so a
scaffolded project starts with a green suite and a pattern to copy rather than
an empty src/ and good intentions.

Nine tests, deliberately. The suite is a template people copy, so its size
sets an expectation: every test here pins something a tool's description
promises and nothing else checks, and none of them exercise the example's own
data structures. A test proving a Map stores things would teach nothing about
tool design while implying a scaffolded project owes one per function.

Two layers, because neither subsumes the other:

- notes-store.test.ts covers the two store guarantees a tool contract depends
  on: that the store reports the full match count rather than the page size,
  and that an unknown tag is distinguishable from a tag with no notes.
- server.test.ts drives a real Client over InMemoryTransport.createLinkedPair()
  and asserts on the parsed tool payload - the surface an agent actually reads.
  A tool can be internally correct and still return something misleading.

What the payload tests pin: truncation is signalled and the hint names how to
narrow; a complete result does not claim truncation; an unknown tag is
distinguishable from an empty one; an unknown id returns isError naming the
next tool to call; create returns an id that get accepts; a ResourceTemplate
has both halves wired, since list and read are easy to implement singly. Plus
a generic check that every tool and parameter carries a description, which
keeps holding after the example is replaced.

Three things worth knowing:

- No vitest.config.ts is emitted and none is needed - vitest discovers
  src/**/*.test.ts and handles TypeScript unaided.
- tsconfig gains src/**/*.test.ts in exclude, or the tests compile into dist/
  and ship in the production Docker image. getTsconfigTemplate now takes
  withTests, on for SDK projects only.
- The emitted test code uses string concatenation rather than template
  literals. Nesting a template literal inside the one that generates it needs
  triple-escaped backticks, and getting it wrong emits a stray backslash that
  fails to parse - which is exactly what happened first time round. A test now
  asserts no escaped template-literal syntax reaches the output.

The template source files are named store-test.ts and server-test.ts, not
*.test.ts, so this repo's own vitest run does not try to execute them as
suites.

Adds vitest and @modelcontextprotocol/client to TEMPLATE_PACKAGES. The client
must track @modelcontextprotocol/server's major; the script cannot enforce
that pairing, so it is called out where the list is documented.

FastMCP gets no tests this round: no test script, no vitest dependency, and no
test exclusion in its tsconfig.

Also refreshes AGENTS.md and the repo README, which still described the greet
example and the pre-split single-file server.
@IBJunior
IBJunior force-pushed the feat/generated-tests branch from 75cf6bb to 3dca895 Compare September 19, 2026 15:53
@IBJunior
IBJunior merged commit 88c1ab0 into main Sep 19, 2026
2 checks passed
@IBJunior
IBJunior deleted the feat/generated-tests branch September 19, 2026 15:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant