Skip to content

Add live-stack MCP 2026-07-28 conformance testing - #82

Open
lucarlig wants to merge 6 commits into
mainfrom
user/luca/mcp-conformance-ci
Open

Add live-stack MCP 2026-07-28 conformance testing#82
lucarlig wants to merge 6 commits into
mainfrom
user/luca/mcp-conformance-ci

Conversation

@lucarlig

@lucarlig lucarlig commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

What this adds

  • runs the official @modelcontextprotocol/conformance@0.2.0-alpha.11 server suite for MCP 2026-07-28
  • tests the real public path: official runner → nginx → dataplane → official fixture
  • provisions the fixture through the real control plane and publishes its configuration through Redis
  • keeps the harness self-contained under tests/conformance/
  • records detailed counts in the Actions log and enforces a checked-in expected-failure baseline

Live stack

The job uses Redis, nginx, and the published latest control-plane and dataplane images. The control plane uses ephemeral SQLite, so PostgreSQL and PgBouncer are not required.

The alpha.11 runner cannot send a bearer header. Nginx therefore adds an ephemeral control-plane-issued token only when the request has no authorization header. There is no auth proxy, score calculator, or repository-owned JavaScript helper. A route probe confirms traffic reached the dataplane before conformance starts.

Current baseline

Total: 76 passed, 80 failed
Baseline check passed: all failures are expected.

The conformance output remains in the Actions log; no separate conformance artifact is uploaded. Because both ContextForge images use latest, the result measures the currently published stack rather than the pull-request build. New failures and stale expected failures both fail CI.

Additional maintenance

  • bumps rmcp and rmcp-macros from 3.1.1 to the latest release, 3.1.2
  • updates the Pages workflow to Node 24-compatible upload/deploy actions
  • installs mdBook 0.5.3 directly without the cargo-binstall fallback warning

Validation

  • hosted MCP conformance workflow
  • hosted Pages build without Node 20 or mdBook fallback warnings
  • all repository PR checks
  • actionlint, ShellCheck, Docker Compose validation, and mdBook build/tests
  • workspace formatting, Clippy, and nextest with rmcp 3.1.2

Signed-off-by: lucarlig <luca.carlig@ibm.com>
Signed-off-by: lucarlig <luca.carlig@ibm.com>
@lucarlig
lucarlig force-pushed the user/luca/mcp-conformance-ci branch from 16486da to 3dd0371 Compare August 11, 2026 10:34
Signed-off-by: lucarlig <luca.carlig@ibm.com>
Signed-off-by: lucarlig <luca.carlig@ibm.com>
@lucarlig lucarlig changed the title Add live MCP 2026-07-28 conformance CI Add live-stack MCP 2026-07-28 conformance testing Aug 11, 2026
@lucarlig lucarlig self-assigned this Aug 11, 2026
@lucarlig
lucarlig requested a review from Lang-Akshay August 11, 2026 11:03
@lucarlig
lucarlig marked this pull request as ready for review August 11, 2026 11:26
Signed-off-by: lucarlig <luca.carlig@ibm.com>
Signed-off-by: lucarlig <luca.carlig@ibm.com>
@lucarlig
lucarlig marked this pull request as draft August 11, 2026 11:36
@lucarlig
lucarlig marked this pull request as ready for review August 11, 2026 12:15
@lucarlig
lucarlig requested review from madhu-mohan-jaishankar and removed request for Lang-Akshay August 11, 2026 13:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant