MCPServerEntry.headerForward.addPlaintextHeaders is silently ignored — never reaches upstream

[lorr1]

Summary

MCPServerEntry.spec.headerForward.addPlaintextHeaders is silently ignored. Headers configured on a remote-URL MCPServerEntry (proxied through a VirtualMCPServer) never reach the upstream MCP server. The field is defined in the CRD and accepted by the operator, but is dropped during the conversion from CR to runtime vmcp.Backend and has no representation anywhere in the vMCP outgoing HTTP client.

Verified in v0.27.1.

Reproduction

MCPServerEntry configured against GitHub's Copilot MCP at api.githubcopilot.com/mcp/, with the X-MCP-Toolsets header configured to select a specific set of toolsets:

apiVersion: toolhive.stacklok.dev/v1alpha1
kind: MCPServerEntry
metadata:
  name: github-copilot-projects
spec:
  remoteUrl: "https://api.githubcopilot.com/mcp/"
  transport: streamable-http
  externalAuthConfigRef:
    name: github-oauth-upstream-inject
  headerForward:
    addPlaintextHeaders:
      X-MCP-Toolsets: "projects,issues,pull_requests,users,repos"

Direct call to https://api.githubcopilot.com/mcp/ with the same token + same X-MCP-Toolsets header returns 39 tools strictly filtered to the 5 requested toolsets, including projects_get, projects_list, projects_write.

Through the vMCP, the upstream returns 43 tools matching GitHub's default toolset for the Copilot endpoint — which includes copilot, orgs, secret_protection tools (NOT requested) and is missing projects_* (which IS requested). The fingerprint matches "header not sent → GitHub returns defaults."

Root cause

Source trace in v0.27.1:

• CRD field exists: cmd/thv-operator/api/v1beta1/mcpserverentry_types.go:38-41 defines HeaderForward *HeaderForwardConfig; cmd/thv-operator/api/v1beta1/mcpremoteproxy_types.go:12-16 defines HeaderForwardConfig.AddPlaintextHeaders.

• Reconciler doesn't read it: pkg/vmcp/workloads/k8s.go:503-586 (mcpServerEntryToBackend) reads RemoteURL, Transport, CABundleRef, ExternalAuthConfigRef — but never reads entry.Spec.HeaderForward.

• No field to carry it: pkg/vmcp/types.go:290-333 vmcp.Backend has no field for static headers.

• No middleware to apply it: pkg/vmcp/client/client.go:265-311 shows the vMCP outgoing HTTP client's transport chain is size limiter → trace propagation → identity propagation → auth → HTTP. There is no static-header layer.

• MCPRemoteProxy works via a separate code path: cmd/thv-operator/controllers/mcpremoteproxy_runconfig.go:270-297 calls runner.WithHeaderForward(proxy.Spec.HeaderForward.AddPlaintextHeaders), which flows through pkg/runner/middleware.go:301-310 into pkg/transport/middleware/header_forward.go:100-137. MCPServerEntry bypasses RunConfig entirely and goes straight to the vMCP Backend struct, so it never picks up this middleware.

Suggested fix

1. Add a HeaderForward map[string]string field (or equivalent) to vmcp.Backend in pkg/vmcp/types.go.
2. Populate it from entry.Spec.HeaderForward.AddPlaintextHeaders in mcpServerEntryToBackend (pkg/vmcp/workloads/k8s.go).
3. Add a static-header middleware layer in the vMCP HTTP client transport chain (pkg/vmcp/client/client.go) that reads the per-backend headers and injects them into outgoing requests.
4. Consider doing the same for addHeadersFromSecret so secret-sourced headers are also forwarded.

Impact

Any vMCP backend that depends on header-based configuration of the upstream (e.g. GitHub's X-MCP-Toolsets, API keys passed via custom headers, correlation IDs) silently does nothing. The field accepting valid input without applying it is the worst form — there is no validation error, no warning in logs, nothing in kubectl describe to indicate the headers aren't being sent.

[lorr1]

Update: PR #5239 only fixed half the bug

Verified on staging today running enterprise vmcp 0.1.36-main.46.c905a7cc (toolhive submodule pinned at 97b0cc3f = v0.27.2). Original symptom from the issue body is still reproducible — MCPServerEntry/github-copilot-projects configured with X-MCP-Toolsets: projects,issues,pull_requests,users,repos, GitHub returns the 43-tool default set with no projects_*.

Re-traced end-to-end. The operator and aggregator paths added in #5239 are working correctly:

• Operator emits TOOLHIVE_HEADER_FORWARD_GITHUB_COPILOT_PROJECTS={"addPlaintextHeaders":{"X-MCP-Toolsets":"..."}} on the vmcp pod ✓ (verified live)
• pkg/vmcp/cli/header_forward_env.go::readHeaderForwardFromEnv parses it ✓
• pkg/vmcp/aggregator/discoverer.go:292 assigns it to Backend.HeaderForward ✓
• pkg/vmcp/registry.go::BackendToTarget copies it to BackendTarget.HeaderForward ✓
• pkg/vmcp/client/client.go:309 wraps the round-tripper chain via buildHeaderForwardTripper ✓

But there are two HTTP-client construction sites in vmcp, and #5239 only wired one of them:

Code path	Uses target.HeaderForward?	When invoked
pkg/vmcp/client/client.go::defaultClientFactory (line 309)	Yes (#5239)	Startup-only capability discovery via the aggregator
pkg/vmcp/session/internal/backend/mcp_session.go::createMCPClient (~line 262)	No	Every long-lived per-client MCP session

When an MCP client connects to vMCP and calls tools/list, the request flows through the session factory (pkg/vmcp/cli/serve.go:332 → NewSessionFactory → backend.NewHTTPConnector → MultiSessionFactory.MakeSessionWithID → createMCPClient). That clients transport chain is http.DefaultTransport → authRoundTripper → identityRoundTripper — no headerForwardRoundTripper, and target.HeaderForward is never consulted. The same gap also affects subsequent CallTool / ReadResource / GetPrompt invocations on the session client.

grep -rn "target.HeaderForward" pkg/vmcp/ hits only pkg/vmcp/client/client.go:309 today. After the fix it should hit at least two files.

Proposed fix

1. Extract buildHeaderForwardTripper + resolveHeaderForward from pkg/vmcp/client/ into a shared subpackage (e.g. pkg/vmcp/headerforward/transport).
2. Plumb a secrets.Provider into NewHTTPConnector (currently only takes registry — see pkg/vmcp/session/factory.go:194).
3. Call the shared helper from createMCPClient between authRoundTripper (inner) and identityRoundTripper (outer), matching the ordering in client.go:299-309 so auth still wins on Authorization.
4. Regression test in mcp_session_capabilities_test.go: build a session with HeaderForward.AddPlaintextHeaders["X-MCP-Toolsets"], intercept the outbound HTTP, assert the header is on the tools/list POST.

This is a vMCP anti-pattern #8 case — two parallel HTTP-client builders that should have been unified. The existing regression test (TestHeaderForwardRoundTripper_InjectsPlaintext) only exercises the working path, which is why the gap shipped.