Skip to content

fix(network): refuse protocol upgrades on JSON-RPC and MCP endpoints - #3753

Open
shiju-nv wants to merge 1 commit into
NVIDIA:mainfrom
shiju-nv:fix/jsonrpc-route-upgrade-inspection
Open

shiju-nv wants to merge 1 commit into
NVIDIA:mainfrom
shiju-nv:fix/jsonrpc-route-upgrade-inspection

Conversation

@shiju-nv

Copy link
Copy Markdown
Collaborator

Summary

JSON-RPC and MCP endpoints apply their rules to each HTTP request, but the proxy could forward a request that also asked to switch protocols. After an upstream accepted, the connection was relayed without inspection. JSON-RPC-family endpoints now refuse any request that carries an Upgrade header with 403 before anything reaches the server, and close the connection if an upstream switches protocols anyway.

Related Issue

No issue required: this is a localized fix to the proxy's upgrade handling.

Changes

  • Add unsupported_upgrade_detail next to the h2c check. Every inspected protocol still refuses h2c. JSON-RPC and MCP endpoints also refuse any request that carries an Upgrade header. Both relay checks that can lead to a protocol switch require that header, so the refusal covers everything the relay would treat as an upgrade.
  • Rename deny_h2c_upgrade_if_requested to deny_unsupported_upgrade_if_requested and call it from relay_jsonrpc as well as route selection, REST and GraphQL. The refusal runs before the L7 policy decision and ignores the enforcement mode, like the existing h2c refusal.
  • The refusal records PolicyDenied for the endpoint and answers {"error":"unsupported_l7_protocol","detail":…}, the same body the forward proxy already used. On the relay paths this replaces the policy-denial body for h2c refusals as well, which told agents to add an allow rule that could not help.
  • Route the forward proxy's inline h2c refusal through the same function. Its OCSF event message now reads "denied unsupported upgrade"; the status detail names which upgrade was refused.
  • In route selection and the forward proxy, close the connection instead of calling handle_upgrade when a JSON-RPC-family endpoint still receives 101, as relay_jsonrpc already did.
  • Document the refusal in architecture/security-policy.md, docs/how-it-works/policies/network-rules.mdx, docs/how-it-works/policies/schema.mdx and docs/observability/logging.mdx, and add unsupported_l7_protocol to the denial table in docs/how-it-works/policies/manage-policies.mdx.

Streamable HTTP traffic is unchanged: POST requests and receive-stream GETs without upgrade headers still pass, and REST and WebSocket endpoints on the same host and port still upgrade as before. Single-endpoint JSON-RPC and MCP now refuse h2c with 403, matching the other L7 paths. openshell policy update already rejects an MCP endpoint that shares a host and port with a differently inspected endpoint, and the MCP docs advise against that layout. A server that also accepts WebSocket needs a separate protocol: websocket endpoint, on a different host or port for MCP, or on a different path for JSON-RPC.

Testing

  • mise run pre-commit passes
  • Unit tests added/updated
  • E2E tests added/updated (if applicable)

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (if applicable)

JSON-RPC and MCP rules apply to each HTTP request, but the proxy could
forward a request that also carried upgrade headers. After an upstream
answered 101, route selection and the forward proxy relayed the
connection without inspection.

Refuse any request that carries an Upgrade header on JSON-RPC-family
endpoints before the L7 policy decision, in every enforcement mode.
Share the check with the existing h2c refusal and call it from
relay_jsonrpc as well. Record the refusal as a policy denial and answer
with the unsupported_l7_protocol error, because no policy rule can
allow the request.

If a JSON-RPC-family endpoint still receives 101, close the connection
instead of relaying raw bytes. Document the refusal and the WebSocket
alternative.

Signed-off-by: Shiju <shiju@nvidia.com>
@johntmyers johntmyers self-assigned this Sep 27, 2026
```

The `error` field is a short machine-readable code (`policy_denied`, `middleware_denied`, `middleware_failed`, `ssrf_denied`, `upstream_unreachable`). The `detail` field is a human-readable explanation suitable for display in an agent transcript. The optional `reason` field, when present, provides the specific denial cause from the policy engine (for example, which binary was not allowed or which rule was missing).
The `error` field is a short machine-readable code (`policy_denied`, `middleware_denied`, `middleware_failed`, `ssrf_denied`, `upstream_unreachable`, `unsupported_l7_protocol`). `unsupported_l7_protocol` means the request used a protocol or upgrade that the endpoint cannot inspect, such as h2c or an upgrade on an MCP or JSON-RPC endpoint; no policy rule can allow it. The `detail` field is a human-readable explanation suitable for display in an agent transcript. The optional `reason` field, when present, provides the specific denial cause from the policy engine (for example, which binary was not allowed or which rule was missing).

@johntmyers johntmyers Sep 27, 2026 •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-defining what the new error here means seems redundant to the table above and also inconsistent as the other error codes are also not re-defined here.

@johntmyers johntmyers added the test:e2e Requires end-to-end coverage label Sep 27, 2026
@github-actions

Copy link
Copy Markdown

Label test:e2e applied for 529b657. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

@johntmyers johntmyers left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gator-agent

PR Review Status

This localized proxy hardening is project-valid, and the initial code review found no blocking defects. The implementation rejects JSON-RPC and MCP upgrade requests before upstream forwarding, preserves the supported non-upgrade paths, and includes matching architecture and Fern documentation.

Blocking findings:

  • No blocking findings remain

Carried findings:

  • None

Non-blocking suggestions:

  • None
Gator metadata
  • Validation: Localized network-proxy security and correctness fix from a repository collaborator with write permission
  • Docs: Architecture and relevant Fern policy and observability docs updated; navigation change not needed
  • Checks: Existing branch, Helm, Trivy, and DCO checks are green; required E2E rerun is queued
  • E2E: test:e2e applied and current-head Branch E2E Checks rerun queued
  • Head SHA: 529b65729bbcce6e6381669047b011458758f704
  • Base SHA: d1a19c70ee3ed730e6e38ab66b76666a57b5e984
  • Merge base SHA: f37d89b5844a50d79726788989b1a5b27aa157e5
  • Patch ID: 73bc65d654777c521da1ba73baf1004c2453ca20
  • Gator payload: 9
  • Review mode: initial
  • Previous reviewed SHA: none
  • Review budget exhausted: no
  • Maintainer decision required: no
  • Next state: gator:watch-pipeline

@johntmyers johntmyers added gator:watch-pipeline Gator is monitoring PR CI/CD status gator:approval-needed Gator completed review; maintainer approval needed and removed gator:watch-pipeline Gator is monitoring PR CI/CD status labels Sep 27, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gator:approval-needed Gator completed review; maintainer approval needed test:e2e Requires end-to-end coverage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants