[sdk] Parse gRPC messages with a raised protobuf recursion limit

[morhaf-ko]

Fixes #2277.

What happens today

Resource inputs and outputs travel as protobuf Structs, where one level of a user-visible object costs three levels of protobuf nesting: Struct -> map entry -> Value. Protobuf's Java runtime refuses to parse past 100 levels, so the Java SDK tops out at roughly 33 levels of property nesting.

gRPC's default marshaller keeps that default and, until recently, offered no way to change it (grpc/grpc-java#8256). When a provider returns something deeper - the recursive JSON Schema shapes AWS flattens into long generated type chains, for example - the parse fails on the channel rather than on the resource:

io.grpc.StatusRuntimeException: CANCELLED: Failed to read message.
Caused by: io.grpc.StatusRuntimeException: INTERNAL: Invalid protobuf byte sequence
Caused by: com.google.protobuf.InvalidProtocolBufferException:
    Protocol message had too many levels of nesting. May be malicious.

Because the failure is on the channel, the whole program aborts: no resource in the stack gets a plan, and no resource is named in the error.

The fix

grpc-java has exposed ProtoUtils.marshallerWithRecursionLimit since 1.56, and this repo is on 1.57.2, so no dependency bump is needed. GrpcRecursionLimit rebuilds the generated marshallers with the limit raised to 10,000 - matching what the Go engine parses with, so the Java SDK is no longer the narrower end of the connection - and installs them:

• GrpcMonitor and GrpcEngine add a channel interceptor that swaps in the raised response marshaller. This is the path in the issue.
• ResourceProviderService rebinds its service definition so a Java component provider parses deeply nested requests too. A ServerInterceptor cannot do this: the method descriptor has already deserialized the request by the time one is consulted.

Serialization is untouched - protobuf imposes no depth limit when writing.

Tests

GrpcRecursionLimitTest covers both directions with 64 levels of Struct nesting, which is comfortably past the default:

• monitorParsesDeeplyNestedResponses runs a real ResourceMonitor server over loopback and calls it through GrpcMonitor, so it exercises the wiring and a real serialize/parse round trip. (An in-process channel would not reproduce the bug - gRPC hands the message across without serializing it.)
• reboundServiceParsesDeeplyNestedRequests and reboundServiceKeepsItsMethodsAndHandlers cover the server-side rebind.

I checked that monitorParsesDeeplyNestedResponses fails with the original INTERNAL: Invalid protobuf byte sequence when the interceptor is removed from GrpcMonitor.

Notes

• 10,000 is a policy choice. Deep enough that the engine's own limit binds first; bounded, because protobuf parses recursively and an unbounded limit trades a clear error for a StackOverflowError. Happy to move it if you would rather it were lower.
• This does not add the conformance test @Frassle mentioned in AgentcoreGatewayTarget with an inline tool schema breaks pulumi preview for the entire program (protobuf recursion limit) #2277 - that lives in pulumi/pulumi - so it is up for grabs alongside this.

[morhaf-ko]

The three red checks here are the fork-OIDC limitation, not the patch.

prerequisites, Lint Go, and ci-ok each fail after 3-10s on their first step, Fetch secrets from ESC, with Unable to get ACTIONS_ID_TOKEN_REQUEST_URL. ci.yml asks for permissions: write-all, but GitHub withholds the OIDC id-token from pull_request runs triggered by a fork, so pulumi/esc-action cannot mint a staging token. java-sdk-tests is the only job in the workflow that does not start with that step, which is why it is the only one that ran - and it passed. go-tests, test_integrations, test_templates, and examples show as skipped because they gate on prerequisites.

To cover what CI could not run, I ran the suite locally against this branch (mise toolchain, JDK 11, Gradle 8.14.1):

Suite	Result
make lint	pass, 0 issues
make build	pass
cd sdk/java && gradle build testAll	pass - 542 tests, 0 failures (test 469, autoTest 72, isolatedTest 1)
cd pkg && go test -timeout 40m ./...	pass - pulumi-language-java 528s, pkg/codegen/java 126s

make test_integrations does not run on my machine, but it fails identically on an unmodified main checkout - the same seven subtests (stale-parameterized-packageref, package-add, about, provider-maven, stack-transformation, provider-gradle, stack-reference), so it is a pre-existing environment gap rather than anything this branch introduced. The causes are local: the test projects pin Gradle 7.4, which cannot run under the JDK 21 that is my system default (Unsupported class file major version 65 while compiling settings.gradle, before any Java runs), and the about test needs the versioning env vars and maven that CI provisions from versions.txt. examples and test_templates need real cloud credentials and the staging backend, so they are out of reach entirely from outside the org.

For a trusted run, I think hosting it from a pulumi-owned branch - the way #2220 did for #2209 - is the only option that actually covers this branch. /run-acceptance-tests would dispatch, but ci.yml reads env.PR_COMMIT_SHA for the go-lint checkout and that variable is not defined anywhere in the repo, and no job reads github.event.client_payload, so a repository_dispatch run checks out the default branch rather than the PR head. Happy to be told I have misread that.

One reviewable judgement call worth your eyes: I set the limit to 10,000 to match what protobuf-go parses with, so the Java SDK stops being the narrower end of the connection. It is deliberately bounded rather than Integer.MAX_VALUE, because protobuf parses recursively and an unbounded limit trades a clear error for a StackOverflowError. Easy to change if you would rather it were lower.

[morhaf-ko]

Thanks - I checked this branch against the 300 levels l2-large-map uses, and pushed the test depth up from 64 to 300 to match it, so the SDK unit test now pins the same shape the conformance test will exercise.

Where Java sits, measured on this branch:

Nesting depth	Result
300 (the conformance test)	pass
350	pass
1000	StackOverflowError

So the raised limit is not what binds any more - protobuf parses recursively, and the thread's stack gives out somewhere between 350 and 1000 levels, well before the 10,000 limit is reached. That answers the "why 10,000" question better than my earlier note did: the constant exists to stop protobuf's artificial cap of 100 from being the limit, and past that the runtime's own limit applies. I have said as much in a comment on RECURSION_LIMIT rather than tuning the number, since lowering it would just reintroduce an arbitrary ceiling under the real one.

Your 350 observation matches ours, for whatever that is worth as a cross-check.

gradle build testAll stays green at the new depth: 542 tests, 0 failures.

[Frassle]

I think this looks ok