feat(binding-llm): llm server/client/proxy — OpenAI↔Anthropic translation via JsonPipeline

[jfallows]

Fixes #2596

Description

Introduces the llm binding (kind: server/client/proxy) for bridging OpenAI Chat Completions and Anthropic Messages traffic, plus examples/llm.proxy demonstrating it.

Wire model. Every part of the app-facing protocol — content and structural signals alike (event name, finish reason, usage) — is a single DATA frame carrying a minimal LlmDataEx extension (a type discriminator plus logProbability/inputTokens/outputTokens), fragmented via ordinary init/fin flags like any oversized value. There is no separate LlmFlushEx union frame for structural signals.

Translation, directly against common-json. Both directions are expressed against common-json's JsonPipeline/JsonStream/JsonTransform/JsonSink/JsonSchema, not the generic engine ModelHandler/ModelPipeline/ModelTransform SPI or a catalog-resolved schema:

• Request-side field renaming (max_tokens↔maxOutputTokens, top_p↔topP, etc.) is a JsonTransform that captures/defers a top-level key on KEY_NAME, decides the rename on the paired scalar value, and writes a resumable KEY-then-VALUE pair to its sink.
• Response-side streaming translation between dialects (OpenAI SSE chunks ↔ Anthropic SSE events, including the flat-tool-call-delta ↔ separate-content-block fan-out) is a per-dialect decode JsonTransform + encode JsonSink pair, sharing one long-lived JsonPipeline per response stream: the decode stage emits genuine START_DOCUMENT/END_DOCUMENT cycles (one per canonical action) to its sink rather than an ad hoc convention, and JsonPipeline.nextDocument() (added to common-json in this PR) advances the pipeline between native chunks without cascading a state-wiping reset() to either stage — reset() is reserved for REJECTED abandonment and stream teardown.
• Each dialect compiles its own request/response JsonSchema directly from a bundled classpath resource at construction; there is no sys: namespace catalog or ModelHandler-backed schema validation anywhere in the binding.

Dialects. openai and anthropic, each with detection (method/path/content-type or headers), credential-header conventions (Authorization: Bearer, x-api-key), and a native JSON schema for both directions.

Routing. llm(proxy) routes purely on LlmBeginEx.dialect/model — both already resolved by the llm(server) that produced the stream — never on request content, mirroring how mcp(proxy) routes on tool name. Supports both cross-dialect translation (the default per dialect) and intra-dialect, model-based routing to a second same-dialect deployment (no translation on that leg).

Credential pass-through. options.authorization templates the caller's own presented credential ("Bearer {credentials}" / "{credentials}") rather than configuring a secret of Zilla's own, so a caller's real API key forwards upstream unchanged in either direction.

examples/llm.proxy demonstrates both frontends (north_llm_server_openai/_anthropic), the shared north_llm_proxy, and four llm(client) legs (two dialects × primary/secondary). etc/zilla.yaml targets the real OpenAI/Anthropic APIs over TLS by default; .github/compose.mock.yaml (merged via .github/.env.test's COMPOSE_FILE) swaps in four mock backends and a mock-hostname zilla.yaml for CI/local testing without real API keys, mirroring the compose-overlay pattern used for analogous mock-vs-real cases elsewhere.

Testing

• k3po ITs: LlmServerIT, LlmClientIT, LlmProxyIT (incubator/binding-llm), ApplicationIT/NetworkIT (incubator/binding-llm.spec), including openai/anthropic streaming, non-streaming, tool-call round trips, guarded-authorization, cross-dialect, and flow-control (10k/100k) scenarios.
• Unit tests for dialect detection/resolution, request field-rename transforms, and the per-dialect canonical decode/encode transforms (driven through a real JsonPipeline, not mocked collaborators), including a test proving cross-chunk state survives nextDocument().
• examples/llm.proxy's own ./.github/test.sh (8 assertions: both cross-dialect translation directions, both tool-call round trips, both credential pass-through directions, both model-based secondary-deployment routes) run against a locally built image.
• grep -rn "engine\.model\." incubator/binding-llm/src returns nothing — confirms no engine Model SPI usage remains anywhere in the binding.

---

Generated by Claude Code

[jfallows]

Build (25) failed on commit 4737831e (superseded) with common-protobuf test failures — unrelated to this PR's own module, but caused by it: this PR's JsonTokenizer fix (fixing binding-llm's streaming-100k hang) made the tokenizer correctly fragment a scalar value that doesn't start at byte 0 of its input window, which ProtobufJsonParserImpl wasn't prepared for — its messageStep()/mapStep() already decline an incomplete KEY_NAME fragment (leaving it unacted-on and re-pulling), but valueStep()/arrayStep()/the map-entry value step never applied that same treatment to a VALUE_STRING/VALUE_NUMBER token.

Fixed and pushed in 41f14db6: give scalar values the same decline as keys already get. ProtobufJsonTest/ProtobufJsonChunkingTest (the two suites that caught this) are green again, and the full common-protobuf and binding-llm module verifies pass together at the current head.

---

Generated by Claude Code

[jfallows]

Third distinct CI failure (unrelated to the previous common-protobuf fix): binding-llm.conf's JaCoCo check failed — instructions covered ratio is 0.99, but expected minimum is 1.00.

Root cause: two branches in LlmOptionsConfigAdapter had no test coverage:

• defaultPort(String)'s ternary only had its HTTPS branch exercised (shouldReadServerOptionWithDefaultHttpsPort); every other server-option test either supplies an explicit port or uses https, so the HTTP-default-port (80) branch was never taken.
• adaptServer()'s catch (URISyntaxException ex) {} block was never reached — the existing shouldReadMalformedServerOptionAsAbsent test's input ("not-a-host-and-port") parses successfully as a relative URI and only fails the subsequent host/scheme check, so it never throws.

Fixed in bd99d81 by adding two unit tests: shouldReadServerOptionWithDefaultHttpPort ("http://example.com", asserts port 80) and shouldReadInvalidServerUriAsAbsent (a malformed IPv6-literal host that genuinely throws URISyntaxException). Verified locally: binding-llm.conf now passes clean verify with 100% instruction coverage, and the full binding-llm clean verify (including all k3po ITs) still passes against the updated dependency.

---

Generated by Claude Code

[jfallows]

Fourth CI failure, on commit bd99d81: testing (inspect.schema) — the examples/inspect.schema example's golden schema.expected.json fixture had drifted from the actual merged JSON schema.

Root cause: this PR's earlier work aligning llm(client)'s server option with the standard SDK base-URL convention changed its schema description/pattern (from a host:port form to a full base-URL form), but examples/inspect.schema/.github/schema.expected.json — which asserts the complete merged schema byte-for-byte via zilla inspect schema — was never updated to match.

Fixed in b9836c2 by updating the fixture's server property block to the actual current schema (description, format: uri, pattern: ^https?://). Confirmed the diff matches exactly what CI reported, and that no other host:port-pattern occurrences in the fixture are related (the other two are unrelated Kafka servers array options).

---

Generated by Claude Code

[jfallows]

Fifth CI failure, and the significant one: testing (llm.proxy) — the example's own end-to-end assertions failed at the very first one (call_openai_text): curl got error 18, "transfer closed with outstanding read data remaining", for a plain non-streaming request.

Root cause

LlmServerFactory (kind: server) and LlmClientFactory (kind: client) both sit directly upstream of a real http binding. Neither declares a content-length for what it writes (the response body, the outbound request body), so http always uses chunked Transfer-Encoding for these — which means http grants a non-zero padding in its WINDOW frames, to cover its own chunk-framing overhead (chunk-size hex line + CRLFs). Per the established convention elsewhere in the codebase (HttpServerFactory's own replyPad usage — see reserved = length + replyPad throughout that class — and runtime/AGENTS.md's per-stream field table), a sender must add the receiver's declared padding to reserved on every outbound DATA frame, and reserve room for it before sizing the frame's payload.

Both LlmServerFactory and LlmClientFactory ignored window.padding() entirely — no field even tracked it — and sent reserved equal to the raw payload length. Every existing binding-llm k3po test simulates the http peer as a synthetic script that (like most k3po fixtures) grants padding: 0 by default, so this was invisible to the entire existing IT suite; a real http binding's non-zero padding causes it to reject/mis-account for the frame, producing exactly the observed truncated-response symptom on the reply leg, and an analogous hang on the request leg once padding is large enough to matter there too.

Fix (0302f15)

Added replyPad (LlmServerFactory) / initialPad (LlmClientFactory) fields, captured from window.padding() on the real WINDOW, and included them in both the available-window check and the reserved value of each outbound DATA frame. With padding=0 (every pre-existing test fixture), the arithmetic is unchanged, so no existing behavior is affected — confirmed by the full binding-llm/binding-llm.spec test suites passing unmodified.

Added regression coverage that simulates a real http peer's non-zero padding via k3po's option zilla:padding on the accept/connect side (confirmed each new test fails with the pre-fix code and passes with it):

• LlmServerIT#shouldForwardOpenaiNonstreamingWithReplyPadding (response leg)
• LlmClientIT#shouldForwardOpenaiRequestWithReplyPadding (request leg)
• plus their NetworkIT/ApplicationIT peer self-consistency counterparts

Also added LlmProxyIT#shouldRouteOpenaiToAppZero, a small (non-10k) single-frame reply-relay case through the proxy — the existing proxy relay tests only ever exercised multi-frame (10k+) replies. This was ruled out as the source of the bug (it passes both before and after the fix) but closes a real coverage gap, so it's kept.

Note: this sandbox has no working Docker daemon, so I could not re-run examples/llm.proxy's own docker-compose end-to-end test directly. Verification here relies on the k3po regression tests above, which reproduce the exact WINDOW-padding mechanics a real http binding uses. I'd appreciate the next CI run (which does have Docker) confirming testing (llm.proxy) now passes end-to-end.

---

Generated by Claude Code

[jfallows]

Build (25) failed on commit 0302f15, but not from anything in this PR's diff: the actual failure is io.aklivity.zilla.runtime.binding.kafka.internal.stream.CacheProduceIT.shouldReconnectPartitionNotLeader in binding-kafka — a module this PR never touches. This looks like a flaky Kafka-reconnection timing test. Re-running the failed jobs once to confirm.

---

Generated by Claude Code

[jfallows]

testing (llm.proxy) — actual root cause found

The WINDOW-padding fix in 096eb25a's predecessor (0302f15a) was real and necessary but, as the last update noted, did not fix the call_openai_text e2e failure. Confirmed via direct local reproduction (not just CI-log archaeology): with a build containing that fix, curl against the real examples/llm.proxy stack got 200 OK + Transfer-Encoding: chunked headers but zero body bytes, and mock-anthropic's Express/raw-body layer aborted the connection before reading a complete request — meaning zilla itself never sent a complete, validly-framed request body to the backend.

Root cause: LlmClientFactory.java's LlmHttpClient.encodeNet (the code path that forwards a translated request body from llm:client to a real http:client binding) computed its outbound DATA frame flags as initialStarted ? 0 : FLAG_INIT — i.e. it never set FLAG_FIN on any request frame, ever. HttpClientFactory.doEncodeHttp1Body (runtime/binding-http) gates writing the HTTP/1.1 chunk-size-hex prefix on FLAG_FIN and the trailing CRLF on FLAG_INIT. Since FLAG_FIN was never set, every outbound request chunk was missing its size-hex line entirely — the backend received raw JSON bytes with no valid chunked-encoding framing at all, which explains the abort exactly.

Fix (096eb25a): always set both FLAG_INIT | FLAG_FIN on every physical outbound write in encodeNet. Each physical write already becomes its own self-contained wire-level chunk (chunk size scoped to just that write), request headers are written on a separate path unaffected by these flags, and HTTP/2 ignores them entirely — so this is safe regardless of how encodeNet's own backpressure buffering may split a single JSON encode-output slice across multiple physical writes. The now-unused initialStarted field was removed.

Notably, no existing k3po test in this module could have caught this: every LlmClientIT scenario stubs the network side as a synthetic k3po script that reads decoded HTTP begin/data content directly, never exercising the real HttpClientFactory chunk-encoding logic. I'm adding a dedicated regression test that chains llm:client -> http:client -> tcp:client for real and asserts the literal wire bytes, to close that coverage gap — will push separately. Full existing binding-llm unit + IT suite (136 unit tests, 57 ITs) passes unchanged with this fix.

Re-running local docker-compose e2e now to confirm directly before considering this closed.

🤖 Generated with Claude Code

https://claude.ai/code/session_01AKZMN6ckSpx1QpHvaE4DwM

---

Generated by Claude Code

[jfallows]

testing (llm.proxy) — confirmed fixed end-to-end (two bugs, both real)

Root-caused this properly by decoding the actual wire-level frame trace out of the engine's own ring-buffer event log (extracted from a CI artifact and read with a small standalone driver against runtime/command-logs' LogsReader/incubator/command-dump's RingBufferSpy — ground truth beats re-deriving behavior from source alone). That surfaced two distinct, sequential bugs blocking this test, not one:

Bug 1 (fixed in 096eb25a) — LlmHttpClient.encodeNet never set FLAG_FIN on outbound request DATA frames to the real http:client binding, so HttpClientFactory's chunk encoder never wrote the HTTP/1.1 chunk-size-hex prefix. CI re-ran against that commit and the symptom changed (confirmed via the frame trace): the request body now arrives at mock-anthropic correctly framed —

40\r\n{"model":"gpt-4","messages":[{"role":"user","content":"Hello"}]}\r\n
0\r\n\r\n

— but the backend replied 404 Not Found: Cannot POST /v1/messages, a different failure than before (previously the malformed chunking made Express abort mid-parse before routing even ran).

Bug 2 (fixed in 3bb08ded) — examples/llm.proxy's four mock backends (mock-openai(-secondary), mock-anthropic(-secondary)) all registered their route on POST / under a comment claiming south_llm_client_* "always issues its outbound request to '/', regardless of dialect." That was stale: LlmClientFactory actually sends each dialect's real canonical path (/v1/messages for Anthropic, /v1/chat/completions for OpenAI — matching the real upstream APIs), confirmed directly from the same frame trace. Bug 1's malformed body previously masked this, since Express's body-parsing middleware aborted before route matching ever got a chance to 404. Fixed by pointing each mock's route at the dialect's actual path.

Verified for real, not just in theory: downloaded the exact zilla-image artifact CI built from 096eb25a (no local rebuild needed) and ran the full local docker-compose stack with mock fix 3bb08ded applied. All 8 assertions in etc/test/verify.sh pass end-to-end: cross-dialect text + tool-call translation both directions, credential pass-through both directions (confirmed via each mock's own logs), and model-based secondary routing both directions. Also caught and worked around an unrelated local-only footgun: compose.yaml's image: defaults to :latest unless ZILLA_VERSION is set — CI always sets it, so this never bit CI, only my local repro.

Expect testing (llm.proxy) green on the next CI run against 3bb08ded.

🤖 Generated with Claude Code

https://claude.ai/code/session_01AKZMN6ckSpx1QpHvaE4DwM

---

Generated by Claude Code