feat(binding-llm): canonical event vocabulary — choice index, token detail by jfallows · Pull Request #2557 · aklivity/zilla

jfallows · 2026-09-08T17:18:40Z

Description

Extends LlmFlushEx's block-lifecycle skeleton (from #2476) with the two places OpenAI's parallel completions and per-token detail need to survive in the canonical vocabulary:

choiceIndex (int32 = 0) added to messageStart, blockStart, blockEnd, finish, and raw — the (choice, block) compound index's outer half. Anthropic is always choice 0, so every existing dialect mapping is unaffected; OpenAI's n > 1 becomes one messageStart per parallel completion, distinguished by choiceIndex. usage stays choiceIndex-free since every dialect reports it aggregated across choices, never per-choice.
logProbability (nullable) added to LlmDataEx — the one per-delta detail the vocabulary carries directly, for dialects exposing per-token detail (OpenAI logprobs), without reopening the DATA/FLUSH split from binding-llm.spec: llm.idl — LlmBeginEx, LlmDataEx, LlmFlushEx union #2476 or growing the vocabulary for the full log-probability structure — richer detail (top-N alternatives, token bytes) stays behind LlmNativeFlushEx.

Documents the three lossiness cases called out in the issue as doc comments alongside the fields they concern:

choiceIndex and logProbability both drop out on any cross-dialect route to Anthropic (structurally exactly one choice, no per-token detail).
Message-start input token counts: resolved by noting the existing design already decouples inputTokens into its own deferred usage event rather than living on messageStart, so a source that discloses tokens late (OpenAI) just emits usage late instead of needing a placeholder-then-correct.

Verified ./mvnw clean install -pl incubator/binding-llm.spec,incubator/binding-llm.conf,incubator/binding-llm -am passes end-to-end: checkstyle, license headers, and all 59 existing unit tests pass unaffected — no Java code consumes the generated flyweight types yet (matching precedent from #2476, which also shipped without tests since it's vocabulary/type definition only).

This branch is built on top of claude/issue-2480-us0ae1 (#2556, the LlmDialect/LlmDialectFactorySpi SPI), which itself stacks on claude/issue-2479-2qhkfm (#2554) and claude/issue-2478-76f5ry (#2553), so this diff includes all three PRs' commits until they merge to develop — at which point this PR's diff will shrink to just this vocabulary commit.

Fixes #2481

🤖 Generated with Claude Code

https://claude.ai/code/session_01AYGZDrytLjmG2AQqUN1hsu

…lects Translates between each dialect's native streaming event sequence and the canonical vocabulary from #2557, in both directions: - LlmAnthropicEventMapper: holds input_tokens from message_start until the paired usage event at message_delta; tracks the currently open block's type to know when content_block_stop needs a canonical blockEnd (tool calls only) and to route content_block_delta payloads (text_delta vs input_json_delta) on encode. - LlmOpenAiEventMapper: translates OpenAI's tool-call-only index space into the canonical (Anthropic-shaped) block index via a per-stream map, and synthesizes blockEnd lazily -- deferred until the next tool call starts or the stream finishes, since OpenAI has no explicit block-close event. Unit-tested against both worked-example tables from the issue (message role/content/tool-call cardinality changes in each direction), plus the held-usage/already-consumed and lazy-blockEnd edge cases. Fixes #2482 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019BHBkLjV2tcbNxMEgxkpSw

Scaffold incubator/binding-llm.spec per AGENTS.md conventions and define LlmBeginEx, LlmDataEx, and the LlmFlushEx union, modelled on binding-mcp.spec's idl. LlmBeginEx carries dialect only; model routing is deferred. LlmDataEx has no fields: content flows through the DATA frame's own payload octets and INIT/FIN through its existing flags, so nothing survives in the extension once block identity moves to the FLUSH plane. LlmFlushEx is a 7-case union covering message start, block start/end, finish, usage, keepalive, and an opaque native/raw case for re-encoding events a same-dialect route doesn't recognize. Fixes #2476 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AuZoMsETwEczJx3cbkb8EJ

@Incubating

Scaffolds incubator/binding-llm and incubator/binding-llm.conf, modelled on binding-mcp's SERVER/CLIENT BindingContext structure. LlmBindingInfo is annotated @Incubating so type: llm config loading is gated behind ZILLA_INCUBATOR_ENABLED via FeatureFilter, matching the AmqpBindingInfo/ PgsqlBindingInfo/RisingwaveBindingInfo precedent. Fixes #2477 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0142buJWS7C89AKr9uDJtSy4

…t-type Registers by content-type and hands back a per-stream LlmContentDecoder; stays in an internal, unexported package for now with no concrete implementation registered yet. Fixes #2478 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mw32oxEw24fLt5Ypakj6pH

…lects Translates between each dialect's native streaming event sequence and the canonical vocabulary from #2557, in both directions: - LlmAnthropicEventMapper: holds input_tokens from message_start until the paired usage event at message_delta; tracks the currently open block's type to know when content_block_stop needs a canonical blockEnd (tool calls only) and to route content_block_delta payloads (text_delta vs input_json_delta) on encode. - LlmOpenAiEventMapper: translates OpenAI's tool-call-only index space into the canonical (Anthropic-shaped) block index via a per-stream map, and synthesizes blockEnd lazily -- deferred until the next tool call starts or the stream finishes, since OpenAI has no explicit block-close event. Unit-tested against both worked-example tables from the issue (message role/content/tool-call cardinality changes in each direction), plus the held-usage/already-consumed and lazy-blockEnd edge cases. Fixes #2482 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019BHBkLjV2tcbNxMEgxkpSw

Closes the non-streaming half of #2478's own scope: "Non-streaming application/json goes through the same abstraction as one event, rather than a special-cased branch." Only text/event-stream had an LlmContentDecoderSpi implementation; application/json requests (non-streaming dialect responses) had no decoder to dispatch to. LlmJsonContentDecoder treats the entire buffered document as a single event (one data + one flush call, no framing loop), mirroring LlmSseContentDecoder's structure and unit-test conventions. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mw32oxEw24fLt5Ypakj6pH

…lects Translates between each dialect's native streaming event sequence and the canonical vocabulary from #2557, in both directions: - LlmAnthropicEventMapper: holds input_tokens from message_start until the paired usage event at message_delta; tracks the currently open block's type to know when content_block_stop needs a canonical blockEnd (tool calls only) and to route content_block_delta payloads (text_delta vs input_json_delta) on encode. - LlmOpenAiEventMapper: translates OpenAI's tool-call-only index space into the canonical (Anthropic-shaped) block index via a per-stream map, and synthesizes blockEnd lazily -- deferred until the next tool call starts or the stream finishes, since OpenAI has no explicit block-close event. Unit-tested against both worked-example tables from the issue (message role/content/tool-call cardinality changes in each direction), plus the held-usage/already-consumed and lazy-blockEnd edge cases. Fixes #2482 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019BHBkLjV2tcbNxMEgxkpSw

…odecSpi LlmContentDecoderSpi and LlmContentEncoderSpi (the latter previously living downstream in #2571) let a content-type register a decoder with no matching encoder, or vice versa, since each was ServiceLoader-discovered and dispatched independently. LlmContentCodecSpi makes "this content-type is fully supported" one type-enforced fact: a single contentType() key with both supplyDecoder() and supplyEncoder(), one META-INF/services registration per content-type, one LlmContentCodecFactory dispatching both directions. Pulls the internal/encode/ base package and its text/event-stream implementation (LlmContentEncoder, LlmSseContentEncoder) forward from #2571 so the collapse can happen where decode already lives, rather than forking that package ahead of its own introduction there; #2571 will need to rebase on top of this and drop its now-duplicate copies. LlmSseContentDecoder, LlmSseContentEncoder, LlmJsonContentDecoder widen from package-private to public (unchanged otherwise) since their new LlmSseContentCodecSpi/LlmJsonContentCodecSpi providers construct them from the sibling internal.codec package. Adds LlmJsonContentEncoder (new): the application/json inverse of LlmJsonContentDecoder, copying content bytes through unchanged with no framing on flush. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mw32oxEw24fLt5Ypakj6pH

Defines the pluggable-dialect contract for binding-llm, exported from the start (unlike LlmContentDecoderSpi, which stays internal): LlmDialect exposes name()/detect()/contentType() plus supplyDecoder(Kind)/ supplyEncoder(Kind) returning common-json JsonTransform stages, and LlmDialectFactorySpi is the ServiceLoader-registered entry point. HttpHeaders is a minimal read-only accessor for detect(path, headers), since no HTTP header abstraction previously existed in this codebase and pulling in jakarta.ws.rs would add a dependency never otherwise used here. Kind is nested on LlmDialect, distinguishing request/response schemas. No concrete dialect implementations yet (OpenAI/Anthropic land later) -- module-info.java exports the dialect package and declares uses without a corresponding provides. Unit-tested via a stub LlmTestDialect/ LlmTestDialectFactorySpi registered under test-scope META-INF/services, mirroring this module's existing LlmContentDecoderSpi/ LlmTestContentDecoderFactorySpi pattern. Fixes #2480 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016NcAVwRPN1Pwjzpobr75w6

…er instance contentType() previously took no parameters, so a dialect could only report one fixed content-type for its lifetime -- insufficient for an API whose response framing (event-stream vs. a single JSON document) depends on a flag in the request body, since neither contentType() nor detect(String, HttpHeaders) offered any way to inspect it. Adds HttpRequestBody, a minimal read-only scalar-member accessor mirroring HttpHeaders, and changes contentType() to contentType(Kind, HttpHeaders, HttpRequestBody): Kind lets request and response resolve independently (a dialect's request body content-type can be fixed while its response varies), and the headers/body context lets that resolution depend on the actual request rather than being fixed at dialect-instance-creation time. Both parameters are nullable for callers without that context available. LlmTestDialect now resolves text/test-event-stream for a streaming response and application/test+json otherwise, exercising the new per-Kind, per-request resolution the stub previously couldn't express. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016NcAVwRPN1Pwjzpobr75w6

…velope, not whole-body buffering Replaces LlmDialect's HttpHeaders/HttpRequestBody/contentType() with the engine's existing ModelEnvelope/ModelTransform (runtime/engine/.../model/), reusing machinery this codebase already has instead of inventing LLM-specific buffering to resolve text/event-stream vs. application/json, or a model name, by peeking one field of an otherwise-unbuffered body. detect(ModelEnvelope) folds :path/:method into the same envelope as ordinary headers -- no separate path parameter. supplyDecoder/supplyEncoder now take (Kind, ModelEnvelope) and return ModelTransform: a per-field stage that can extract a field (e.g. a model name) into the envelope while the body still flows through unchanged, mirroring KafkaExtractTransform (runtime/binding-kafka/.../cache/) -- so a caller reads that signal back off the envelope as decoding proceeds rather than buffering the whole body first to inspect it. contentType() is removed entirely: nothing in this shape needs it once the streaming/non-streaming signal is just another envelope entry a caller reads after extraction. common-json/JsonTransform is no longer used anywhere in this module now that the dialect SPI itself doesn't need it, so the dependency comes back out of module-info.java and pom.xml along with it. LlmTestDialect/LlmTestDialectFactorySpi become LlmTestConditionalDialect/ LlmTestConditionalDialectFactorySpi, since what they now demonstrate is exactly this: request detection and model-name extraction conditional on envelope contents, not a fixed per-instance answer. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016NcAVwRPN1Pwjzpobr75w6

…ken detail Extend LlmFlushEx's block-lifecycle skeleton (from #2476) with the two places OpenAI's parallel completions and per-token detail need to survive in the vocabulary, per the issue's scope: - choiceIndex (default 0) on messageStart, blockStart, blockEnd, finish, and native/raw: the (choice, block) compound index's outer half. Anthropic is always choice 0, so every existing dialect mapping is unaffected; OpenAI's n > 1 becomes one messageStart per parallel completion, distinguished by choiceIndex. usage stays choiceIndex-free since every dialect reports it aggregated across choices, never per choice. - logProbability (nullable) on LlmDataEx: the one per-delta detail the vocabulary carries directly, for dialects exposing per-token detail (OpenAI logprobs) without reopening the DATA/FLUSH split from #2476 or growing the vocabulary for the full log-probability structure — richer detail than one value per token stays behind LlmNativeFlushEx. Documents the three lossiness cases as doc comments alongside the fields they concern: choiceIndex and logProbability both drop out on any cross-dialect route to Anthropic (structurally exactly one choice, no per-token detail); message-start input token counts are resolved by decoupling inputTokens into its own deferred usage event rather than emitting a placeholder on messageStart and correcting it later, so a source that discloses tokens late (OpenAI) just emits usage late instead of needing a correction event. Fixes #2481 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AYGZDrytLjmG2AQqUN1hsu

…lects Translates between each dialect's native streaming event sequence and the canonical vocabulary from #2557, in both directions: - LlmAnthropicEventMapper: holds input_tokens from message_start until the paired usage event at message_delta; tracks the currently open block's type to know when content_block_stop needs a canonical blockEnd (tool calls only) and to route content_block_delta payloads (text_delta vs input_json_delta) on encode. - LlmOpenAiEventMapper: translates OpenAI's tool-call-only index space into the canonical (Anthropic-shaped) block index via a per-stream map, and synthesizes blockEnd lazily -- deferred until the next tool call starts or the stream finishes, since OpenAI has no explicit block-close event. Unit-tested against both worked-example tables from the issue (message role/content/tool-call cardinality changes in each direction), plus the held-usage/already-consumed and lazy-blockEnd edge cases. Fixes #2482 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019BHBkLjV2tcbNxMEgxkpSw

This was referenced Sep 9, 2026

feat(binding-llm): event sequence mapper for anthropic and openai dialects #2558

Open

feat(binding-llm): dialect detection dispatcher and fixed-dialect config #2568

Open

claude added 3 commits September 11, 2026 19:43

jfallows force-pushed the claude/issue-2481-ip27lo branch from 2c220c5 to d2f8c73 Compare September 11, 2026 20:25

jfallows force-pushed the claude/issue-2481-ip27lo branch from d2f8c73 to 548d254 Compare September 12, 2026 01:43

jfallows force-pushed the claude/issue-2481-ip27lo branch from 548d254 to 037b0ce Compare September 12, 2026 04:16

claude added 5 commits September 14, 2026 22:49

jfallows force-pushed the claude/issue-2481-ip27lo branch from 037b0ce to 300c55f Compare September 14, 2026 23:26

Navigation Menu

feat(binding-llm): canonical event vocabulary — choice index, token detail - #2557

feat(binding-llm): canonical event vocabulary — choice index, token detail#2557
jfallows wants to merge 9 commits into
developfrom
claude/issue-2481-ip27lo

jfallows commented Sep 8, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

jfallows commented Sep 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jfallows commented Sep 8, 2026 •

edited

Loading