Skip to content

Streamhouse (v3.0, next): streamhouse cluster, Connect sink segment, IO/KCW/RCW/RCN codes - #11

Merged
mmatloka merged 1 commit into
mainfrom
ai/streamhouse-url-grammar
Nov 19, 2026
Merged

mmatloka merged 1 commit into
mainfrom
ai/streamhouse-url-grammar

Conversation

@mmatloka

Copy link
Copy Markdown
Contributor

What

The simulator's v3.0 Streamhouse track (Apache Iceberg and Iceberg's Kafka Connect sink) documents its URL grammar in docs/playground-url-api.md (website PR #977, merged). sim-link now writes and reads all of it. Everything here is at release stage next: the public playground does not ship the streamhouse feature yet, so it opens cluster=streamhouse as single-dc and skips the four codes like an unknown kind. Tool descriptions, README and docs/mapping.md say "v3.0, next" wherever these appear.

MCP input Wire form What it opens or does
cluster: "streamhouse" ?cluster=streamhouse 3 brokers, orders 3x RF 3, a Schema Registry, 2 Connect workers running the Iceberg sink orders-sink into the table orders
shape.connectSink size tail index 18: <workers>.<tasksMax>.<commitIntervalMs>.<commitTimeoutMs>.<0|1> the sink knobs (workers 1-4, tasksMax 1..partitions of orders, interval 1000-30000 / timeout 2000-60000 in steps of 500, schema evolution)
iceberg_op + icebergOp IO:<op request as JSON> one Iceberg table operation (26 ops)
kill_connect_worker + workerId KCW:<workerId> a Connect worker dies
restart_connect_worker + workerId RCW:<workerId> it rejoins the Connect group
restart_connector + connectorName RCN:<name> restarts the sink connector
  • actionlog/streamhouse.go: the four kinds. Payloads are escaped like a produce value (~ to ~t, , to ~c). IO ports the playground's parseIcebergOpRequest (exact keys and JSON types per op, so a request the playground would skip is an error on encode and decode) and writes the JSON as JSON.stringify does (key order kept, JavaScript number form). That way a link sim-link decodes re-encodes byte for byte.
  • mcpserver/shape.go: the streamhouse topology (shared 9-broker / 6-partition caps), ClusterShape.ConnectSink, the 19-segment tail. The shape refuses control-iceberg (the sink's control topic, which the playground renames) and diskless topics or timing (the sink runs on classic topics only). Values are checked, not clamped.
  • Schema Registry on Streamhouse: the preset has a registry, but a slug without tail index 14 decodes without one (size=4x3x3 opens Streamhouse with no registry). So every streamhouse slug the tool writes carries at least 1 at index 14 (4x3x3xxxxxxxxxxxxxxx1). The tool cannot turn the registry off.
  • tasksMax clamp: an unset tasksMax on a 1-partition orders is lowered to 1, as the playground does. That makes the sink non-default, so the segment is written (3x1x3xxxxxxxxxxxxxxx1xxxx2.1.2000.6000.1), which is the playground's own canonical slug.
  • mcpserver/actions.go: the four kinds in build_playground_url, the echo and validate_actions. icebergOp is a JSON object, so its keys arrive unordered and are written sorted.
  • mapping/testdata/ts-fixtures.txt: 22 SIZE-SINK lines dumped from the website's TypeScript codec (main 27b30dca) with a throwaway tsx script, not checked in.

Files changed

README.md, docs/mapping.md, actionlog/{actionlog,decode,streamhouse}.go, actionlog/{actionlog,property,streamhouse}_test.go, mapping/testdata/ts-fixtures.txt, mcpserver/{actions,mcpserver,shape}.go, mcpserver/streamhouse_test.go.

Commands run

  • go vet ./..., go vet -tags e2e ./e2e/, go test -count=1 ./...: all pass locally (Go 1.27.4, Windows).
  • gofmt -l on the changed files (LF-normalized): clean.
  • e2e (e2e/run.sh, Docker): not run locally; left to this PR's CI matrix.

Tests added

  • codec: the doc's examples verbatim (IO: set_properties and append, KCW:connect-1@8000,RCW:connect-1@20000,RCN:orders-sink@25000) decode and re-encode byte for byte. Typed wire forms; JSON canonicalization pinned to the playground's decode+encode output (whitespace, 1.0, 1e3, é, repeated keys); jsNumber; one accepted request per op (all 26) and 16 rejections; empty ids; both property generators.
  • shape: the 19 SIZE-SINK fixtures with a registry replayed (the 3 without one are counted and skipped), presets, and 14 refusals.
  • MCP: TestBuildURLStreamhouse builds a streamhouse link with a sink and all four kinds, checks the echo and required fields, and previews the doc's IO append example in validate_actions.

Known risks

  • next surfaces: if the grammar changes before v3.0 is cut, the simulator changes first and sim-link follows.
  • The IO shape check is a port; a new op or field on the website needs the same change here, or sim-link refuses a request the playground would take.
  • A playground without the streamhouse feature (the public one today) opens cluster=streamhouse as single-dc and drops the codes.

Not covered (follow-up)

  • warpstream-3az and redpanda-3az (website #975), with their codes and segments (CP:0/CP:1, tail index 17 c (Ripcord) and a (rack awareness), the lightning topic field l), are not in sim-link yet. sim-link refuses those clusters ("not a free-play topology").
  • Docs found while porting (website, not this repo): the §11.3 sink example 3x3x3xxxxxxxxxxxxxxxxxxx3.3.5000.10000.0 has no tail index 14, so it opens Streamhouse without its Schema Registry; the slug the playground writes for that sink with the preset's registry is 3x3x3xxxxxxxxxxxxxxx1xxxx3.3.5000.10000.0.
  • Release: unreleased since v0.21.0 is this PR only. Its surfaces are next, so a tag can wait for v3.0 unless the maintainer wants it earlier; tag only on the maintainer's go.

🤖 Generated with Claude Code

…IO/KCW/RCW/RCN codes

The simulator's v3.0 Streamhouse track (Apache Iceberg and its Kafka Connect
sink) documents its URL grammar in playground-url-api.md (website PR #977).
All of it is at release stage next: the public playground opens
cluster=streamhouse as single-dc and skips the codes until it ships the
streamhouse feature. sim-link learns it now:

- actionlog: IcebergOp (IO:<op request as JSON>), KillConnectWorker (KCW),
  RestartConnectWorker (RCW) and RestartConnector (RCN), each payload escaped
  like a produce value. IO ports the playground's parseIcebergOpRequest shape
  check (exact keys and JSON types for all 26 ops) and writes the JSON the way
  JSON.stringify does (key order kept, JavaScript number form), so a request
  the playground would skip is an error and a decoded link re-encodes byte
  for byte.
- mcpserver shape: cluster streamhouse (3x3x3, the shared 9-broker /
  6-partition caps) and shape.connectSink, size tail index 18
  (<workers>.<tasksMax>.<commitIntervalMs>.<commitTimeoutMs>.<0|1>),
  validated against the playground's clamp. A slug without tail index 14
  decodes without the preset's Schema Registry, so every streamhouse slug
  carries at least "1" there; an unset tasksMax follows the playground's
  clamp to the partitions of orders. control-iceberg and diskless topics are
  refused.
- mcpserver actions: iceberg_op (icebergOp object), kill_connect_worker and
  restart_connect_worker (workerId), restart_connector (connectorName), in
  build_playground_url, the echo and validate_actions.
- Tests: the doc's IO/KCW/RCW/RCN examples verbatim, every Iceberg op shape,
  rejections, both property generators, 22 SIZE-SINK fixtures dumped from the
  website's TypeScript codec (main 27b30dca), and an MCP end-to-end build.
- docs/mapping.md and the README describe the cluster, the sink and the codes.
@mmatloka
mmatloka merged commit 593c21f into main Nov 19, 2026
10 checks passed
@mmatloka
mmatloka deleted the ai/streamhouse-url-grammar branch November 19, 2026 13:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant