Repository navigation
Streamhouse (v3.0, next): streamhouse cluster, Connect sink segment, IO/KCW/RCW/RCN codes - #11
Merged
Merged
Conversation
…IO/KCW/RCW/RCN codes The simulator's v3.0 Streamhouse track (Apache Iceberg and its Kafka Connect sink) documents its URL grammar in playground-url-api.md (website PR #977). All of it is at release stage next: the public playground opens cluster=streamhouse as single-dc and skips the codes until it ships the streamhouse feature. sim-link learns it now: - actionlog: IcebergOp (IO:<op request as JSON>), KillConnectWorker (KCW), RestartConnectWorker (RCW) and RestartConnector (RCN), each payload escaped like a produce value. IO ports the playground's parseIcebergOpRequest shape check (exact keys and JSON types for all 26 ops) and writes the JSON the way JSON.stringify does (key order kept, JavaScript number form), so a request the playground would skip is an error and a decoded link re-encodes byte for byte. - mcpserver shape: cluster streamhouse (3x3x3, the shared 9-broker / 6-partition caps) and shape.connectSink, size tail index 18 (<workers>.<tasksMax>.<commitIntervalMs>.<commitTimeoutMs>.<0|1>), validated against the playground's clamp. A slug without tail index 14 decodes without the preset's Schema Registry, so every streamhouse slug carries at least "1" there; an unset tasksMax follows the playground's clamp to the partitions of orders. control-iceberg and diskless topics are refused. - mcpserver actions: iceberg_op (icebergOp object), kill_connect_worker and restart_connect_worker (workerId), restart_connector (connectorName), in build_playground_url, the echo and validate_actions. - Tests: the doc's IO/KCW/RCW/RCN examples verbatim, every Iceberg op shape, rejections, both property generators, 22 SIZE-SINK fixtures dumped from the website's TypeScript codec (main 27b30dca), and an MCP end-to-end build. - docs/mapping.md and the README describe the cluster, the sink and the codes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The simulator's v3.0 Streamhouse track (Apache Iceberg and Iceberg's Kafka Connect sink) documents its URL grammar in
docs/playground-url-api.md(website PR #977, merged). sim-link now writes and reads all of it. Everything here is at release stagenext: the public playground does not ship thestreamhousefeature yet, so it openscluster=streamhouseas single-dc and skips the four codes like an unknown kind. Tool descriptions, README anddocs/mapping.mdsay "v3.0, next" wherever these appear.cluster: "streamhouse"?cluster=streamhouseorders3x RF 3, a Schema Registry, 2 Connect workers running the Iceberg sinkorders-sinkinto the tableordersshape.connectSink<workers>.<tasksMax>.<commitIntervalMs>.<commitTimeoutMs>.<0|1>orders, interval 1000-30000 / timeout 2000-60000 in steps of 500, schema evolution)iceberg_op+icebergOpIO:<op request as JSON>kill_connect_worker+workerIdKCW:<workerId>restart_connect_worker+workerIdRCW:<workerId>restart_connector+connectorNameRCN:<name>actionlog/streamhouse.go: the four kinds. Payloads are escaped like a produce value (~to~t,,to~c).IOports the playground'sparseIcebergOpRequest(exact keys and JSON types per op, so a request the playground would skip is an error on encode and decode) and writes the JSON asJSON.stringifydoes (key order kept, JavaScript number form). That way a link sim-link decodes re-encodes byte for byte.mcpserver/shape.go: thestreamhousetopology (shared 9-broker / 6-partition caps),ClusterShape.ConnectSink, the 19-segment tail. The shape refusescontrol-iceberg(the sink's control topic, which the playground renames) and diskless topics or timing (the sink runs on classic topics only). Values are checked, not clamped.size=4x3x3opens Streamhouse with no registry). So every streamhouse slug the tool writes carries at least1at index 14 (4x3x3xxxxxxxxxxxxxxx1). The tool cannot turn the registry off.tasksMaxon a 1-partitionordersis lowered to 1, as the playground does. That makes the sink non-default, so the segment is written (3x1x3xxxxxxxxxxxxxxx1xxxx2.1.2000.6000.1), which is the playground's own canonical slug.mcpserver/actions.go: the four kinds inbuild_playground_url, the echo andvalidate_actions.icebergOpis a JSON object, so its keys arrive unordered and are written sorted.mapping/testdata/ts-fixtures.txt: 22SIZE-SINKlines dumped from the website's TypeScript codec (main 27b30dca) with a throwaway tsx script, not checked in.Files changed
README.md,docs/mapping.md,actionlog/{actionlog,decode,streamhouse}.go,actionlog/{actionlog,property,streamhouse}_test.go,mapping/testdata/ts-fixtures.txt,mcpserver/{actions,mcpserver,shape}.go,mcpserver/streamhouse_test.go.Commands run
go vet ./...,go vet -tags e2e ./e2e/,go test -count=1 ./...: all pass locally (Go 1.27.4, Windows).gofmt -lon the changed files (LF-normalized): clean.e2e/run.sh, Docker): not run locally; left to this PR's CI matrix.Tests added
IO:set_properties and append,KCW:connect-1@8000,RCW:connect-1@20000,RCN:orders-sink@25000) decode and re-encode byte for byte. Typed wire forms; JSON canonicalization pinned to the playground's decode+encode output (whitespace,1.0,1e3,é, repeated keys);jsNumber; one accepted request per op (all 26) and 16 rejections; empty ids; both property generators.SIZE-SINKfixtures with a registry replayed (the 3 without one are counted and skipped), presets, and 14 refusals.TestBuildURLStreamhousebuilds a streamhouse link with a sink and all four kinds, checks the echo and required fields, and previews the doc'sIOappend example invalidate_actions.Known risks
nextsurfaces: if the grammar changes before v3.0 is cut, the simulator changes first and sim-link follows.IOshape check is a port; a new op or field on the website needs the same change here, or sim-link refuses a request the playground would take.streamhousefeature (the public one today) openscluster=streamhouseas single-dc and drops the codes.Not covered (follow-up)
warpstream-3azandredpanda-3az(website #975), with their codes and segments (CP:0/CP:1, tail index 17c(Ripcord) anda(rack awareness), the lightning topic fieldl), are not in sim-link yet. sim-link refuses those clusters ("not a free-play topology").3x3x3xxxxxxxxxxxxxxxxxxx3.3.5000.10000.0has no tail index 14, so it opens Streamhouse without its Schema Registry; the slug the playground writes for that sink with the preset's registry is3x3x3xxxxxxxxxxxxxxx1xxxx3.3.5000.10000.0.next, so a tag can wait for v3.0 unless the maintainer wants it earlier; tag only on the maintainer's go.🤖 Generated with Claude Code