Skip to content

#305 - extract calypso-sim into a standalone MQTT simulator crate#326

Draft
bracyw wants to merge 30 commits into
Developfrom
305-calypso-sim
Draft

#305 - extract calypso-sim into a standalone MQTT simulator crate#326
bracyw wants to merge 30 commits into
Developfrom
305-calypso-sim

Conversation

@bracyw

@bracyw bracyw commented Jun 28, 2026

Copy link
Copy Markdown
Collaborator

Changes

Extracts the MQTT simulator out of the main calypso binary (deleting the in-tree src/bin/simulate.rs) into a standalone, detached-workspace crate at calypso-sim/ with four input modes — mock heartbeat, interactive keyboard, deterministic replay, and a JSON-RPC-over-stdio stream — kept from fighting over the same topic by a static ownership partition resolved once at startup (the heartbeat cedes any topic a driver owns, up front — no runtime negotiation). Interactive and replay share one recursive scenario model (an action is a list of publish/sleep/invoke steps), so there's no separate keymap-vs-script pair of files to coordinate. Also hoists the shared CAN data-model types (DecodeData/EncodeData/FormatData) into calypso-cangen so the sim reuses them without depending on the Linux-only calypso crate, removes the now-dead simulate path from main, ports the old Python VCU-mimic harness to broker-free Rust tests, keeps the simulate Docker entry point, and adds a path-filtered CI workflow for the crate.

Notes

  • calypso-sim is its own detached workspace (empty [workspace], own Cargo.lock/target). It does not build from a top-level cargo build --allcd calypso-sim first. It has its own path-filtered CI (calypso-sim-ci.yml) that runs only when the crate or its path-deps change.
  • One recursive scenario model for the keyboard/replay side: an action is a named list of steps, and a step is publish · sleep · or invoke another action. Interactive binds an action to a key; --play <action> runs one by name (this replaces the old --script FILE); actions compose by invoking each other, so a scenario is one self-contained file. This collapses the previous four keymap entry shapes, the parallel KeyEntry/KeyMode enums, and the separate script parser into a single primitive — and steps are shape-disambiguated, so there's no order-dependent untagged matching. Invalid/cyclic invokes, duplicate keys, and bad publish-arity are rejected at load. The keymap/script formats were new in this PR, so there's no external consumer to break.
  • Ownership is a static startup partition, not a runtime registry. The mock heartbeat and a foreground driver never negotiate at runtime: any topic the driver owns (a scenario's topics for --key-map/--play, auto-derived; nothing for --stream, where you reserve topics with --disable-topic) is removed from the heartbeat's set up front, so the two publish disjoint topics by construction — and the split is printed at startup. This deleted the Arc<RwLock<…>> ownership registry and the stream claim/release/silence/status RPCs; the stream protocol is now just publish/list_topics/ping.
  • Data model went into calypso-cangen, not daedalus — daedalus is proc-macro = true and can't export plain types, while calypso-cangen is a normal lib both crates already depend on. Main's src/data.rs is now a thin re-export shim, so the decode/encode codegen macros (crate::data::…) are unchanged.
  • Heartbeat mode renamed from autonomous/--auto to mock/--mock (flag, module, and docs).
  • Sim CI fmt step uses plain cargo fmt --check, not --all--all makes cargo-fmt walk the sim's path-deps up into the main workspace and try to format its generated src/proto, which never exists in the sim CI checkout.

Test Cases

All from inside calypso-sim/:

  • cargo run -- --list-topics → prints every simulatable topic, then exits.
  • cargo run → mock heartbeat; publishes randomized values on every topic with a sim_freq in the spec.
  • cargo run -- --key-map manual_sim_buttons.keymap.json --mock → interactive + heartbeat; prints an Ownership: mock heartbeat drives N topic(s); M reserved for the driver … line showing the scenario's topics are ceded, and each action fires on its bound key.
  • cargo run -- --key-map manual_sim_buttons.keymap.json --play demo → runs the demo action (its nested invokes + sleep_ms waits, in order), then exits.
  • cargo run -- --stream, then feed {"jsonrpc":"2.0","id":1,"method":"publish","params":{"topic":"Wheel/Buttons/button_id","value":5}} → responds {"jsonrpc":"2.0","id":1,"result":{"ts_us":...}}.
  • cargo test → 6 unit + 3 integration tests pass, no broker needed.
  • cargo fmt --check and cargo clippy --all -- -D warnings → clean.

Checklist

  • No merge conflicts
  • All checks passing
  • Remove any non-applicable sections of this template
  • Assign the PR to yourself
  • Request reviewers & ping on Slack
  • PR is linked to the ticket (fill in the closes line below)

Closes #305

bracyw added 23 commits April 26, 2026 15:35
…ous modes

Replace `simulate` with a single `calypso-sim` binary organized as a
folder of focused modules under `src/bin/calypso-sim/`.

Modes (can run together):
- Autonomous heartbeat (`--auto`, default when no other mode is chosen)
- Interactive keymap (`--key-map`)
- Scripted keymap replay (`--script` with `--key-map`)
- JSON-RPC 2.0 over stdio (`--stream`, for an agent)

Per-topic ownership (`auto` / `stream` / `silenced`) lets stream and
keymap modes claim/release/silence topics; the autonomous loop checks
ownership before each publish.

Stream protocol: NDJSON over stdin/stdout, methods `publish`, `claim`,
`release`, `silence`, `status`, `list_topics`, `ping`. Tracing on stderr
so stdout stays clean for JSON-RPC.

Other:
- Startup warning lists CAN topics with no `sim_freq`
- README documents all four modes, keymap formats, and the stream protocol
- Keymap supports Random / Pinned / Increment / Sequence per key, with
  optional `desc` per entry
Adds calypso-sim/scripts/: vcu_mimic.py (drives the sim's stream mode to
emulate the VCU), the generated serverdata_pb2.py, and a README.
# Conflicts:
#	Cargo.toml
#	Odyssey-Definitions
#	src/bin/simulate.rs
Add `"unit": ""` to the 9 Wheel/Buttons/button_id entries so they publish
without relying on a `sim` block in Odyssey-Definitions. We dropped the
local-only OD sim-block commit and point the submodule at Develop's pointer,
where button_id is not auto-simulatable; explicit units keep these keymap
entries working. button_id is the only non-simulatable topic in this keymap
(home_mode/nero_index/not_in_reverse remain simulatable via Develop's OD).
- autonomous: validate --enable/--disable-topic regexes up front so a bad
  pattern exits non-zero instead of silently disabling the heartbeat
- script: fail-fast on malformed sleep / multi-char / unknown-key lines
  (deterministic replay shouldn't silently skip)
- stream: async stdout so a stalled consumer can't block a runtime worker
- build.rs: rerun-if-changed=Odyssey-Definitions so spec edits rebuild sim data
- simulatable_message: guard SimValue::initialize against empty range/options
  to avoid a startup panic on a degenerate spec
- drop set_topic_alias_max(600) to match the decoder and prior simulator
- fix S5 README description (REVERSE is gateless) and vcu_mimic cleanup branch
- remove Embedded-Base gitlink (not on Develop, no .gitmodules entry, unreferenced)
…st degenerate spec

- Remove `crossterm` from the root Cargo.toml (zero refs in src/ or libs/;
  only the separate calypso-sim crate uses it, via its own manifest) and drop
  its now-orphaned transitive crates from Cargo.lock — pure removal, no other
  version changes.
- Guard `SimValue::initialize` in the main crate against degenerate spec
  entries (empty min==max range, empty discrete options) so a malformed CAN
  spec can't panic at startup, bringing it to parity with the already-hardened
  calypso-sim copy.
- Remove the now-stale "Divergence from upstream" note from the vendored
  calypso-sim copy, since upstream now carries the same guards.
…compile placeholder regex once

Centralize the per-topic publish/ownership policy that was re-derived inline at
four call sites across three mode files: add TopicRegistry::auto_may_publish
(the heartbeat publishes only while a topic is still Auto-owned) and
driver_may_publish (stream/keymap drivers publish unless a topic is Silenced),
co-located with the Owner enum, and route autonomous/keymap/stream through them.
Behavior-preserving.

In the vendored simulatable_message.rs, hoist the `{}` placeholder Regex in
topic_values_inject into a compile-once LazyLock and drop the per-call
intermediate Vec - a documented sim-local divergence from the upstream copy,
which still recompiles per call.
These per-repo Claude Code command/skill files were force-added in fdab310
despite `.claude` already being in .gitignore (since 7ac8de7). Untrack them
with `git rm --cached` so the ignore rule takes effect — the files stay on
disk locally and remain in history at fdab310; this only drops them from the
branch tip going forward.
Replace scripts/vcu_mimic.py with native Rust tests that run under
`cargo test` with no broker required:

- tests/: spawn the real `calypso-sim --stream` binary and drive
  JSON-RPC over stdio — protocol conformance plus a VcuMock (mirroring
  Cerberus-2.0's u_statemachine.c) exercising boot/claim-all, the PIT
  brake+shutdown gate, and ownership isolation.
- registry.rs: unit tests for the auto/stream/silenced ownership model.
- publish.rs: extract encode_server_data so ServerData encoding is
  covered by a round-trip test, independent of any client/broker.
- Add serde_json dev-dependency; document the suite in README.md.

No broker is needed because publish only enqueues and ownership is
answered from the JSON-RPC responses; broker-backed end-to-end value
observation (drive-sweep/reverse/fault scenarios) is a planned
follow-up.
Cover the real, previously-untested logic:
- keymap: advance_increment wrap/saturate semantics, the serde
  untagged entry disambiguation (step beats value; sequence wins),
  and build_topic_states skip/keep rules
- cli: run_autonomous mode arbitration

Drop encode_handles_empty_unit_and_values (it exercised protobuf,
not our code; the field-mapping round-trip stays). Reframe the
README + scenarios e2e notes from a "planned follow-up" to
intentionally out of scope, since Siren in the compose stack covers
real bytes-on-the-wire.
Replace the sprawling suite with a lean one where each assertion is
independently verifiable:

- delete scenarios.rs (a VcuMock mirroring unverifiable Cerberus pedal /
  fault constants; s3 only asserted the mock's own gate) and fold the
  protocol + ownership-isolation integration tests into one dedicated
  tests/stream.rs (harness inlined)
- trim inline unit tests: registry 5->3, keymap 10->6, cli 3->1
  (publish unchanged)

Net 29 -> 18 tests. No JSON-RPC protocol coverage lost; the dropped
cases are low-risk edges (snapshot sort order, empty-sequence /
multi-char keymap guards, unbounded increment). README Testing section
updated to match.
… a minimal regression-guarding set (18 -> 7)
…ares it

DecodeData/EncodeData/FormatData now live in calypso-cangen; the calypso
crate re-exports them through its own data module (macros unchanged) and
calypso-sim imports DecodeData directly instead of vendoring a copy.
…docs)

--auto becomes --mock, Owner::Auto becomes Owner::Mock, and the
run_autonomous / modes::autonomous identifiers follow suit. The stream
ownership wire string changes from "auto" to "mock" accordingly (the
previous_owner / owner fields in claim/release/silence responses).
- Fmt CI step: drop `--all` so cargo-fmt stays scoped to the sim's detached
  workspace instead of walking path deps up into the main calypso workspace
  (whose generated `src/proto` never exists in the sim CI checkout). This was
  failing the calypso-sim CI deterministically at the Fmt step.
- Stream mode: a request missing `method` is now an Invalid Request (-32600)
  rather than a parse error (-32700), matching JSON-RPC 2.0.
- Docs: note that `publish` on a silenced topic returns `{skipped: "silenced"}`.
@bracyw bracyw self-assigned this Jul 8, 2026
bracyw added 6 commits July 8, 2026 14:52
… Clippy

The calypso-sim CI previously died at the Fmt step, so its Clippy step never
ran. With Fmt fixed, Clippy flags two `.map(f).unwrap_or(a)` chains on `Result`
(main.rs, publish.rs) under the CI toolchain; rewrite them as `.map_or(a, f)`.
…model

An action is a named list of steps (publish | sleep | invoke another action).
Interactive binds an action to a key; --play runs one by name (replacing
--script); actions compose by invoking each other, so scenarios are one
self-contained file. Removes the four keymap entry shapes, the parallel
KeyEntry/KeyMode enums + increment cursor, and the separate script parser.
…tup partition

Ownership is now resolved once at load instead of negotiated at runtime: the
driver's topics (a scenario's, or none for --stream) are removed from the mock
heartbeat's set up front and the split is printed, so the two publish disjoint
topics with no locking. Deletes the registry (Owner enum / RwLock / per-publish
predicates), the keymap lazy-claim, and the stream claim/release/silence/status
RPCs -- the stream protocol is now publish/list_topics/ping, and a stream driver
reserves topics with --disable-topic. Also adds a scenario example to the
keymap model docs.
Review + /simplify follow-ups on the static-ownership work:

- simulatable_message: guard inverted (min > max / inc_min > inc_max) sim
  ranges, not just degenerate (min == max), so a malformed spec can't panic
  random_range on a backwards range. update() already clamps via get_rand_offset.
- keymap: move the duplicate-key check into validate() so it runs at load for
  --play too (not just interactive mode), and add a regression test.
- stream: distinguish empty `values` from missing value/values in the JSON-RPC
  error message.
- keymap: derive scenario_topics by unioning each action's own publish steps
  instead of flattening invokes -- equivalent (every invoke target is a
  top-level action), and drops a throwaway Vec<Prim> alloc and the acyclic
  precondition.
- stream: cache the list_topics (name, unit) pairs in a LazyLock instead of
  rebuilding every component (with per-point RNG init) on each request.
…math into shared helpers

- Add `publish::resolve_values` as the single "exactly one of value/values,
  non-empty -> Vec<f32>" policy; scenario validation, replay flattening, and
  the stream RPC all call it (drops the bespoke `check_publish` and stream's
  duplicate match arm).
- Add `SimValue::{quantize, sample_range_or_low}`; initialize/update/
  get_rand_offset now share the snap-to-increment/round and the
  range-sample panic-guard instead of each repeating them.
- Drop redundant `#[serde(default)]` on Option fields (serde already maps a
  missing Option to None), a stale cross-file comment, dead commented-out
  code, C-port section banners, and an over-long doc example.
- interactive: `classify()` + an `Input` enum isolate the crossterm
  `KeyEvent` destructuring, so the `select!` loop is a flat 4-arm match
  (behavior unchanged: Ctrl+C/EOF quit, bound key runs, error reports+quits).
- mock: the `interval.tick()` arm calls `publish_due()`; the walk-and-publish
  logic moves into that helper.
- main: the mock-heartbeat partition setup (build filter, resolve `Partition`,
  print split, spawn) moves into `spawn_mock()`, slimming `main` to a sequence.
…helpers

- detect_cycle: add a shared "verified acyclic" set so an action reused by
  many others is walked once, not once per path that reaches it — avoids
  exponential blowup on diamond-shaped invoke graphs. Cycle detection and
  error messages are unchanged.
- resolve_values: take Option<&[f32]> instead of Option<Vec<f32>>, dropping
  the throwaway clone in validate/flatten; it now clones once internally only
  when returning the accepted array.
- stream: replace two #[allow(clippy::needless_pass_by_value)] with
  #[expect(..., reason = ...)] so the suppressions are self-checking and
  documented.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[SIM] - Unified calypso-sim binary with JSON-RPC streaming control plane

1 participant