feat!: model cycle overflows as persistent fixed points#394
feat!: model cycle overflows as persistent fixed points#394diegonehab wants to merge 35 commits into
Conversation
fbc10a8 to
b349b33
Compare
2b549d9 to
db50fcc
Compare
Persist machine and uarch overflow reasons in halt state, expose explicit overflow break reasons, and align rollup limits, APIs, docs, and tests with the new fixed-point semantics.
db50fcc to
91cf818
Compare
a3713d8 to
bc43481
Compare
There was a problem hiding this comment.
Pull request overview
This PR makes cycle overflow semantics consistent across execution, logging/verification, and hash collection by modeling both machine-cycle (mcycle/imcyclemax) and microarchitecture-cycle (uarch_cycle/UARCH_CYCLE_MAX) overflow as derived, state-preserving fixed points with defined precedence.
Changes:
- Introduces
imcyclemaxandmcycle_overflowand wires them through interpreter/run paths and hash collectors so same-target calls report the current fixed-point reason without forcing a transition. - Renames and reshapes uarch halt/cycle concepts (
uarch_halt_flag→uarch_halt,reached_target_*,*_cycle_overflow) and updates C/C++/Lua/JSON-RPC APIs accordingly. - Centralizes rollup limits under
ROLLUP_*constants and propagates them into the public C API (CM_ROLLUP_*) and Lua bindings.
Reviewed changes
Copilot reviewed 59 out of 59 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| uarch/compute-uarch-pristine-hash.cpp | Updates pristine uarch state initialization to use the renamed halt register/init constant. |
| tests/misc/test-machine-c-api.cpp | Adds/updates C API tests for rollup constants, new registers (IMCYCLEMAX), and renamed uarch break reasons/registers. |
| tests/lua/uarch-riscv-tests.lua | Updates Lua tests to use uarch_halt and correct yield/response setup for logging. |
| tests/lua/spec-verify-uarch-failure.lua | Renames uarch halt register usage in verification-failure tests. |
| tests/lua/spec-machine-api.lua | Adds imcyclemax to machine API register coverage and renames uarch_halt_flag mapping. |
| tests/lua/spec-hash-tree-lua.lua | Switches from CMIO constant to rollup output-count constant. |
| tests/lua/spec-collect-hashes.lua | Aligns hash collection expectations with fixed-point/overflow semantics and updated uarch reset padding behavior. |
| tests/lua/spec-cm-cli.lua | Updates CLI proof expectations to use rollup output-count constant. |
| tests/lua/mcycle-overflow.lua | Adds extensive regression coverage for overflow fixed points and precedence across run/log/verify/uarch-step. |
| tests/lua/machine-bind.lua | Updates Lua binding tests for new constants, imcyclemax, renamed uarch register/break reasons, and new semantics. |
| tests/lua/cartesi-machine-tests.lua | Updates tests to use uarch_halt register semantics. |
| tests/fuzz/gen-seed-corpus.lua | Adjusts assumed shadow-register block size to match the new register layout. |
| tests/fuzz/fuzz-interpret-step.cpp | Updates fuzz target to read the renamed uarch halt register. |
| src/uarch-step.hpp | Renames uarch step status enum value to UArchCycleOverflow and updates docs/wording. |
| src/uarch-step.cpp | Implements overflow-precedence logic in uarch stepping and renames halt accessor usage. |
| src/uarch-state-access.hpp | Renames uarch halt accessors and maps to the new halt field. |
| src/uarch-solidity-compat.hpp | Renames halt wrappers and adds mcycle/imcyclemax wrappers for Solidity-converted code. |
| src/uarch-interpret.hpp | Renames interpreter break reasons for uarch to *_uarch_cycle / uarch_cycle_overflow. |
| src/uarch-interpret.cpp | Enforces overflow/halt/same-target precedence in uarch interpreter and updates break-reason mapping. |
| src/uarch-defines.h | Centralizes uarch constants (halt/pc/cycle init; max-cycle derived from rollup constants). |
| src/uarch-constants.hpp | Introduces uarch init constants and switches machine-state address mapping to shadow-register enums. |
| src/state-access.hpp | Adds imcyclemax read/write support to state access layer. |
| src/shadow-uarch-state.hpp | Renames uarch halt field and updates register enum/name mapping. |
| src/shadow-registers.hpp | Adds imcyclemax to shadow registers and updates size/static asserts and name mapping. |
| src/send-cmio-response.hpp | Replaces local CMIO constant usage with rollup constants include. |
| src/send-cmio-response.cpp | Updates imcyclemax when sending an advance-state response (with saturating limit). |
| src/rollup-defines.h | Adds centralized rollup limit macros. |
| src/rollup-constants.hpp | Adds centralized rollup limit constants for C++ consumers. |
| src/riscv-constants.hpp | Adds IMCYCLEMAX_INIT and removes uarch init constants now housed in uarch headers. |
| src/replay-send-cmio-state-access.hpp | Extends replay state-access to read mcycle and write imcyclemax for send-cmio replay. |
| src/record-send-cmio-state-access.hpp | Extends record state-access to read mcycle and write imcyclemax for send-cmio logging. |
| src/processor-state.hpp | Adjusts padding to preserve page alignment after shadow register expansion. |
| src/Makefile | Bumps emulator minor version. |
| src/machine.hpp | Documents overflow precedence and adds get_state_break_reason helper for consistent behavior. |
| src/machine.cpp | Implements imcyclemax register plumbing, new break-reason derivation, and updated hash-collection fixed-point semantics. |
| src/machine-reg.hpp | Adds imcyclemax and renames uarch halt register enum. |
| src/jsonrpc-machine.cpp | Updates default uarch break reason to the renamed reached_target_uarch_cycle. |
| src/jsonrpc-discover.json | Extends JSON-RPC schema with imcyclemax, mcycle_overflow, and renamed uarch fields/break reasons. |
| src/json-util.cpp | Adds imcyclemax reg name mapping and updates (de)serialization for renamed uarch fields/break reasons. |
| src/interpret.hpp | Adds mcycle_overflow interpreter break reason. |
| src/interpret.cpp | Enforces overflow fixed-point precedence and clamps effective target cycle to imcyclemax. |
| src/i-uarch-state-access.hpp | Renames uarch halt accessor macros. |
| src/i-state-access.hpp | Adds imcyclemax accessor macros. |
| src/i-machine.hpp | Updates collect_uarch_cycle_root_hashes API documentation for revised tail/fixed-point rules. |
| src/htif-address-range.cpp | Adds include needed after new register/init usage. |
| src/collect-uarch-cycle-hashes-state-access.hpp | Updates static-assert assumptions for renamed first uarch shadow register. |
| src/collect-mcycle-hashes-state-access.hpp | Adds imcyclemax read/write support for hash collection state access. |
| src/cm.h | Updates public C API with rollup constants, CM_UARCH_CYCLE_MAX semantics, new break reasons, and new registers. |
| src/cm.cpp | Adds static asserts for rollup constants and maps new/renamed registers. |
| src/clua-i-machine.cpp | Adds Lua-C register mapping for imcyclemax and exposes mcycle_overflow break reason. |
| src/clua-cartesi.cpp | Exposes rollup constants and new overflow/uarch-break constants to Lua. |
| src/cartesi-machine.lua | Updates CLI script logic to use rollup constants and renamed uarch halt/break reason symbols. |
| doc/recipes/run-rolling-calculator-output-proofs.lua | Updates recipe to use rollup output-count constant. |
| doc/recipes/rootfs-docs.ext2 | Updates LFS pointer for docs rootfs image. |
| doc/recipes/rolling-verification-game.lua | Updates recipe to use rollup output-count constant. |
| doc/recipes/find-uarch-putchar-ecall.lua | Updates recipe to use renamed uarch halt register. |
| doc/README.md.template | Updates template docs for new constants/registers and revised semantics. |
| doc/README.md | Updates generated docs/examples for new constants/registers, renamed uarch fields, and revised semantics/hashes. |
| CHANGELOG.md | Documents new overflow break reasons, rollup constants rename, and hash-collection behavior fix. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Update machine-guest-tools to v0.18.0-test6 and rebuild the documentation rootfs to make the rollup --utf8-payload option available. Use schema-based hex, base64, and UTF-8 conversion in cartesi-rollup-data.lua, and simplify the calculator and puppet examples by representing textual payloads directly as JSON strings. Replace the legacy Lua hex helpers with cartesi.tohex, update hash consumers for its 0x-prefixed output, and regenerate the README.
fe7dfda to
06b524d
Compare
Replace the per-byte codec helpers and Base64 string streams with bulk span-based encoding and decoding. Write allocating results directly into string storage with resize_and_overwrite, avoiding intermediate streams and redundant output initialization. Use lookup tables for hex and Base64 decoding. Preserve whitespace-tolerant Base64 input while enforcing canonical quartets, padding positions, and padding bits. Reject incomplete input and data following padding. Expose bulk codec functions for buffer and streaming consumers. Add Lua coverage for RFC Base64 vectors, malformed input, whitespace, and round trips over every byte value. Use cartesi.tohex in the executable documentation recipes instead of downloading and compiling the machine-guest-tools hex utility. Remove the obsolete documentation image builder stage and build argument. Update the changelog, bump machine-guest-tools to v0.18.0-test7, and regenerate the documentation rootfs and rendered README.
Return a repeatable fixed-point suffix from mcycle and uarch-cycle collection, including same-target calls. Preserve execution hashes when uarch halt and reset share the final bundle. Rename back_tree to partial_bundle and reset_indices to mcycle_hash_offsets throughout the public APIs. Simplify the computation hash builders to consume these results and remove the unreleased mcycle_computation_hash_leaves option. Use MCYCLE_MAX and saturating chunk boundaries so CLI and GDB collection can run through the full unsigned mcycle range.
Match the naming already used by rollups-contracts and dave (outputsMerkleRoot) instead of the emulator's own terms. Covers the cartesi-machine.lua sub-options (outputs_merkle_root, outputs_merkle_root_proof, check_outputs_merkle_root), stored file name patterns (input-%i-outputs-merkle-root.bin), the doc template and regenerated README, doc recipes, a cm.h doc comment, the Lua tests, and the CHANGELOG entry.
ab83ebb to
7dde8cf
Compare
Add manifest-driven mcycle and uarch computation hash cases covering fixed points, rejects, outputs, terminal conditions, saturated counters, and boundary uarch programs. Record validated hashes once on normal amd64, archive the complete corpus, and verify immutable results across arm64, coverage, and sanitizer builds. Publish the corpus as a CI artifact and attach it to tagged releases. Cross-check manifest, binary, and printed hashes with representative independent tree reconstructions and equivalence checks. Keep uarch overflow as the sole no-hash failure and move max-mcycle truncation coverage into scoped CLI tests. Finalize computation hashes before reporting unexpected manual yields, support tall frontiers and saturated windows, and strengthen integer and computation-hash option validation.
Move scope_exit into cartesi.tests.util and remove the generic cartesi.utils test module. Standardize the local module name as tests_util throughout the Lua test suite, keeping test-only helpers clearly separated from cartesi.util.
Make frontiers require and retain their selected hash function, allowing the same implementation to support Keccak-256 and SHA-256. Replace duplicated test Merkle-tree helpers with cartesi.hash-tree, and allow bundled SHA-256 mcycle computation hashes.
7dde8cf to
d2e6561
Compare
Add machine::sync_stored(dir), mirroring the existing clone_stored and remove_stored family, to fsync a machine's stored files and directory without locking, so a machine mapped by a running process can still be synced. Factor the shared file enumeration of remove_stored and sync_stored into a for_each_stored_filename helper. Add os::sync_file and os::sync_directory. On macOS, sync_file tries F_FULLFSYNC before falling back to fsync. Warn at compile time on platforms without a unified page cache, where fsync from a fresh file descriptor is not guaranteed to capture pages dirtied through another description's shared mmap. Thread sync_stored through i-machine, local-machine, and jsonrpc-machine, add the machine.sync_stored jsonrpc method and discover schema, expose cm_sync_stored in the C API, and add the machine:sync_stored Lua method. Cover sync after close, sync while in use, invalid arguments, and remove_stored with an extraneous file in the directory.
Fix INTERPTER_CXXFLAGS typo so -fomit-frame-pointer is actually applied when building interpret.cpp. Cast cmio request length to size_t before constructing the span in cm_receive_cmio_request, avoiding a sign-conversion warning. Remove the unused raise_misaligned_fetch_exception helper, and mark assert_no_brk and dumpInsn maybe_unused since their only callers are compiled out in release builds.
| -- that execution never reached. | ||
| claim:end_epoch() | ||
| error("unexpected manual yield reason") | ||
| end |
There was a problem hiding this comment.
what will the backup_closer do after this for ends with active backup_machine, revert? should we commit()?
There was a problem hiding this comment.
Very nice catch. In these weird error conditions, the computation hash is perfectly defined: they are sticky fixed points that will never change (send_cmio_response is a noop on them). This is true of an unexpected manual yield, an mcycle overflow, halt, or exception. We should commit. Will do.
The main chunk had reached the Lua limit of 200 local variables, which left no room for new file-scope functions. The values assembled by the option handlers now live in a single cmdline table. The former cmdline_config and cmdline_opts_finished become cmdline.config and cmdline.opts_finished.
A halt, an mcycle overflow, an exception, and an unexpected manual yield are sticky fixed points, so the epoch runner now commits the interrupted input's snapshot instead of leaving it for the exit-time revert. An unexpected manual yield also prints a banner and sets the exit code like an exception does, instead of raising an error.
Problem
Machine and microarchitecture cycle limits were not handled consistently across execution, logging, verification, and
hash collection.
In particular, reaching a cycle limit needed to produce a fixed point without requiring an additional machine-state
transition. Calls targeting the current cycle also always reported that they had reached the target, even when the
current state already implied overflow, halt, or yield.
Solution
This PR models machine and microarchitecture cycle overflow as derived, state-preserving fixed points.
mcycleandimcyclemax.uarch_cycleandUARCH_CYCLE_MAX.imcyclemaxis updated when an accepted advance-state input starts a new processing interval.ROLLUP_*constants.The microarchitecture step now reads
uarch_cyclebefore checkinguarch_halt. After executing an instruction, itchecks overflow immediately after incrementing the cycle, before reporting a halt produced by that instruction. This
keeps the emulator, recorded access logs, replay verification, and Solidity step verifier aligned.
API changes
This PR adds explicit machine and microarchitecture overflow break reasons across the C++, C, Lua, and JSON-RPC APIs.
Notable renames include:
uarch_halt_flagtouarch_halthalt_flagtohaltreached_target_cycletoreached_target_uarch_cyclecycle_overflowtouarch_cycle_overflowCM_CMIO_LOG2_MAX_OUTPUT_COUNTtoCM_ROLLUP_LOG2_MAX_OUTPUT_COUNTThe obsolete halt-value constants used to distinguish ordinary halt from cycle overflow were removed.
Testing
The updated tests cover: