Skip to content

feat!: model cycle overflows as persistent fixed points#394

Open
diegonehab wants to merge 35 commits into
mainfrom
feature/fixed-points
Open

feat!: model cycle overflows as persistent fixed points#394
diegonehab wants to merge 35 commits into
mainfrom
feature/fixed-points

Conversation

@diegonehab

@diegonehab diegonehab commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Problem

Machine and microarchitecture cycle limits were not handled consistently across execution, logging, verification, and
hash collection.

In particular, reaching a cycle limit needed to produce a fixed point without requiring an additional machine-state
transition. Calls targeting the current cycle also always reported that they had reached the target, even when the
current state already implied overflow, halt, or yield.

Solution

This PR models machine and microarchitecture cycle overflow as derived, state-preserving fixed points.

  • Machine overflow is derived from mcycle and imcyclemax.
  • Microarchitecture overflow is derived from uarch_cycle and UARCH_CYCLE_MAX.
  • A machine at either limit does not advance further.
  • Overflow takes precedence over halt, manual yield, and reaching the target cycle.
  • Calls targeting the current cycle return the break reason implied by the current state.
  • Hash collectors return the same break reason that the corresponding run operation would return.
  • imcyclemax is updated when an accepted advance-state input starts a new processing interval.
  • Rollup limits are centralized under ROLLUP_* constants.

The microarchitecture step now reads uarch_cycle before checking uarch_halt. After executing an instruction, it
checks overflow immediately after incrementing the cycle, before reporting a halt produced by that instruction. This
keeps the emulator, recorded access logs, replay verification, and Solidity step verifier aligned.

API changes

This PR adds explicit machine and microarchitecture overflow break reasons across the C++, C, Lua, and JSON-RPC APIs.

Notable renames include:

  • uarch_halt_flag to uarch_halt
  • configuration field halt_flag to halt
  • reached_target_cycle to reached_target_uarch_cycle
  • cycle_overflow to uarch_cycle_overflow
  • CM_CMIO_LOG2_MAX_OUTPUT_COUNT to CM_ROLLUP_LOG2_MAX_OUTPUT_COUNT

The obsolete halt-value constants used to distinguish ordinary halt from cycle overflow were removed.

Testing

The updated tests cover:

  • Machine and microarchitecture overflow fixed points
  • Overflow precedence over halt and manual yield
  • Calls targeting the current cycle
  • Logging and verification at cycle limits
  • Machine-cycle and microarchitecture-cycle hash collection
  • Local and remote machine APIs
  • C, Lua, and JSON-RPC break-reason mappings
  • Microarchitecture access-log ordering

@diegonehab diegonehab self-assigned this Jul 14, 2026
@diegonehab diegonehab added bug Something isn't working enhancement New feature or request labels Jul 14, 2026
@diegonehab diegonehab added this to the v0.21.0 milestone Jul 14, 2026
@diegonehab
diegonehab marked this pull request as draft July 14, 2026 21:48
@diegonehab
diegonehab force-pushed the feature/fixed-points branch from fbc10a8 to b349b33 Compare July 14, 2026 22:12
mpernambuco
mpernambuco previously approved these changes Jul 15, 2026

@mpernambuco mpernambuco left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-project-automation github-project-automation Bot moved this from Todo to Waiting Merge in Machine Unit Jul 15, 2026
@diegonehab
diegonehab force-pushed the feature/fixed-points branch 5 times, most recently from 2b549d9 to db50fcc Compare July 15, 2026 20:14
Persist machine and uarch overflow reasons in halt state, expose explicit
overflow break reasons, and align rollup limits, APIs, docs, and tests with
the new fixed-point semantics.
@diegonehab
diegonehab force-pushed the feature/fixed-points branch from db50fcc to 91cf818 Compare July 15, 2026 20:20
@diegonehab
diegonehab marked this pull request as ready for review July 15, 2026 21:30
@diegonehab
diegonehab requested a review from mpernambuco July 15, 2026 21:30

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR makes cycle overflow semantics consistent across execution, logging/verification, and hash collection by modeling both machine-cycle (mcycle/imcyclemax) and microarchitecture-cycle (uarch_cycle/UARCH_CYCLE_MAX) overflow as derived, state-preserving fixed points with defined precedence.

Changes:

  • Introduces imcyclemax and mcycle_overflow and wires them through interpreter/run paths and hash collectors so same-target calls report the current fixed-point reason without forcing a transition.
  • Renames and reshapes uarch halt/cycle concepts (uarch_halt_flaguarch_halt, reached_target_*, *_cycle_overflow) and updates C/C++/Lua/JSON-RPC APIs accordingly.
  • Centralizes rollup limits under ROLLUP_* constants and propagates them into the public C API (CM_ROLLUP_*) and Lua bindings.

Reviewed changes

Copilot reviewed 59 out of 59 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
uarch/compute-uarch-pristine-hash.cpp Updates pristine uarch state initialization to use the renamed halt register/init constant.
tests/misc/test-machine-c-api.cpp Adds/updates C API tests for rollup constants, new registers (IMCYCLEMAX), and renamed uarch break reasons/registers.
tests/lua/uarch-riscv-tests.lua Updates Lua tests to use uarch_halt and correct yield/response setup for logging.
tests/lua/spec-verify-uarch-failure.lua Renames uarch halt register usage in verification-failure tests.
tests/lua/spec-machine-api.lua Adds imcyclemax to machine API register coverage and renames uarch_halt_flag mapping.
tests/lua/spec-hash-tree-lua.lua Switches from CMIO constant to rollup output-count constant.
tests/lua/spec-collect-hashes.lua Aligns hash collection expectations with fixed-point/overflow semantics and updated uarch reset padding behavior.
tests/lua/spec-cm-cli.lua Updates CLI proof expectations to use rollup output-count constant.
tests/lua/mcycle-overflow.lua Adds extensive regression coverage for overflow fixed points and precedence across run/log/verify/uarch-step.
tests/lua/machine-bind.lua Updates Lua binding tests for new constants, imcyclemax, renamed uarch register/break reasons, and new semantics.
tests/lua/cartesi-machine-tests.lua Updates tests to use uarch_halt register semantics.
tests/fuzz/gen-seed-corpus.lua Adjusts assumed shadow-register block size to match the new register layout.
tests/fuzz/fuzz-interpret-step.cpp Updates fuzz target to read the renamed uarch halt register.
src/uarch-step.hpp Renames uarch step status enum value to UArchCycleOverflow and updates docs/wording.
src/uarch-step.cpp Implements overflow-precedence logic in uarch stepping and renames halt accessor usage.
src/uarch-state-access.hpp Renames uarch halt accessors and maps to the new halt field.
src/uarch-solidity-compat.hpp Renames halt wrappers and adds mcycle/imcyclemax wrappers for Solidity-converted code.
src/uarch-interpret.hpp Renames interpreter break reasons for uarch to *_uarch_cycle / uarch_cycle_overflow.
src/uarch-interpret.cpp Enforces overflow/halt/same-target precedence in uarch interpreter and updates break-reason mapping.
src/uarch-defines.h Centralizes uarch constants (halt/pc/cycle init; max-cycle derived from rollup constants).
src/uarch-constants.hpp Introduces uarch init constants and switches machine-state address mapping to shadow-register enums.
src/state-access.hpp Adds imcyclemax read/write support to state access layer.
src/shadow-uarch-state.hpp Renames uarch halt field and updates register enum/name mapping.
src/shadow-registers.hpp Adds imcyclemax to shadow registers and updates size/static asserts and name mapping.
src/send-cmio-response.hpp Replaces local CMIO constant usage with rollup constants include.
src/send-cmio-response.cpp Updates imcyclemax when sending an advance-state response (with saturating limit).
src/rollup-defines.h Adds centralized rollup limit macros.
src/rollup-constants.hpp Adds centralized rollup limit constants for C++ consumers.
src/riscv-constants.hpp Adds IMCYCLEMAX_INIT and removes uarch init constants now housed in uarch headers.
src/replay-send-cmio-state-access.hpp Extends replay state-access to read mcycle and write imcyclemax for send-cmio replay.
src/record-send-cmio-state-access.hpp Extends record state-access to read mcycle and write imcyclemax for send-cmio logging.
src/processor-state.hpp Adjusts padding to preserve page alignment after shadow register expansion.
src/Makefile Bumps emulator minor version.
src/machine.hpp Documents overflow precedence and adds get_state_break_reason helper for consistent behavior.
src/machine.cpp Implements imcyclemax register plumbing, new break-reason derivation, and updated hash-collection fixed-point semantics.
src/machine-reg.hpp Adds imcyclemax and renames uarch halt register enum.
src/jsonrpc-machine.cpp Updates default uarch break reason to the renamed reached_target_uarch_cycle.
src/jsonrpc-discover.json Extends JSON-RPC schema with imcyclemax, mcycle_overflow, and renamed uarch fields/break reasons.
src/json-util.cpp Adds imcyclemax reg name mapping and updates (de)serialization for renamed uarch fields/break reasons.
src/interpret.hpp Adds mcycle_overflow interpreter break reason.
src/interpret.cpp Enforces overflow fixed-point precedence and clamps effective target cycle to imcyclemax.
src/i-uarch-state-access.hpp Renames uarch halt accessor macros.
src/i-state-access.hpp Adds imcyclemax accessor macros.
src/i-machine.hpp Updates collect_uarch_cycle_root_hashes API documentation for revised tail/fixed-point rules.
src/htif-address-range.cpp Adds include needed after new register/init usage.
src/collect-uarch-cycle-hashes-state-access.hpp Updates static-assert assumptions for renamed first uarch shadow register.
src/collect-mcycle-hashes-state-access.hpp Adds imcyclemax read/write support for hash collection state access.
src/cm.h Updates public C API with rollup constants, CM_UARCH_CYCLE_MAX semantics, new break reasons, and new registers.
src/cm.cpp Adds static asserts for rollup constants and maps new/renamed registers.
src/clua-i-machine.cpp Adds Lua-C register mapping for imcyclemax and exposes mcycle_overflow break reason.
src/clua-cartesi.cpp Exposes rollup constants and new overflow/uarch-break constants to Lua.
src/cartesi-machine.lua Updates CLI script logic to use rollup constants and renamed uarch halt/break reason symbols.
doc/recipes/run-rolling-calculator-output-proofs.lua Updates recipe to use rollup output-count constant.
doc/recipes/rootfs-docs.ext2 Updates LFS pointer for docs rootfs image.
doc/recipes/rolling-verification-game.lua Updates recipe to use rollup output-count constant.
doc/recipes/find-uarch-putchar-ecall.lua Updates recipe to use renamed uarch halt register.
doc/README.md.template Updates template docs for new constants/registers and revised semantics.
doc/README.md Updates generated docs/examples for new constants/registers, renamed uarch fields, and revised semantics/hashes.
CHANGELOG.md Documents new overflow break reasons, rollup constants rename, and hash-collection behavior fix.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread tests/fuzz/gen-seed-corpus.lua
edubart
edubart previously approved these changes Jul 16, 2026
Update machine-guest-tools to v0.18.0-test6 and rebuild the documentation
rootfs to make the rollup --utf8-payload option available.

Use schema-based hex, base64, and UTF-8 conversion in
cartesi-rollup-data.lua, and simplify the calculator and puppet examples
by representing textual payloads directly as JSON strings.

Replace the legacy Lua hex helpers with cartesi.tohex, update hash
consumers for its 0x-prefixed output, and regenerate the README.
@diegonehab
diegonehab force-pushed the feature/fixed-points branch from fe7dfda to 06b524d Compare July 19, 2026 18:11
Replace the per-byte codec helpers and Base64 string streams with bulk
span-based encoding and decoding.

Write allocating results directly into string storage with
resize_and_overwrite, avoiding intermediate streams and redundant output
initialization. Use lookup tables for hex and Base64 decoding.

Preserve whitespace-tolerant Base64 input while enforcing canonical
quartets, padding positions, and padding bits. Reject incomplete input
and data following padding.

Expose bulk codec functions for buffer and streaming consumers. Add Lua
coverage for RFC Base64 vectors, malformed input, whitespace, and round
trips over every byte value.

Use cartesi.tohex in the executable documentation recipes instead of
downloading and compiling the machine-guest-tools hex utility. Remove
the obsolete documentation image builder stage and build argument.

Update the changelog, bump machine-guest-tools to v0.18.0-test7, and
regenerate the documentation rootfs and rendered README.
Comment thread src/cartesi-machine.lua
Return a repeatable fixed-point suffix from mcycle and uarch-cycle
collection, including same-target calls. Preserve execution hashes when
uarch halt and reset share the final bundle.

Rename back_tree to partial_bundle and reset_indices to
mcycle_hash_offsets throughout the public APIs. Simplify the computation
hash builders to consume these results and remove the unreleased
mcycle_computation_hash_leaves option.

Use MCYCLE_MAX and saturating chunk boundaries so CLI and GDB collection
can run through the full unsigned mcycle range.
Match the naming already used by rollups-contracts and dave
(outputsMerkleRoot) instead of the emulator's own terms. Covers the
cartesi-machine.lua sub-options (outputs_merkle_root,
outputs_merkle_root_proof, check_outputs_merkle_root), stored file
name patterns (input-%i-outputs-merkle-root.bin), the doc template
and regenerated README, doc recipes, a cm.h doc comment, the Lua
tests, and the CHANGELOG entry.
@diegonehab
diegonehab force-pushed the feature/fixed-points branch from ab83ebb to 7dde8cf Compare July 22, 2026 13:02
Add manifest-driven mcycle and uarch computation hash cases covering
fixed points, rejects, outputs, terminal conditions, saturated counters,
and boundary uarch programs.

Record validated hashes once on normal amd64, archive the complete corpus,
and verify immutable results across arm64, coverage, and sanitizer builds.
Publish the corpus as a CI artifact and attach it to tagged releases.

Cross-check manifest, binary, and printed hashes with representative
independent tree reconstructions and equivalence checks. Keep uarch
overflow as the sole no-hash failure and move max-mcycle truncation
coverage into scoped CLI tests.

Finalize computation hashes before reporting unexpected manual yields,
support tall frontiers and saturated windows, and strengthen integer and
computation-hash option validation.
Move scope_exit into cartesi.tests.util and remove the generic
cartesi.utils test module.

Standardize the local module name as tests_util throughout the Lua test
suite, keeping test-only helpers clearly separated from cartesi.util.
Make frontiers require and retain their selected hash function, allowing
the same implementation to support Keccak-256 and SHA-256.

Replace duplicated test Merkle-tree helpers with cartesi.hash-tree,
and allow bundled SHA-256 mcycle computation hashes.
@diegonehab
diegonehab force-pushed the feature/fixed-points branch from 7dde8cf to d2e6561 Compare July 22, 2026 13:28
Add machine::sync_stored(dir), mirroring the existing clone_stored and
remove_stored family, to fsync a machine's stored files and directory
without locking, so a machine mapped by a running process can still be
synced. Factor the shared file enumeration of remove_stored and
sync_stored into a for_each_stored_filename helper.

Add os::sync_file and os::sync_directory. On macOS, sync_file tries
F_FULLFSYNC before falling back to fsync. Warn at compile time on
platforms without a unified page cache, where fsync from a fresh file
descriptor is not guaranteed to capture pages dirtied through another
description's shared mmap.

Thread sync_stored through i-machine, local-machine, and jsonrpc-machine,
add the machine.sync_stored jsonrpc method and discover schema, expose
cm_sync_stored in the C API, and add the machine:sync_stored Lua method.

Cover sync after close, sync while in use, invalid arguments, and
remove_stored with an extraneous file in the directory.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 132 out of 136 changed files in this pull request and generated 1 comment.

Comment thread src/Makefile Outdated
Fix INTERPTER_CXXFLAGS typo so -fomit-frame-pointer is actually
applied when building interpret.cpp.

Cast cmio request length to size_t before constructing the span in
cm_receive_cmio_request, avoiding a sign-conversion warning.

Remove the unused raise_misaligned_fetch_exception helper, and mark
assert_no_brk and dumpInsn maybe_unused since their only callers are
compiled out in release builds.
Comment thread src/cartesi-machine.lua
-- that execution never reached.
claim:end_epoch()
error("unexpected manual yield reason")
end

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what will the backup_closer do after this for ends with active backup_machine, revert? should we commit()?

@diegonehab diegonehab Jul 23, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice catch. In these weird error conditions, the computation hash is perfectly defined: they are sticky fixed points that will never change (send_cmio_response is a noop on them). This is true of an unexpected manual yield, an mcycle overflow, halt, or exception. We should commit. Will do.

The main chunk had reached the Lua limit of 200 local variables, which
left no room for new file-scope functions. The values assembled by the
option handlers now live in a single cmdline table. The former
cmdline_config and cmdline_opts_finished become cmdline.config and
cmdline.opts_finished.
A halt, an mcycle overflow, an exception, and an unexpected manual yield
are sticky fixed points, so the epoch runner now commits the interrupted
input's snapshot instead of leaving it for the exit-time revert. An
unexpected manual yield also prints a banner and sets the exit code like
an exception does, instead of raising an error.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working enhancement New feature or request

Projects

Status: Waiting Merge

Development

Successfully merging this pull request may close these issues.

4 participants