bump - #47
Open
yapishu wants to merge 127 commits into
Open
Conversation
…w decoding error type
… to reduce fees for power-users
…for examining raw-txs, add batch writes to file driver, add file exts to http driver
* Sync from Nockup development repo. * Change to workspace. * Post with openssl
… some edge-case v0 notes
* Update README to remove status and development notes Removed pre-release status and development notes from README. * Fix URL branch reference in installation command * Add permissions for contents in release workflow
* Post new deploy chain. * Add pruning. * Add protobuf support * Add protobuf support * Add protobuf support * Tweak CI * Tweak CI * Tweak CI * Tweak CI * Fix nockup for new manifest format. * Adjust * Fix so build cleanup doesn't happen on PR.
* Post new deploy chain. * Add pruning. * Add protobuf support * Add protobuf support * Add protobuf support * Tweak CI * Tweak CI * Tweak CI * Tweak CI * Fix nockup for new manifest format. * Adjust * Fix so build cleanup doesn't happen on PR. * Fix install script and update old links to sigilante repo. * Fix last URLs
* Post new deploy chain. * Add pruning. * Add protobuf support * Add protobuf support * Add protobuf support * Tweak CI * Tweak CI * Tweak CI * Tweak CI * Fix nockup for new manifest format. * Adjust * Fix so build cleanup doesn't happen on PR. * Fix install script and update old links to sigilante repo. * Fix last URLs * Fix cleanup logic.
Update the `block-size` unit test to use `%+ expect-eq` with proper Hoon rune structure, preserving the same assertion while correcting the invocation form.
…ault max block size Raise the default `max_block_size` from 8 MB to 16 MB in `BlockchainConstants` and update the corresponding test expectation to match the new default.
Update `heard-tx` to treat already-seen transactions differently based on wire source: re-broadcast on local `%grpc` resubmissions, but suppress re-gossip for peer `%libp2p` repeats to prevent gossip loops. Added `local-tx-submission` detection in dumbnet, plus test helpers for custom poke wires and new integration tests covering both grpc resend (re-gossips) and peer resend (does not re-gossip) behavior.
* Return orphaned-block transactions to the mempool, and repair stranded ones on load
A transaction carried by a block that then lost a chain race was stranded
permanently. This is the root cause of transactions sitting "pending" in a
node's mempool forever while being unmineable network-wide.
The bug
-------
+accept-block claims every tx in a block: blocks-needed-by += block, and the
tx leaves excluded-txs (the mempool). +reject-pending-block releases those
claims for a PENDING block -- but nothing ever released them for an ACCEPTED
one. So once an accepted block was orphaned, its txs were:
- never mined the miner's candidate set is excluded-txs plus
pending-block txs, and the tx is now in neither
- never re-gossiped the per-block re-gossip walks excluded-txs
- never dropped +drop-tx refuses anything still in blocks-needed-by
- inputs pinned spent-by is only cleared by +drop-tx, so +inputs-spent
rejected any REPLACEMENT tx spending the same notes --
the funds could not be re-spent by any means
Nothing was structurally broken -- the tx is still in exactly one of
blocks-needed-by / excluded-txs, so +apt is happy -- which is why this went
unnoticed. The claim was simply never released. The kernel even emits a
`reorg-orphan` span: it knew the block was orphaned.
The fix, in two halves
----------------------
1. +release-orphaned-branch, on every reorg. Walks ONLY the branch just
abandoned, back to the common ancestor, handing each block's txs to the
mempool. O(reorg depth): no steady-state event ever walks the chain.
- a tx also carried by the WINNING block keeps that claim and stays out
of the mempool: it really is mined
- heard-at is refreshed on release, or the retention sweep later in the
same +garbage-collect would evict the tx on sight (it was heard before
the block that mined it) and the release would be a no-op
- a tx whose inputs the winning chain spent via another tx is dropped by
the spent-input sweep in that same pass, not parked unspendable
2. +repair-orphaned-claims, once from +load. The live path only walks branches
that a LIVE reorg abandons, so it cannot reach txs stranded by reorgs that
happened under the old kernel -- and every upgrading node is carrying
exactly those. This one-time repair frees them, which is what makes the fix
retroactive rather than merely prospective. It walks all of .blocks, and
that is why it lives in +load and nowhere else: a boot already pays to load
the whole state, whereas an event must never pay for the size of the chain.
Derived-state update is hoisted above garbage collection so heaviest-chain
names the WINNING chain when the release classifies blocks. No consensus-state
shape change.
Not included: deleting orphaned blocks to reclaim .blocks / .balance / .txs.
Deferring deletion until no reorg can restore a block means remembering which
blocks were orphaned and when, and .blocks has no such index -- finding them
later would mean walking every block inside an event. That needs a new index
in consensus state; until then the memory is left alone rather than paid for
with a chain-sized walk.
Tests
-----
Seven new integration tests, plus one corrected. Each asserts the kernel's own
+apt oracle (exposed as +consensus-invariants) is ~, which reports
%txs-fell-through-cracks on exactly the stranded state:
reorg-orphaned-tx-returns-to-mempool the core bug
reorg-tx-in-winning-block-stays-mined a naive release would re-offer a mined tx
reorg-orphaned-tx-is-regossiped returned AND actually re-announced
reorg-orphaned-branch-releases-every-block walk reaches the common ancestor
reorg-orphaned-tx-double-spent-is-dropped winning chain spent the inputs
reorg-orphaned-tx-survives-retention-sweep guards the heard-at refresh
boot-repair-frees-stranded-orphan-tx the retroactive repair
test-pending-txs-reorg-2 previously required EVERY tx to remain in
blocks-needed-by after a fork, including one the winning chain never carried.
That expectation WAS the bug; it now asserts the tx returns to the mempool,
while the tx the winning chain does carry correctly stays claimed.
roswell test-dumb: 268 passed, 0 regressions (261 on master + these 7). The one
remaining crash, test-v1-mempool-reject-oversize-tx, fails identically on clean
master and is untouched here.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Drive the real +load in the boot-repair test
The test called +repair-orphaned-claims directly, which only proved the repair
logic works -- not that +load runs it. That wiring is exactly what can fail
silently: if heaviest-chain.d were not populated at load time, the repair would
find no orphans, no-op, and leave every stranded tx stranded while the logs
looked healthy.
+boot-with now drives the kernel's real +load over the stranded state, as the
runtime does when swapping a kernel in over existing state. It logs
'repair-orphaned-claims: releasing txs of orphaned blocks: 1' and frees the tx,
confirming +load invokes the repair and heaviest-chain is populated there.
(check-checkpoints is a no-op on a test chain: is-mainnet is %.n, so the state
is not reset out from under the repair.)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Delete orphaned blocks on load, and log the stages of +load
Two changes that ride together: +load now says what it is doing, and the
boot repair now reclaims the orphaned blocks it was already finding.
Staged logging
--------------
+load ran four ~> %bout hints back to back. %bout prints a bare
"took: <duration>" with no label, so a boot produced a column of anonymous
durations that could not be attributed to a stage. The stages are now split
apart (they were composed in one expression) and each is announced first, so
a slow boot names the stage responsible. Boot-only: no event pays for it.
Measured on a mainnet node at height ~105.7k, untraced: whole +load ~10s --
check-and-repair 4.6s, repair-orphaned-claims 5.3s, state-n-to-9 ~2us. It
found 176 orphaned blocks and freed 9 stranded txs into the mempool.
Deleting orphaned blocks
------------------------
+repair-orphaned-claims already walks every block and already knows which are
orphaned; deleting them there costs nothing extra and needs no new index. The
previous note said deletion needed to remember which blocks were orphaned and
WHEN, and that .blocks has no such index -- but a block's height is in its
local-page, and +load already pays for the walk, so the index was never the
obstacle at boot. It is only an obstacle for an event, which this is not.
Deletes all six maps keyed by the block-id: .blocks, .balance, .txs,
.min-timestamps, .epoch-start, .targets. All or nothing. Every cross-map
inconsistency crashes loudly except .balance: +validate-page-with-txs reads
balance[parent] with `get`, not `got`, so a block kept in .blocks with its
balance dropped does not crash -- it validates that block's children against
an EMPTY utxo set and silently rejects every one.
Classification is by ANCESTRY (+canonical-block-ids), not by a heaviest-chain
height lookup, and that is the load-bearing decision here.
heaviest-chain.d is derived and revised lazily: +update walks down from each
new tip, stops at the first height that already agrees, and never prunes
entries ABOVE the tip. Heaviness is accumulated-work, not height
(+compare-heaviness), so a reorg onto a chain with more work but less height
lowers the tip and strands entries above it naming blocks that are now
orphans. derived.hoon's own TODO asks for the missing check.
That is not hypothetical. On the mainnet node: 105.916 blocks, tip 105.731,
so 184 blocks are not on the canonical chain -- but the height lookup called
only 176 of them orphans. The other 8 sit above the tip, retained on a stale
or absent entry, and their parents are among the 176. Deleting a parent while
retaining its child crashes the kernel on the next child of it, via the
11-deep `got` walk in +update-min-timestamps.
Ancestry gives closure: the retained set is built by walking parents, so every
retained block's parent is retained, and every reference the kernel chases
with `got` -- parent, .targets, .epoch-start, the 11-deep walk -- lands inside
it. A depth cutoff would not: a retained orphan's .epoch-start reaches up to
blocks-per-epoch back. The release now uses the same set, because a block
deleted without its claims released strands those txs forever (+apt:
%txs-fell-through-cracks); it also means the 8 finally get released. If the
tip does not chain back to genesis the repair does nothing at all rather than
act on a partial answer.
Deleting is not lossy: a child arriving fresh fails +heard-block's parent
check and takes the missing-parent path, which re-requests the ancestors.
Tests
-----
+con-referential-integrity is a new oracle: +apt checks only the raw-txs
partition, and nothing in the kernel checks that the block-keyed maps agree
with each other. It is asserted on the state BEFORE deletion as well as after
-- an oracle that only passes afterwards proves nothing. It caught one wrong
assumption already: genesis has no coinbase, so it legitimately has no
.balance entry, exactly as +get-cur-balance encodes.
roswell test-dumb: 272 passed, 0 regressions (268 on master + these 4). The
one remaining crash, test-v1-mempool-reject-oversize-tx, fails identically on
clean master and is untouched here.
Not included: +update still leaves stale heaviest-chain entries above the tip.
That is a live bug in its own right -- +release-orphaned-branch classifies
with that index, so a tip-lowering reorg finds u.canonical == cur at the old
tip's own height, concludes it is already at the common ancestor, and releases
nothing. Nothing here depends on that index, and the boot repair catches what
the live path misses, so it self-heals at the next boot. Fixed separately.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Prune heaviest-chain entries above the tip, fixing a silent reorg release
Closes the TODO in +update. It was not cosmetic: it is why a whole class of
reorg releases nothing.
The bug
-------
heaviest-chain is contiguous 0..tip, and +update only ever walks DOWN from a
new tip -- it revises heights below and never touches anything above. Nothing
else removed those entries either. Heaviness is accumulated-work, not height
(+compare-heaviness), so a reorg onto a chain carrying more work in FEWER
blocks LOWERS the tip and leaves every entry above it still naming the chain
we just abandoned.
Those entries are not inert, because +release-orphaned-branch classifies with
this index:
=/ canonical (~(get z-by heaviest-chain) block-height)
?~ canonical c
?: =(u.canonical cur) c :: "reached the common ancestor"
After a tip-lowering reorg it is handed old-heavy, looks up old-heavy's own
height, and finds the stale entry there still naming old-heavy. It reads that
as having reached the common ancestor and returns immediately, having released
nothing at all. The entire abandoned branch stays stranded -- the exact bug
+release-orphaned-branch exists to fix, silently not fixed, on the one reorg
shape that also strands the entries.
This is not hypothetical. The mainnet node holds 105.916 blocks at tip
105.731: 184 blocks are off the canonical chain, but only 176 were ever
released. The other 8 sit above the tip.
The fix
-------
+prune-above walks up from tip+1 until a height is absent, dropping what it
finds. O(height dropped), never O(chain), and O(1) in the ordinary case where
the tip only rises and there is nothing above it -- so no event pays for the
size of the chain.
It runs BEFORE the walk, which is load-bearing. The walk returns as soon as it
finds a height that already agrees, and on a tip-lowering reorg the tip's own
entry usually already agrees -- so a prune placed after it would never run on
precisely the reorgs that strand these entries.
Tests
-----
test-derived-update-prunes-heaviest-chain-above-tip plants entries above the
tip and asserts +update drops them while leaving the tip's own entry alone.
It fails on the parent commit, with both stale entries surviving. Planting is
how the state is reached: forging a genuinely heavier-but-shorter chain would
mean driving accumulated-work through targets, which the page builders do not
expose.
roswell test-dumb: 273 passed, 0 regressions (268 on master + these 5). The
one remaining crash, test-v1-mempool-reject-oversize-tx, fails identically on
clean master and is untouched here.
+repair-orphaned-claims deliberately keeps classifying by ancestry rather than
switching to this now-correct index. The index is right going forward, but a
node upgrading today carries one maintained by kernels that had this bug, and
the boot repair is what has to be trustworthy without assuming otherwise. It
costs a second O(chain) walk at boot and it is worth it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Restore +repair-orphaned-claims's doc block to its arm
+canonical-block-ids was inserted between +repair-orphaned-claims's comment
and the arm it documents, so the comment read as though it described
canonical-block-ids while the arm itself had none. Comment-only move; also
notes that the repair now makes two passes over the size of the chain (the
ancestry walk plus the .blocks scan) rather than one, and that it covers what
a tip-lowering reorg silently failed to release.
roswell test-dumb: 273 passed, unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Revert "Prune heaviest-chain entries above the tip, fixing a silent reorg release"
This reverts commit 81250d0. The fix it claims does not occur, and it
regresses a peek.
The claimed fix is a no-op
--------------------------
+release-orphaned-branch walks from old-heavy and exits on the first of two
conditions:
?~ canonical c :: absent at this height
?: =(u.canonical cur) c :: names this block: common ancestor
On a tip-lowering reorg height(old-heavy) is above the new tip by definition.
Before the prune, the stale entry at that height named old-heavy and the
second line returned, releasing nothing. After the prune that entry is gone,
so the FIRST line returns, releasing nothing. The prune swapped one
zero-release exit for another; the branch stays stranded either way.
Making it real needs a companion change: absence above the tip is only
meaningful as "orphaned" once the index is exactly 0..tip, so the `?~` would
have to release and keep walking rather than stop. Without that the arm has no
consumer that benefits.
The test asserted only that +update drops the planted entries; it never
invoked +release-orphaned-branch, so it passed while the claim was false.
The regression
--------------
The %heaviest-chain peek (inner.hoon) indexes heaviest-chain by
highest-block-height, which is a monotone max over every accepted block --
including side-chain blocks that never become the tip -- and is never lowered.
So highest > tip is ordinary, not rare: the mainnet node holds 8 blocks above
its tip right now. The peek previously found the stale entry and returned a
resolvable block-id; after the prune it finds nothing and returns [~ ~], which
is indistinguishable from "this node has no chain". Consumers polling it for
the tip would have seen a chainless node immediately on deploy.
The underlying bug is real and stays open: heaviest-chain is never pruned above
the tip, so a reorg onto a chain with more accumulated-work in fewer blocks
lowers the tip and leaves entries naming the abandoned chain. Fixing it means
+prune-above AND the +release-orphaned-branch condition AND indexing that peek
by the tip rather than by highest-block-height. Until then
+repair-orphaned-claims picks the branch up at the next boot, because it
classifies by ancestry and does not consult the index at all.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Trim narration from the orphan-deletion comments
The comments explained the change rather than the code: what the previous
state was, what evidence justified the diff, which alternatives were rejected
and why. That belongs in a commit message, where its audience is. In the file
it is noise once merged, and it misleads -- a comment about what the code "now
also does" reads as a claim about behavior.
What is left states intent and invariants: that the retained set must be closed
under parent, that deletion is all-or-nothing because .balance is read with
`get`, that the repair is boot-only because it walks the chain twice.
Comment-to-code across this work drops from 125:83 to 49:73.
roswell test-dumb: 272 passed, unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Release the abandoned branch when a reorg lowers the tip
Heaviness is accumulated-work, not height (+compare-heaviness), so the winning
chain can end BELOW the branch it abandons. Three parts, none of which works
alone.
heaviest-chain is never pruned above the tip
--------------------------------------------
+update walks down from each new tip and stops at the first height that
already agrees; nothing ever removed an entry above one. A reorg that lowers
the tip therefore leaves entries above it still naming the abandoned chain.
+prune-above walks up from tip+1 until a height is absent, restoring the
invariant that the index's keys are exactly 0..tip. O(height dropped), O(1)
when the tip only rises, so no event pays for the size of the chain. It runs
before the walk, which returns early on the common path.
+release-orphaned-branch released nothing
-----------------------------------------
It classifies by that index, and exits on the first of two conditions:
?~ canonical c :: absent at this height
?: =(u.canonical cur) c :: names this block: common ancestor
Walking from old-heavy, whose height is above the new tip by definition: the
stale entry at that height named old-heavy, so the second line returned and
the whole branch stayed stranded. Pruning alone converts that into the FIRST
line returning -- same nothing. Absence is only evidence once the index is
exactly 0..tip: above the tip it now proves the block is not on the heaviest
chain, so the walk releases and continues, while absence at or below the tip
still proves nothing and stops, rather than hand back a tx that is really
mined.
The %heaviest-chain peek reported the wrong block
-------------------------------------------------
It indexed heaviest-chain by highest-block-height, a monotone max over every
accepted block -- side chains included -- that is never lowered. So it named a
height the heaviest chain need not have reached: before the prune a stale
block-id, after it nothing at all, which reads as "this node has no chain".
The mainnet node holds 8 blocks above its tip, so this is the ordinary case
there, not a rare one. It now reads the tip from heaviest-block, which is what
the caller wants (block_explorer.rs: "Peek /heaviest-chain ~ to get current
tip").
Tests
-----
test-reorg-lowering-tip-releases-abandoned-branch drives the real +update into
the real +release-orphaned-branch and asserts both blocks' txs return to the
mempool. Reverting only the release condition, with the prune still in, fails
it -- the earlier attempt at this fix shipped a passing test that never
invoked the arm it claimed to fix, so the discrimination is checked here
rather than assumed. test-heaviest-chain-peek-reports-tip likewise fails
against the old peek, reporting height 4 with the wrong block where the tip is
at 2. test-release-orphaned-branch-stops-at-gap-below-tip pins the
conservative half: a hole at or below the tip releases nothing.
Reaching a lowered tip needs accumulated-work driven through targets, which
the page builders do not expose, so the tests move the tip directly and let
the real +update rebuild the index around it.
roswell test-dumb: 276 passed, 0 regressions (268 on master + these 8). The
one remaining crash, test-v1-mempool-reject-oversize-tx, fails identically on
clean master and is untouched here.
+repair-orphaned-claims still classifies by ancestry rather than this index. A
node upgrading today carries an index maintained by kernels without the prune,
and the boot repair must be trustworthy without assuming otherwise.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Report the tip's own height from the balance peeks, and let the v1 snapshot follow it down
Fallout from re-indexing %heaviest-chain by the tip.
The balance peeks returned an incoherent triple
-----------------------------------------------
%balance-by-first-name and %balance-by-pubkey paired highest-block-height with
heaviest-block's id AND heaviest-block's balance: the height named one block,
the other two named another. That is the same defect %heaviest-chain had, left
in its two siblings.
It matters because the two ends of the v1 balance cache key are supplied by
different peeks. The insert side (cache.rs) keys on the balance peek's
(height, block-id); the lookup side (server.rs) keys on %heaviest-chain's. Both
read highest-block-height before, so they agreed. Re-indexing only
%heaviest-chain by the tip would leave the lookup asking for the tip's height
while the insert stamped highest-block-height -- and since the cache matches
exactly, every v1 balance query would miss, silently (the miss metric only
fires on an absent snapshot) and permanently, because highest-block-height is
monotone and never re-converges. The mainnet node holds 8 blocks above its
tip, so that is its ordinary state, not a rare one.
Both peeks now report the tip's height, matching the id and balance they
already returned.
The v1 snapshot pinned itself to the high-water mark
----------------------------------------------------
The refresh only replaced the snapshot when the new height was >= the cached
one. That was safe while the peek was monotone. Now that it reports the tip,
which a reorg onto a shorter heavier chain lowers, refusing to follow it down
pins the snapshot to a height the chain no longer holds -- missing every cache
lookup and letting heaviest_chain_age_seconds climb on a healthy node. The tip
is the truth, so the snapshot always takes it, and warns when it moves down.
Also hoists get-cur-height out of +release-orphaned-branch's walk: the release
mutates .c but not .heaviest-block or .blocks, so it was loop-invariant and
cost O(depth log n).
Tests
-----
test-reorg-lowering-tip-releases-abandoned-branch now forks the winning chain
at block 1 rather than ending it at block-3's own parent. A prefix of the
abandoned chain can never out-weigh it, and the walk would stop at the fork
point without ever taking the "index names a different block at this height"
branch -- so the test covered only absence-above-tip. It now releases block-2
through that branch as well. Still fails with the release condition reverted.
roswell test-dumb: 276 passed, 0 regressions.
Not fixed: the v2 block explorer only fetches forward (current_height <=
last_height returns early), so a lowering reorg leaves it serving blocks above
the new tip until the chain grows past the old high-water mark. It has never
handled a reorg at any height -- blocks replaced at or below max_height are not
re-fetched either -- so this is that pre-existing gap, now reachable through
one more path rather than a new one.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Make +release-orphaned-branch reject an index that does not describe the tip
The arm's result depends on +update having revised and pruned heaviest-chain
for the new tip before it runs (inner.hoon: update:der, then the release).
Nothing enforced that. Handed the index from before the reorg, the walk finds
old-heavy's own stale entry at old-heavy's height, reads it as the common
ancestor and returns having released nothing -- silently, and identically to
the bug this whole line of work exists to fix. The two are passed separately:
heaviest-chain as an argument, the tip via the door's .c, so a reordering makes
them disagree with nothing to catch it.
It now checks the one property +update establishes and the walk relies on: the
index names the tip at the tip's height. A reordering crashes on the first
reorg, named, instead of quietly under-releasing. O(log n), once per reorg.
Safety never depended on this -- releasing requires height > tip, and the
heaviest chain's maximum height is the tip, so a stale index can only cause
under-release. What it buys is that the failure is loud.
test-release-orphaned-branch-rejects-stale-heaviest-chain hands it a pre-reorg
index and expects the named crash; without the assert it reports "expected
crash, got:" -- the silent return.
roswell test-dumb: 277 passed, 0 regressions.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Let the v2 snapshot follow the tip down as well
0e65c48 fixed the monotone refresh guard in the v1 server and missed that v2
carries an identical one. v2 is the copy that matters: public_nockchain/mod.rs
re-exports v2's driver, so that is the server the node runs (v1 is reached only
by the bridge, via crates/bridge/src/main.rs).
Both versions key their balance cache the same way -- insert from the balance
peek, look up from %heaviest-chain -- so both depend on those two peeks
agreeing, and both are fixed by reporting the tip from the kernel. The guard is
per-server and had to be fixed in each.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Drop the gate hints from two arms that are no longer gates
+repair-orphaned-claims lost its sample when it stopped taking heaviest-chain,
and +canonical-block-ids never had one, but both kept a ~/ hint. ~/ registers a
GATE battery, so the runtime cannot match a parent battery for either and
prints on every boot:
serf: cold: register: could not match parent battery at given axis:
canonical-block-ids
serf: cold: register: could not match parent battery at given axis:
repair-orphaned-claims
Semantically transparent -- hints compile to nock-11 and change no result --
but neither arm is jetted, so the hints bought nothing and printed an error
each boot. No other nullary arm in this library carries one (+apt,
+check-and-repair, +get-cur-height, +get-cur-balance). +delete-orphan-blocks
and +prune-above keep theirs: both are still gates.
Caught on the mainnet backbone deploy, not by hoonc -- compiling proves only
that a hint parses, never that it registers. A runtime run is the only check,
and roswell test-dumb is now clean of these.
roswell test-dumb: 277 passed, unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Restore the kernel as the only authority for relay and fetch liveness in the Nous req-res driver. A syntactically valid future heard-block could previously enter Rust catch-up state before kernel validation, suppress kernel gossip, suppress or replace kernel BlockByHeight singleton requests, survive peer cleanup, and queue speculative raw-tx fetches from attacker-declared tx ids. The base commit d278ef8 raises the Serf interpreter stack reserve for the tx-in-mined-block OOM/stack-exhaustion path. This patch is separate: it preserves that runtime mitigation and removes the networking liveness footgun that let unvalidated transport observations censor relay/fetch behavior. The driver now always relays kernel %gossip effects, always keeps the kernel BlockByHeight singleton/bundle request live, and treats range prefetch as concurrent work gated by kernel demand depth and the validated frontier. Inflight range coverage suppresses only duplicate range prefetches. The obsolete catch_up classifier, sync-mode metrics, gossip suppression metric, observed-height candidate state, and unsafe catch-up config keys are deleted. The replacement knob is prefetch_kernel_demand_threshold, and duplicate range avoidance is reported through nockchain-libp2p-io.prefetch_duplicate_range_avoided_total. Future heard-block and bundle-remainder tx ids now record only source hints; they no longer queue speculative raw-tx network requests before the kernel asks. Peer cleanup drops that peer's deferred heard-block entries without deleting honest candidates from other peers. The active Nous protocol spec documents the kernel-demand range prefetch rule, retired config/metrics, and current verification matrix. The e2e runner and scenarios were updated for the current wallet CLI output and fakenet fee scale so the security gates exercise the network path instead of stale wallet assumptions. Verification: - RUSTFLAGS="-C target-cpu=native" cargo test -p nockchain-libp2p-io unvalidated_future_block_does_not_suppress_outbound_gossip -- --nocapture - RUSTFLAGS="-C target-cpu=native" cargo test -p nockchain-libp2p-io inflight_prefetch_does_not_suppress_kernel_singleton -- --nocapture - RUSTFLAGS="-C target-cpu=native" cargo test -p nockchain-libp2p-io deferred_candidate_does_not_suppress_kernel_singleton -- --nocapture - RUSTFLAGS="-C target-cpu=native" cargo test -p nockchain-libp2p-io peer_cleanup_discards_only_its_deferred_blocks -- --nocapture - RUSTFLAGS="-C target-cpu=native" cargo test -p nockchain-libp2p-io route_response_fact_deferred_future_heard_block_records_tx_hints_without_prefetch -- --nocapture - RUSTFLAGS="-C target-cpu=native" cargo test -p nockchain-libp2p-io route_bundle_envelope_unpacks_block_txs_and_records_unincluded_hints -- --nocapture - RUSTFLAGS="-C target-cpu=native" cargo test -p nockchain-libp2p-io --lib - RUSTFLAGS="-C target-cpu=native" cargo test -p nockchain-libp2p-io --test req_res_kernel_pressure - make assets/wal.jam, then RUSTFLAGS="-C target-cpu=native" cargo build -p nockchain -p nockchain-wallet --release - cargo run -p nockchain-e2e --release -- run tests/e2e/scenarios/nous_gen2_partition_reorg.yaml --nockchain-bin target/release/nockchain --wallet-bin target/release/nockchain-wallet --work-dir target/nockchain-e2e-nous-security --base-p2p-port 24100 --base-grpc-port 26100 --base-private-grpc-port 27100 - cargo run -p nockchain-e2e --release -- run tests/e2e/scenarios/nous_gen2_tx_lifecycle.yaml --nockchain-bin target/release/nockchain --wallet-bin target/release/nockchain-wallet --work-dir target/nockchain-e2e-nous-security --base-p2p-port 24100 --base-grpc-port 26100 --base-private-grpc-port 27100 - cargo fmt --check - git diff --check
Adds a shared outbound-work tracker in `nockapp` so shutdown waits for in-flight network effects to resolve instead of dropping runtime tasks mid-request. The exit driver now registers `nockchain-grpc` effects, drains pending outbound work with a 120s timeout before honoring `%exit`, and logs if work never resolves. The gRPC public driver marks `SendTx` work complete as soon as the RPC future resolves (before logging), ensuring exit unblocks reliably. Also removes the wallet boot-time mainnet/fakenet mismatch error path and marks `is_fakenet` as dead code.
Retained mempool transactions now re-gossip on chain progress and local resubmission, not on every timer tick. Peer-origin duplicates stay silent so echo loops cannot form. The retention window still leaves pending-block txs unbounded, but orphaned excluded txs use a bounded safety lease so a reorg can recover them without pinning spent txs forever. Verification: make assets/dumb.jam; make assets/roswell.jam; cargo build --release --bin roswell --bin nockchain; target/release/roswell test test-v1-mempool; target/release/roswell test test-pending-tx; target/release/roswell test test-reorg-orphaned-tx-survives-retention-sweep.
Roswell and nockchain use Exact jet dispatch. They must treat %sham hints as transparent even when a workspace build enables the nockvm sham_hints cargo feature for honk. %sham direct name dispatch now follows JetDispatchMode::HintBlind, the honk-only mode that needs native-prelude direct dispatch. Feature-poisoned roswell builds no longer execute direct %in/%put sham lookups. Verification: cargo build --release -p roswell --features nockvm/sham_hints; target/release/roswell test test-v1-mempool-new-heaviest-regossips-retained-tx returned unknown_jet_count=0; cargo build --release --bin roswell --bin nockchain; standard roswell test-v1-mempool-new-heaviest-regossips-retained-tx returned unknown_jet_count=0.
CI failed on roswell kernel parity because honk stopped collecting /= imports at a commented-out import line in hoon/tests/dumb/main.hoon, leaving later faces such as asert-tests out of the native compile subject.\n\nMake comment-only and blank lines transparent while scanning the leading import block, matching the Hoon parser's behavior, and cover the Roswell-shaped case with a parser unit test.\n\nVerified:\n- cargo test -p honk pipeline::tests::parses_imports_after_comment_lines_in_import_block -- --exact\n- cargo run --release -p honk --bin honk -- --new --output target/roswell_native_verify.jam --prelude hoon/common/hoon.hoon hoon/apps/roswell/roswell.hoon hoon\n- cargo fmt --check\n- git diff --check
The type-probe parity job exhausts GitHub runner disk while Bazel retains every hoonc/honk scratch directory as a declared output. Only the generated JAM is consumed downstream, so keep PMA/checkpoint state in an action-local scratch directory and delete it at action exit. Lower hoonc Bazel action logging from trace to warn; trace logs add noise without being part of the parity artifact.
Filter retained candidate transactions against the heaviest-chain balance before retrying them in the miner candidate. Skip absent raw transactions when regossiping accepted block transactions, and make disabled mining a no-op in candidate refresh paths. Validation: - make assets/dumb.jam && make assets/miner.jam && make assets/roswell.jam - RUSTFLAGS="-C target-cpu=native" cargo build --release --bin roswell --bin nockchain - target/release/roswell test test-miner-add-txs-to-candidate-no-keys-is-noop - target/release/roswell test test-add-tx-to-candidate-block - target/release/roswell test test-fail-add-double-spend-to-candidate-block - target/release/roswell test test-pending-reject-inputs-not-in-balance
Parse elders peer IDs as the base58 text emitted by the Hoon kernel, so the request is queued to the peer that supplied the missing-parent block instead of falling back to an arbitrary limited peer set. Preserve malformed non-atom rejection before parse fallback and cover both source-peer dispatch and malformed-shape behavior. Validation: - RUSTFLAGS="-C target-cpu=native" cargo test -p nockchain-libp2p-io test_elders_request -- --nocapture
Build Hoon kernel jams one at a time and format only root-owned workspace packages. This avoids shared hoonc cache contention and prevents formatting vendored path dependencies.
Exclude generated Python state, local model files, miner fakenet data, and other workstation-only artifacts from repository status.
Keep the workspace lint configuration explicit and update existing compiler, node, type, and wallet code to satisfy the enforced checks without changing their behavior.
Treat the default local endpoint as optional for the peek-refresh benchmark while preserving a supplied NOCKCHAIN_BENCH_SERVER as a required target.
Move shared candidate decoding, reward configuration, and node control into nockchain-mining-common; add the external zk-pow miner and its effect-stream transport. Remove the node-resident miner driver while preserving the kernel's mining effects.
Exercise the external miner against a private gRPC mock node with a real Serf worker and verify that a solved candidate is returned as a %mined command.
Route external ZK miner candidates and returned proofs through explicit %mine-zk and %dumb-zkpow messages. Permit a v1-only reward configuration and emit the initial candidate when mining is enabled so a live subscription starts work immediately.\n\nExercise the tagged proof command in the standalone miner test and the real-worker mock-node submission flow.
regossip-excluded-txs-effects evaluates directly rather than forming a gate. Its ~/ hint produced an invalid cold-state registration on every new heaviest block; remove the non-semantic hint.
+heard-elders bailed the event on [oldest=0 ids=~], and refused to intersect a window at a genesis block it holds. The starting height of the ancestor walk, oldest + len - 1, was computed before the list was inspected, so an empty list at oldest zero decremented zero and crashed the poke. The message needs no proof of work to send: NockchainRequest::Gossip skips verify_pow, legacy gossip acceptance defaults on, and heard-elders arriving over gossip is warned about but poked anyway. Each one costs two serialized kernel events and a wipe of the Nock memo cache, which poke_swap clears on every crashed event. An empty ancestor list is now rejected as a lie: +get-elders always names at least the block at the top of its window, so it is not a response this kernel can produce. The walk also gave up at height 0 before testing membership there, so a window bottoming out at the shared genesis reported no intersection and earned the peer %differing-genesis -- a permanent block_peer plus a fail2ban log line, for agreeing with us. Windows reach height 0 on any fork inside the first 24 blocks of a chain, so two nodes on opposite sides of an early fork banned each other and the partition stuck. Mainnet is tall enough that honest windows never start at 0. Testing membership before the guard fixes it; the guard still stops the underflow on the recursion. Three regression tests drive a real dumbnet kernel: the empty list is answered with %liar-peer, a window ending at our own genesis requests height 1, and a window ending at a genesis we do not hold still reports %differing-genesis, so the ban is narrowed rather than retired. Gate: cargo test -p nockchain-libp2p-io --lib, 330 passed 0 failed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
+heard-block emits [%request %block %elders <digest> <peer>] from the missing-parent branch, which runs before check-digest and before check-pow. The only gate is compare-heaviness, a bare gth on the page's own accumulated-work field. So a page with an unknown parent, a height at the tip and an inflated work field triggers the request, and both the digest and the work field are read straight off it, unvalidated. The driver damped this with a 5s cooldown keyed on block_id:peer. The attacker picks the digest, so every message was a fresh key and the cooldown never engaged. suppress_duplicate_active_outbound_request missed for the same reason. The cost is asymmetric. Inbound legacy gossip carries no proof of work at all -- NockchainRequest::Gossip returns Ok(()) from verify_pow and req_res_legacy_gossip_accept_enabled defaults to true -- while every outbound request solves an equix puzzle synchronously on the swarm event loop. Measured with crates/equix-latency on an M-series mac: solve p50 6.7ms, p95 13.5ms, against 0.10ms to verify. The per-IP gossip bucket allows a 120 burst at 2/s, so one address could stall the event loop for ~0.8s per burst and ~13ms per second indefinitely, scaling with source addresses. A peer now gets one elders request in flight at a time. The peer a page arrived on is the only part of the trigger that cannot be forged, so it is what the limit keys on. The recovery walk is sequential -- each response drives the next request -- so it never wants two at once and is not slowed; the slot is released when the peer's heard-elders response is routed, when a request to it fails, and when its session ends, with a TTL above the req-res timeout so a silent peer cannot wedge recovery. The identical-request cooldown is unchanged and still serves its own purpose: the kernel re-emits the same request on every duplicate-block poke. Gate: cargo test -p nockchain-libp2p-io --lib, 333 passed 0 failed. elders_requests_from_one_peer_are_bounded_by_its_slot fails 6 != 1 with the slot disabled, so it bites. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
+v1-to-v1 requires a v1 tx to pay +calculate-min-fee, which is at least .min-fee.data whatever the tx weighs. +heard-tx never applied that bound, so a tx paying less was admitted, gossiped, retained for the tx lease, re-gossiped on every new heaviest block, and re-processed by the miner on every candidate refresh -- while no block carrying it could ever validate. Its inputs stayed pinned in .spent-by, so the sender could not replace it either. The cost is one signature per lease window per note, and it is free: the tx is never mined, so the sender keeps the coins and can resubmit forever. Every peer that hears it repeats the whole cycle. The bound is now measured at the earliest height that could carry the tx, which is the height +v1-to-v1 will measure it at. It depends on the height only through the bythos base-fee halving, so a tx admitted here is one later heights also accept. Underpaying txs are discarded rather than answered with %liar-peer, matching the neighbouring context check: nodes straddling a phase boundary can legitimately disagree. This is the same asymmetry the size guard above it already closes. The sibling gap it does not close: +heard-tx applies no v1-phase gate either, so a v0 tx after the cutoff or a v1 tx before it is admitted and can never be mined. That one needs pre-cutoff v0 notes to exercise, and mainnet is long past v1-phase. The mempool fixtures built fee-0 txs and asserted they were accepted, encoding the old behaviour; they now pay the floor. Gate: roswell test test-v1-mempool, 10 passed. The pending/reorg integration arms, the other heavy +heard-tx user, 30 passed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Preserve historical proof encodings below activation. At height 112500, reject zero-padded composition polynomials whose distinct proof hashes leave Horner evaluation unchanged. Re-anchor %zk ASERT to three million proofs per second. Canonical and scheduled anchors share one puzzle-keyed schedule; dynamic timestamps remain isolated by puzzle type and are recovered from a validated branch after a late upgrade. Regression coverage includes the activation boundary, late-upgrade recovery, state migration, ASERT and Dumb suites, and the release prover benchmark.
Update ASERT target computation to use `min-timestamp` from the anchor consensus constant when it is present, instead of always looking it up from retained state. This prevents failures when historical block state has been pruned and keeps fixed anchor data truly constant-driven. Added an integration test that clears retained blocks and confirms target calculation still succeeds using the fixed anchor timestamp.
Switch blockchain constant formatting in mismatch errors to full `Debug` output so all fields are shown, not just a subset. Update the mismatch test to mutate `asert_anchor_min_timestamp` and assert that field is surfaced in the error message.
Keep normal bridge startup fail-closed when persisted blockchain constants differ from the connected Nockchain node. Add a maintenance-only opt-in that updates and read-backs the stored value while withdrawals are disabled, with field-level mismatch diagnostics for operators.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
8 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.