Skip to content

fix(spec): content hashes sort by code unit, never by locale collation - #436

Merged
LamaSu merged 11 commits into
masterfrom
fix/spec-content-hash-code-unit-order
Oct 1, 2026
Merged

fix(spec): content hashes sort by code unit, never by locale collation#436
LamaSu merged 11 commits into
masterfrom
fix/spec-content-hash-code-unit-order

Conversation

@LamaSu

@LamaSu LamaSu commented Sep 29, 2026

Copy link
Copy Markdown
Owner

What

Three content hashes in packages/spec sorted their inputs with String.prototype.localeCompare, which orders by ICU collation, not by code unit:

They now sort with compareCodeUnits (util/code-unit-order.ts), the order canonicalize already gives object keys.

Why

ICU collation is not code-unit order (oracle, bus #3348 and #3350):

  • ICU puts _ before -, : and ., and before digits.
  • ICU compares case-insensitively first.

On Node 22 / ICU 78 the kernel ids [kernel_a, kernel-1, kernel-a, kernel:a, kernel.a, kernela] sort in that order. Code units, and Python's sorted, give [kernel-1, kernel-a, kernel.a, kernel:a, kernel_a, kernela].

So a Python, Go or Solidity publisher or verifier, or the next ICU release, computes a different hash for the same content, and the pinned hash stops matching. The oracle holds its ident.registered_key mirror until this rule is pinned.

Vectors

packages/spec/src/util/code-unit-order.vectors.json:

  • the map-snapshot hash 0xfcb26a94…14ce over the six punctuation kernel ids. The ICU-order hash 0x55dd2ccc…5b94 is pinned as a negative;
  • a work schema with cmm_2 / cmm2 / cmm-1 and photo_set / photo2 / photo-1;
  • a job spec with co-funders 0xab…, 0xAB…, 0xAb….

An independent Python recomputation (sorted-key compact JSON over byte-sorted entries) reproduces all three hashes. A mirror in another language must sort by UTF-16 code units. For ASCII that is byte order; beyond the BMP, a Python mirror sorts with key=lambda s: s.encode("utf-16-be").

Safety

  • No consumer outside packages/spec calls these three functions on master, and no test pins their old values.
  • computeSetSnapshotHash already used code-unit .sort() and is unchanged.
  • Other localeCompare calls in the repo sort display lists, not hashed content.

Tests (DGX Spark)

  • spec 802/802, of which 5 are new in content-hash-order.test.ts; tsc is clean.
  • Mutation check: each of the 4 call sites reverted to localeCompare fails its vector.

Draft. Merging is the operator's call. #416 merges this before the oracle mirrors it.

🤖 Generated with Claude Code

@LamaSu @claude
fix(spec): content hashes sort by code unit, never by locale collation
bd048b2 
computeMapSnapshotHash (registry.ts), computeWorkSchemaHash
(work-schema.ts) and computeJobSpecHash (job-spec.ts) sorted with
localeCompare, which uses ICU collation. For kernel ids with - _ . : and
for mixed-case addresses, ICU order differs from code-unit order, so a
Python, Go or Solidity implementation, or the next ICU release, would
compute a different hash for the same content (oracle #3348, #3350).

All three now sort with compareCodeUnits (util/code-unit-order.ts), the
order canonicalize gives object keys. Vectors with punctuation ids and
mixed-case buyers are in util/code-unit-order.vectors.json; a Python
recomputation matches all three, and the ICU-order map hash is pinned as
a negative. No consumer outside spec, and no pinned golden, depends on
the old order.

spec 802/802, tsc clean. Each call site reverted to localeCompare fails
its vector (4 mutants).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
LamaSu added a commit that referenced this pull request Sep 29, 2026
@LamaSu @claude
merge: bring #436 (content hashes sort by code unit) into ident.regis…
d362aef 
…tered_key

computeMapSnapshotHash now sorts keys by UTF-16 code unit, never locale
collation, so a non-JS publisher of the kernel signing-key registry
computes the same snapshotHash (oracle #3348). #416's pinned vector
0x38ff407c..28db is unchanged: kernel-a and kernel-b sort the same way
under both orders.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@LamaSu LamaSu mentioned this pull request Sep 29, 2026
Draft
LamaSu and others added 10 commits September 29, 2026 19:29
@LamaSu @claude
fix(federation): LWW tie-break uses code-unit order, never locale col…
430d3fd 
…lation (E1 finding 1)

fixer-echo1: reproduced on bd048b2 (Node 22.23.2, ICU 78.2). At equal ticks,
compareTimestamps("r_a", "r-a") returned -1 while a code-unit mirror returns
+1, so both merge orders picked "quarantined" where a Go/Python/Rust replica
picks "trusted". localeCompare also returned 0 for distinct ids (U+00E9 vs
e+U+0301, "r"+U+0000 vs "r", "r"+U+200B vs "r"), so lwwRegisterMerge(a, b)
kept "trusted" and lwwRegisterMerge(b, a) kept "quarantined": two replicas
merging the same states in different orders diverged.

compareTimestamps now breaks ties with compareCodeUnits from @pcc/spec
(federation already depends on it). Distinct ids never tie, and the
U+FFFF "kill" sentinel still beats any real id at an equal tick.

New tests: 3 in lww-register.test.ts (2 failed before the fix).
federation 131/131, tsc clean.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@LamaSu @claude
fix(dht-core): endpoint URLs sort by code unit, never locale collatio…
011a737 
…n (E1 finding 2)

fixer-echo1: reproduced on bd048b2 (Node 22.23.2, ICU 78.2). For two
priority-1 endpoints ending "x_a" and "x-a", canonicalIdentityJson put x_a
first while a code-unit mirror puts x-a first, so the documented
hashing/signing preimage differed from a non-JS implementation. For URLs ICU
calls equal (".../U+00E9" vs ".../e+U+0301") localeCompare returned 0 and the
canonical JSON followed the input order (two input orders, two strings).

sortedEndpoints now breaks priority ties with compareCodeUnits from
@pcc/spec (dht-core already depends on it).

New tests: 2 in identity.test.ts (both failed before the fix).
dht-core 39/39, tsc clean; federation 131/131.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@LamaSu @claude
fix(spec): co-funder order is total over the hashed entry (E1 finding 3)
3c2a3dd 
fixer-echo1: reproduced on bd048b2. The same buyer twice with amounts [1,2]
hashed to 0x9fdcb017...dfd8 and with [2,1] to 0x29770397...2474, because the
comparator only looked at buyer and the stable sort kept input order. A
sealed job whose cofundedBy list was reordered then failed
assertJobSpecIsWellFormed with "jobSpecHash mismatch".

computeJobSpecHash now breaks buyer ties with the code-unit order of each
entry's canonical JSON, so any permutation hashes the same. Duplicate buyers
stay valid (no new rejection). Hashes of lists with distinct buyers are
unchanged (the existing jobSpec vector still reproduces).

New vector jobSpecDuplicateBuyers (4 entries, pins "10 sorts before 2");
an independent Python mirror (utf-16-be surrogatepass keys) reproduces its
hash 0xe8424ee8...6867 and the existing jobSpec vector.
New tests: 3 in content-hash-order.test.ts (2 failed before the fix).
spec 805/805, tsc clean.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@LamaSu @claude
fix(spec): registry keys must be printable ASCII before lowercasing (…
5e69e32 
…E1 finding 4)

fixer-echo1: reproduced on bd048b2 (Node 22.23.2, Unicode 17.0). JavaScript
lowercases U+0130 to two code units (U+0069 U+0307), so the distinct keys
U+0130 and i+U+0307 collide: both set snapshots hash to 0x01b19a0f...6fcd and
the map throws "duplicate key". On the same host, U+A7CB (added in Unicode
16.0) lowercases to U+0264 in Node but is unchanged in Python 3.12 (Unicode
15.0): the set snapshot of [U+A7CB] hashes to 0x50faa1c7...3232 in Node and
0xb58df978...2000 in a Python mirror, while an ASCII set hashes the same in
both (0xc9c00f3c...b8a2).

computeSetSnapshotHash and computeMapSnapshotHash now refuse any entry or
key outside printable ASCII (0x21-0x7e), before lowercasing, with a thrown
Error like the existing duplicate-key refusal; SetEntrySchema and
MapEntrySchema.key carry the same regex. Checked first: every existing
vector and test key is printable ASCII, and there is no non-test caller of
either function (lamasu/master ac86a40 and this branch).

New tests: 3 in content-hash-order.test.ts (2 failed before the fix). The
refused list includes U+212A (Kelvin sign, lowercases to ASCII "k"), which
kills a validate-after-lowercasing mutant.
spec 808/808, tsc clean.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@LamaSu @claude
test(spec): code-unit vectors beyond ASCII; UTF-16 reference; surroga…
7ae489f 
…tepass recipe (E1 finding 6)

fixer-echo1: reproduced on bd048b2. The test's "independent" comparator was
UTF-8 byte order: for U+E000 vs U+10000 compareCodeUnits returned 1 and it
returned -1, and it returned 0 for distinct lone surrogates (U+D800 vs U+DC00,
U+DFFF vs U+FFFD) because UTF-8 encodes each as U+FFFD. Every vector was
ASCII, so nothing caught this. The documented Python recipe
s.encode("utf-16-be") raised UnicodeEncodeError on a lone surrogate.

- vectors: new codeUnitOrder vector (11 strings: ASCII, U+00E9, U+E000,
  U+FFFD, U+FFFF, U+10000, U+10FFFF, lone D800/DC00/DFFF) with its sorted
  order and UTF-16BE bytes; the top-level rule no longer says byte order.
- test: the reference comparator is element-wise charCodeAt order; new
  checks that canonicalize orders these keys the same way and that UTF-8
  and code-point order differ. With the old UTF-8 reference the new vector
  test failed (1 failed | 14 passed).
- code-unit-order.ts: the Python recipe is
  key=lambda s: s.encode("utf-16-be", "surrogatepass").

A Python recomputation with that recipe reproduces every vector value
(codeUnitOrder, mapSnapshot incl. negative, workSchema, jobSpec,
jobSpecDuplicateBuyers); the old recipe raises on codeUnitOrder.
spec 812/812, tsc clean.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@LamaSu @claude
fix(spec): registry lookups refuse a non-ASCII candidate before lower…
7e64ccd 
…casing (review E1)

Follow-up to E1 finding 4, found while it was being fixed and reproduced
here. With registry keys now printable ASCII, setContains and mapGet still
lowercased the LOOKUP value with full Unicode rules. U+212A KELVIN SIGN
lowercases to ASCII "k", so "Kernel-1" matched the member "kernel-1":
setContains returned true and mapGet returned its value (two tests failing
before the fix). Both now treat any candidate outside ^[\x21-\x7e]+$ as
absent, before lowercasing.

spec 814/814; tsc clean.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@LamaSu @claude
fix(verifier): the deterministic fallback orders by code units, not t…
8a94121 
…he host's collation (review E1)

Review E1 (requested check 4) asked for an explicitly deterministic
comparator in VerifierSelector's not-enough-nodes fallback, which sorted
seeded sha256 digests with localeCompare. Reproduced: localeCompare
without a locale follows the host LANG, and Danish collation reads "aa" as
"å", after "z". For the seed in the tests, node-13's digest begins
"aa" and node-6's "ed". The new test passes under en_US and FAILS under
da_DK at the old code. It now uses compareCodeUnits, and passes under both.

verifier 586/586 (en_US; the selector file also 14/14 under da_DK);
tsc clean.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@LamaSu @claude
fix(verifier): verifier choice is independent of pool order, with ASC…
a65abf9 
…II ids (review E1b)

Cross-family review E1b on 8a94121, DO-NOT-SHIP.
- Finding 1 (HIGH): only the not-enough-nodes fallback canonicalized
  order; the weighted path walked the pool in its input order.
- Finding 2 (MEDIUM): exclusions and node ids went through Unicode
  toLowerCase, so U+212A (the Kelvin sign) excluded the distinct id "k-1".

Reproduced at 8a94121 first (4 failing tests):
- the five-node fixture, count 3, SEED: [v3,v5,v1] for the pool, but
  [v2,v1,v5] for the reversed pool (the reviewer's trace);
- a pool with duplicate ids was accepted;
- a Kelvin-sign exclusion was accepted and folded onto "k-1";
- a node whose id is not printable ASCII (or empty) was eligible.

Now: ids and exclusions are 1-128 printable ASCII and compare after
ASCII-only case folding. Eligible nodes are put in canonical order
(folded id, code units) before weights are computed, so any permutation
of the pool chooses the same verifiers. A duplicate id (ASCII case
folded) refuses the pool; a non-ASCII exclusion refuses the request; a
node with a non-ASCII id is never eligible.

verifier 590/590, tsc clean; 4 mutants (each check removed) killed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qb6kQhDYRwUDd6AVes3Fwx
@LamaSu @claude
fix(dht-core): endpoints order totally, and serialize in one key orde…
7621965 
…r (review E1b)

Cross-family review E1b on 8a94121, finding 3 (MEDIUM): sortedEndpoints
compared only priority and URL, so two endpoints differing only in
transport kept their input order, and canonicalIdentityJson's bytes
changed with it. Lane-found, same class: each endpoint was serialized in
its own key order, so {transport,url,priority} and {priority,url,transport}
gave different preimages.

Reproduced at 8a94121 first: both tests failed (the reversed pair, and
the reordered keys, each produced different canonicalIdentityJson bytes).

Now the sort breaks a priority-and-URL tie by transport (code units),
and the preimage rebuilds each endpoint as {transport, url, priority},
the declared fields in a fixed order. Bytes for endpoints already written
in that order are unchanged.

dht-core 41/41, tsc clean; 2 mutants killed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qb6kQhDYRwUDd6AVes3Fwx
@LamaSu @claude
fix(dht-core): refuse a non-finite endpoint priority (review E1c)
7bc7251 
Cross-family review E1c on 7621965: SHIP-WITH-FIXES, one MEDIUM. A NaN
priority made sortedEndpoints' comparator answer NaN (treated as equal),
so endpoints kept their input order and canonicalIdentityJson's bytes
changed with it; JSON also writes non-finite numbers as null.

Reproduced at 7621965 first: the new test failed ("NaN: expected
[Function] to throw"), i.e. NaN, Infinity and -Infinity were accepted.

sortedEndpoints now refuses any priority that is not a finite number
(RangeError) before sorting, so canonicalIdentityJson and
preferredEndpoint refuse it too. The test covers NaN, Infinity,
-Infinity, and the reviewer's reversed NaN pair. No other package calls
these helpers.

dht-core 42/42, tsc clean; removing the check fails the test.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qb6kQhDYRwUDd6AVes3Fwx
@LamaSu
LamaSu marked this pull request as ready for review October 1, 2026 18:48
@LamaSu
LamaSu merged commit 64fa02d into master Oct 1, 2026
8 checks passed
LamaSu added a commit that referenced this pull request Oct 3, 2026
@LamaSu @claude
merge: master @753feb43 into #360 (merge-up after EC1c's SHIP)
8fe16b6 
#360 was SHIP at c73d708 (EC1c) with full CI green, but master moved on 9/30–10/01 (#424,
#459 N80, #436, #338, #477, #483 and others), and three files conflicted. Resolutions:

- packages/spec/src/index.ts: both sides added exports at the same place. Both are kept: #360's
  `economics` namespace, and master's money-status and readmodels exports.
- packages/gateway/src/routes/swf.ts: both sides made POST /api/swf/epochs/:epochId/distribute
  answer 501 not_available before anything is read. Master's hunk (#421's, reviewed and merged
  with #424) is kept. The behavior is identical.
- packages/gateway/src/routes/rewards.ts: #360 made the reward, certificate and treasury routes
  answer 501 and deleted their fixtures. Master's N80 (#459, reviewed and merged) changed only
  the certificates: an empty list, 404 for one, and a 501 mint that claims nothing. It kept the
  reward and treasury fixtures. Kept: master's N80 certificates, and #360's 501s for rewards and
  treasury, with no fixture import. economics-no-fabrication.test.ts drops its three certificate
  rows (certificates-n80.test.ts pins N80), and the reward and treasury rows still assert 501.

On the merged tree: spec 1286/1286; contracts 267/267; mcp-server 76/76; gateway 3717 passed,
13 skipped, 0 failed; dashboard 353/353; tsc clean for contracts, mcp-server, gateway and
dashboard.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
LamaSu added a commit that referenced this pull request Oct 3, 2026
@LamaSu @claude
merge: master (7d688ce) into #359, which was CONFLICTING (DIRTY) afte…
6f513fe 
…r master moved

#359 (N15, canonicalize refuses non-JSON) was SHIP at aa01536. It went
CONFLICTING once master merged #436, #477, #483, #447 and #367, and it
dropped off MERGE NOW.

The only conflicts were import lists:
- packages/kernel-sdk/src/job-handler.ts: #359's canonicalSnapshot plus
  master's LO-EV-1 imports (parseEd25519SignatureHex,
  sessionKeyDelegationPreimage, signingPreimage);
- packages/kernel/src/digital/accounting-kernel.ts: canonicalSnapshot plus
  signingPreimage.

The code on both sides is unchanged. The suites check master's newer code
under #359's stricter canonicalize.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VGNHoFFhbAdeNBc4BWigst
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@LamaSu