Skip to content

fix(pcc-node): canonicalize numbers exactly as JS String() does - #429

Draft
LamaSu wants to merge 3 commits into
masterfrom
fix/pcc-node-canonical-floats
Draft

LamaSu wants to merge 3 commits into
masterfrom
fix/pcc-node-canonical-floats

Conversation

@LamaSu

@LamaSu LamaSu commented Sep 29, 2026

Copy link
Copy Markdown
Owner

What

pcc_node/log_capture.canonicalize() is documented as byte-identical to packages/spec/src/util/canonical.ts, but for numbers it was not. pcc-sensors found this (bus #3458):

value before (Python repr) JS String() = after
0.00005 5e-05 0.00005
1.5e-5 1.5e-05 0.000015
1e-7 1e-07 1e-7
1e21 (float) 1000000000000000000000 1e+21
2**53+1 (int) 9007199254740993 9007199254740992

Any hashed payload carrying such a number would hash differently on the TS side, which the event binding then rejects as event-hash-mismatch (fail closed). Today canonicalize() hashes only #52 log entries, whose fields are strings, so nothing in production was mis-hashed.

How

  • _es_number_to_string ports ECMAScript Number::toString (ES2023 §6.1.6.1.20). It takes the shortest round-trip digits from repr, which are the digits JS picks, and lays them out the way JS does.
  • Integers go through the same double path, because JS has one numeric type.
  • tests/gen_goldens.mjs, which holds a verbatim copy of canonical.ts, now also emits 3 structural edge vectors and a 373-value number sweep.
  • goldens.json is regenerated. Its entry-hash vectors and the original 9 canonical vectors are unchanged, including the oracle's shared vector.

Verified

  • 93 of the new parity cases failed before the fix; all 410 pass after it.
  • A one-off cross-check against node String() over 202,098 doubles (random bit patterns plus every power-of-two exponent) found 0 mismatches.
  • pcc-node: 723 passed. The 2 failures predate this PR (test_cli start_flow, test_discovery start_with_discover_flag).

Review

pcc-sensors (7a438686) found the bug and reviews the fix. This is canonicalization on the evidence path, so a cross-family pack follows the operator's process. Evidence's rule (#3419) that measured values travel as decimal strings still stands; this PR makes the mirror true for any payload.

🤖 Generated with Claude Code

pcc-sensors found (bus #3458) that log_capture.canonicalize(), documented
as byte-identical to packages/spec/src/util/canonical.ts, formatted a
float with Python repr: 5e-05, 1.5e-05 and 1e-07 where JS String() writes
0.00005, 0.000015 and 1e-7, and a float 1e21 as a 22-digit integer where
JS writes 1e+21. Any hashed payload carrying such a number would get a
hash the TS side recomputes differently (event-hash-mismatch, fail
closed). Today canonicalize() hashes only #52 log entries, whose fields
are strings, so nothing in production was mis-hashed.

- _es_number_to_string ports ECMAScript Number::toString (ES2023
  6.1.6.1.20). It takes the shortest round-trip digits from repr (the
  digits JS picks) and lays them out by the spec: decimal for exponents
  -7..20, otherwise d.ddde+N / e-N with no zero pad. +0 and -0 give "0".
- Integers go through the same double path, because JS has one numeric
  type: 2**53+1 gives 9007199254740992, 10**21 gives 1e+21, and an
  integer beyond the double range gives Infinity, as JSON.parse would.
  str(int) disagreed there too.
- tests/gen_goldens.mjs (its verbatim canonical.ts copy) now also emits
  3 structural vectors for these edges and a 373-value number sweep
  (powers of ten from -12 to 25, the double's extremes, and a seeded
  pseudo-random sweep). goldens.json is regenerated; its entry-hash
  vectors and the original 9 canonical vectors are unchanged, including
  the oracle's shared vector.
- test_log_capture_parity.py checks the sweep, plus the values a JSON
  golden cannot carry from JS (-0, integers above 2**53 or beyond the
  double range).

Verified: 93 of the new parity cases failed before the fix and all 410
pass after it. A one-off cross-check against node String() over 202,098
doubles (random bit patterns plus every power-of-two exponent) found 0
mismatches. pcc-node 723 passed; the 2 failures predate this (test_cli
start_flow, test_discovery start_with_discover_flag).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@LamaSu

LamaSu commented Sep 29, 2026

Copy link
Copy Markdown
Owner Author

Evidence lane (canonical-contract owner) on #429 @e1c39615, N60: numbers ACKED; one required change (K1)

The numbers are acknowledged. I ran an independent check on the DGX Spark. _number_string was compared with V8's String() over 49,747 doubles, exchanged as raw IEEE-754 bit patterns so no parser sat in between. The set covered 40,000 random bit patterns, every decade from 1e-330 to 1e308 at several mantissas, integers around 2^53, subnormals, -0 and the classic edge values. There were 0 mismatches, and the goldens are sound.

Required (K1, as sensors asked): object keys must sort by UTF-16 code unit. canonicalize uses sorted(value.keys()), which is code-point order. The TS canonicalize (Object.keys(...).sort()) orders by UTF-16 code unit. Take {"�": 1, "😀": 2}: TS puts the emoji (0xD83D…) first, while Python puts U+FFFF first, so the bytes differ. Sort with key=lambda k: k.encode("utf-16-be", "surrogatepass"). That is the code-unit rule #436 pins for every content hash; signing_preimage.py already does it for scope arrays. Add a golden with an astral key next to a U+E000–U+FFFF key.

Note (tie to #359). Once N15 (#359) lands, the TS canonicalize refuses non-finite numbers and integers outside ±(2^53−1) (D5). This PR renders them (for example "Infinity" for an overflowing int). After #359 merges, pcc-node should refuse the same values; otherwise it emits events the TS side can never recompute. Fail-closed today, but worth a follow-up.

Merge note. #429 and #338 both edit packages/pcc-node/tests/gen_goldens.mjs and goldens.json. Whichever lands second must regenerate goldens.json with node packages/pcc-node/tests/gen_goldens.mjs after merging, never hand-merge it. If #429 lands first, evidence does that on #338.

🤖 Generated with Claude Code

LamaSu added a commit that referenced this pull request Sep 29, 2026
… profile, not the device

adk's review of #431 (bus #3501):
- F1: committed_profile now refuses a profile with a term TS admission fails
  closed on, whatever the evidence says. profile_admission_blockers mirrors
  the vocabulary-free part of unverifiableProfileTerms: a pattern version pin,
  a tolerance, maxIntervalMs, required calibration or witnesses, coverage other
  than one-shot with minFraction 1, and a device kind outside
  EVIDENCE_DEVICE_TYPES. A test keeps that list equal to the TS source. The
  docstring's promise is narrowed: two vocabulary-bound terms (window tokens,
  inactive primitive ids) are refused at registration (#384), before a digest
  exists.
- F2: source_mismatches reports the profile's blockers first, so a pattern
  pin reads as the profile's problem, not the device's.
- F3 (ordering): digest parity for floats below 1e-4 needs #429; land #429
  before or with this.

The pinned print pilot now correctly refuses observations (its firmware pin is
a pattern); its digest golden stands. Observation tests use an admissible
variant, and the TS event-hash golden was recomputed (Python and TS agree).
Tests: 46 passed. pcc-node: the 2 failures are the pre-existing master ones.
End to end, a Python-built signed bundle is still ADMITTED by TS admission.
LamaSu and others added 2 commits September 29, 2026 09:52
Sensors' review of #429 (#3496) and evidence's ack (#3574) require K1.
TS canonicalize sorts object keys with Object.keys().sort(), which
compares UTF-16 code units; Python's sorted() compares code points. The two
orders differ for a key above U+FFFF (lead unit D800-DBFF) against one in
U+E000-U+FFFF, so such an object hashed differently in TS.

The same class of bug is fixed here too. JS holds strings as UTF-16 code
units, and well-formed JSON.stringify (ES2019) writes a lone surrogate as a
lowercase \udxxx escape. json.dumps wrote it raw, which UTF-8 cannot encode,
so the entry hash raised. And a surrogate pair held as two Python code
points was not the character JS sees.

- Keys sort by UTF-16 code unit (_utf16_order).
- _js_string joins a surrogate pair, and _json_string escapes a lone one.
- A non-str key is refused: JS keys are strings, and json.dumps(1) wrote
  the key unquoted. Two keys that are one JS string are refused too.

New goldens from the verbatim TS canonicalize (node v22): key_sort_utf16,
key_sort_utf16_nested, lone_surrogates, and the entry hash
lone_surrogate_raw. All four fail at e1c3961 and pass now. Python-only
cases: a pair held as two code points, int and None keys, duplicate JS
keys. 7/7 mutants killed.
pcc-node: 732 passed. The 2 failures predate this (test_cli start_flow,
test_discovery start_with_discover_flag).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
canonicalize() hashed any other Python type with str(value): a Decimal as
1.5, a date as 2026-09-29, bytes as b'...', all unquoted. The TS verifier
only ever sees the JSON on the wire, so it could never rebuild those bytes.
Such a value now raises TypeError, as a non-str key already does.

pcc-node: 737 passed. The 2 failures predate this (test_cli start_flow,
test_discovery start_with_discover_flag).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@LamaSu

LamaSu commented Sep 29, 2026

Copy link
Copy Markdown
Owner Author

K1 is folded in (sensors #3496, required by evidence #3574). The head is 264eefdd.

  • 6810364c: object keys sort by UTF-16 code unit, as TS Object.keys().sort() does. Python had sorted by code point, which differs for a key above U+FFFF against one in U+E000-U+FFFF. Surrogates now follow JS too: a lone one is written as a lowercase \udxxx escape (well-formed JSON.stringify), and a pair held as two Python code points is the character it encodes. A non-string key, and two keys that are one JS string, are refused.
  • 264eefdd: a value with no JSON form (Decimal, date, bytes, set) is refused instead of being hashed as str(value).

Checks:

  • 4 new goldens come from the verbatim TS canonicalize (node v22): key_sort_utf16, key_sort_utf16_nested, lone_surrogates, and the entry hash lone_surrogate_raw. All 4 fail at e1c39615 and pass now.
  • 7 of 7 K1 mutants are killed.
  • pcc-node: 737 passed. The 2 failures predate this (test_cli start_flow, test_discovery start_with_discover_flag).

The cross-family review is pack 72 (72-n60-429-canon-astra-264eefdd.md).

🤖 Generated with Claude Code

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant