v0.2.2: sort member names by UTF-16 code units (RFC 8785 §3.2.3) - #5
Merged
Merged
Conversation
… lineages)
Object member names were sorted by native byte/codepoint order in every
lineage (C strcmp, C++ std::sort, Rust BTreeMap<&str>, Go sort.Strings,
D keys.sort, Haskell Map codepoint order, OCaml String.compare). RFC 8785
§3.2.3 requires ordering by the UTF-16 encoding, comparing code units as
unsigned 16-bit integers. The two orders diverge only at the astral/BMP
boundary: a key >= U+10000 encodes to a high surrogate 0xD800-0xDBFF, which
is less than BMP scalars U+E000-U+FFFF, so astral keys must sort BEFORE such
BMP keys. Codepoint order placed them after.
Each lineage now compares keys by their UTF-16 code-unit sequence; only the
key comparator changed (escaping, number formatting, whitespace, and
duplicate-key detection are untouched). The existing conformance corpus has
no astral-plane keys, so all previously pinned hashes are unchanged — the fix
is purely additive.
Locks the behavior cross-lineage: adds conformance vector
utf16_key_order_astral_before_bmp ({"":1,"\uU10000":2}) to the shared
corpus (gen_corpus pins it only because all seven agree on the same SHA-256)
plus a per-lineage golden-vector test. verify_all_lineages.sh: 7/7 agree on
26 accept + 31 reject.
Also harden verify_all_lineages.sh reject gate: a crash (SIGSEGV/SIGABRT ->
exit >=128, or exec failure 126/127) is no longer scored as a clean
rejection. Only a controlled nonzero exit (1..125) counts as REJECT ok; a
crash on malformed input now fails the gate as a robustness defect.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
fuzz_agreement.py (seeded structured/mutation/garbage sweep that fails on any seven-way accept/reject or hash split AND on any lineage that dies on a signal) existed but was never run in CI — only verify_all_lineages.sh, differential_probe, and the ASan/UBSan gate were. A crash-on-input regression could therefore land on main unseen. Wire it into the verify job after the differential probe, seed pinned (20260715) so a CI failure reproduces exactly. Locally green (0 failures, 0 signal deaths) against the UTF-16-fixed CLIs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
STD-03: the handoff described the build as offline-capable, but a fresh C configure clones cJSON and C++ falls back to fetching nlohmann/json. README now says first build needs network (or a warm cache) and gives the exact offline overrides. No build behavior changed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Bumps rust/Cargo.toml, rust/Cargo.lock, haskell/baionstd-public.cabal and cpp/CMakeLists.txt (which still said 0.2.0) to 0.2.2. The CHANGELOG entry states that v0.2.1 sorted member names by byte / code-point order and that hashes differ only for objects mixing names above U+FFFF with names in U+E000..U+FFFF. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
v0.2.2 — member names sorted by UTF-16 code units (RFC 8785 §3.2.3)
This releases a fix made on 2026-07-19 that was committed locally but never pushed.
What changes
utf16_key_order_astral_before_bmppins the case:{"":1,"𐀀":2}puts the astral name first, sha2569d4cdc71dda603c42f9b21d88d0c2ffc31a76cd1bd461d7359406cf169845f1e.Checked locally, on a Linux x86-64 workstation
build_all.sh: all 7 lineages built.verify_all_lineages.sh:PASS: 7/7 lineages agree on 26 accept + 31 reject vectors.conformance/run_corpus_cli.pyagainst each of the sevenbaion_canon_hashbuilds: corpus green, 7 of 7.fuzz_agreement.py 20260715: 8000 cases, 0 failures.Corpus after the change:
accept.jsonl:3cef8214…reject.jsonl: unchanged,ef419edc…conformance_reference.json: unchanged,612d2b80…🤖 Generated with Claude Code