Skip to content

Restamp mechanically-explained verified.tsv rows to the config spelling - #2039

Merged
andrewboudreau merged 6 commits into
mainfrom
symbols/verified-restamp
Aug 30, 2026
Merged

Restamp mechanically-explained verified.tsv rows to the config spelling#2039
andrewboudreau merged 6 commits into
mainfrom
symbols/verified-restamp

Conversation

@tangosdev

@tangosdev tangosdev commented Aug 30, 2026

Copy link
Copy Markdown
Owner

Stacked on #2038 (the tool that generated it); review this one for the data, that one for the code. Merging: land #2038 first, then retarget or merge this; do not close it with a --delete-branch of the base.

What this is

The 334 mirror rows whose divergence from config the classifier fully explains, restamped to the spelling config asserts. Breakdown by verdict:

  • 248 class-rename (Actor -> dActor_c, MeshCollider -> dBgW_Kc, ActorBase -> fBase_c, ...), each class pair corroborated by at least one method row whose member and demangled arguments already agree
  • 52 class-rename + arg-rename, 8 arg-rename (the same waves seen inside argument lists)
  • 8 class-rename + dtor-variant, 2 dtor-variant, 2 ctor-variant, 1 class-rename + ctor-variant (the D1/D2/C1/C2 identity fixes config already carries, e.g. 0x0200651c _ZN8CapEnemyD2Ev -> _ZN11dCapEnemy_cD1Ev)
  • 7 signature-fix + 4 class-rename+signature-fix: REAL arity/type corrections that landed in config via migration byte-matches (e.g. 0x020156dc Clipper::Func_020156DC() -> Clipper::Func_020156DC(int, unsigned short, int, int)). Worth the closest read — three of the eleven rest on judgement rather than the function's own bytes: 0x020ada40's third Fix12<int> is unused and argued from call sites, 0x020ae2b8's Fix12<int>->int is ABI-identical so the ROM cannot arbitrate, and 0x0203c6ac's signedness rests on a blt in a different function plus family consistency. None of those three decisions is made by this PR — all eleven already live in config, backed by byte-matching src/ definitions; this PR only stops the mirror contradicting them.
  • 2 canonical-spelling (hand-spelled PKcPKcPKci vs the compiler's PKcS1_S1_i)

What it deliberately does not touch

The 27 rows the classifier cannot explain: the _ZThn80 Animation thunks, BootScene/FaderColor dtor identities, Spawn-vs-ctor naming debates, and every malformed or operator name. They stay divergent in the banked report for human eyes.

Three rows to eyeball

Cross-overlay addresses (flat mirror, two modules): 0x020ada40 (ov002/ov004, the KillByInvincibleChar find from the provenance note), 0x020aed98 (ov002/ov007), 0x020ee55c (ov002/ov007). In each case the one real config name at the address is what gets stamped.

No src/ files move, contributions.json is untouched, and dtor class pairs were never inferred from dtor rows alone (byte-identical bodies, the #774 lesson).

config/**/symbols.txt (what the build links against) and symbols/verified.tsv
(the import mirror) disagree on hundreds of addresses, and a name-keyed check
across them manufactured phantom crack targets out of matched, enrolled
functions on 2026-08-30. The tool enumerates every divergence, classifies each
one (class-rename wave, dtor/ctor variant, arg rename, real signature fix,
canonical mangling, needs-a-human), answers --where for one address, and can
plan or apply a restamp of the mechanically-explained mirror rows.

The classifier refuses what it cannot prove: dtor pairs need a method row to
corroborate a class rename (dtor bodies are byte-identical across classes),
operators and malformed names are refused rather than guessed, and thunk
fields, const-ness, and the full nested class path all have to agree. The
tests pin the refusals next to the acceptances, and pin the allowlist of
tools reading verified.tsv directly.
833 addresses where the records disagree today: 361 with a real name on both
sides and no shared spelling, 5 kind mismatches, 382 where config still holds
a placeholder, 85 with no config slot at all. Of the 361, the classifier
explains 334 mechanically and leaves 27 for a human.

Generated by tools/symrecords.py --report.
Names may only be looked up in the record the artifact in hand links against;
joins between records happen on address, module-qualified for overlays. Also
corrects the note's claim that verified.tsv is left stale on purpose: rows
have been restamped as corrections land for a while now, and the divergence
surface is banked in symbols/record_divergences.tsv.
@tangos-validator

tangos-validator Bot commented Aug 30, 2026

Copy link
Copy Markdown

✅ PR validation — Passed

noverify: no source/build-data changes in this PR

Each changed src/*.c|*.cpp is compiled and its relocated bytes compared to the binary data on a private build box. Passing requires every changed file to reproduce the ROM byte-for-byte with correct relocation targets — this catches WRONG-DEST relocations and non-reproducing near-misses that ledger-scoped linkcheck skips.

…elling

Every row whose divergence tools/symrecords.py can fully explain (class-rename
wave, dtor/ctor variant fix, argument rename, real signature fix, canonical
mangling) now spells the name the way config/**/symbols.txt asserts it. The 27
rows it cannot explain are untouched, as are the placeholder and no-slot rows;
the banked report shrinks accordingly.

Generated with tools/symrecords.py --restamp --apply, report refreshed with
--report. Import-time spellings remain in git history.
@tangosdev
tangosdev force-pushed the symbols/verified-restamp branch from d5cb13c to 6390b4a Compare August 30, 2026 18:57
@andrewboudreau

Copy link
Copy Markdown
Collaborator

Gatekeeper review — MERGE AFTER FIXES (two applied by me, one is a merge-order instruction)

The data is correct, complete and conservative. I could not find a data defect. The 334-row restamp is exactly what it claims: a mechanical re-spelling of the mirror to the name config/**/symbols.txt already asserts at the same address — no invented names, no address changes, no kind changes, no rows added or removed.

The decisive audit was demangling all 334 old and new names and looking for changes the body does not claim: zero unclaimed arity changes, zero unclaimed ctor/dtor variant changes. Every arity or C1/C2/D0/D1/D2 change in the diff is one of the 11 signature-fix or 13 variant rows you declare. An independent re-classification of all 334 rows reproduced your breakdown exactly (248/52/8/8/7/4/2/2/2/1 = 334); the four apparent disagreements were all artifacts of my tokenizer (S_ substitution semantics, _ZTV single-id rows, nested-class collapse) and resolved against tools/demangle.py in your favour.

Two things I want to call out as genuinely good practice, because they are what made this reviewable:

  • Inside the merged export, python tools/symrecords.py --report regenerates symbols/record_divergences.tsv byte-identically to the committed file. That proves the 334 rows were produced by the tool, not hand-deleted — which is the thing a reviewer otherwise has to take on trust.
  • python tools/symrecords.py --restamp on the merged result reports restamp plan: 0 rows. The plan was applied completely, with nothing mechanically-explained left behind.

And the class-rename corroboration rule is properly satisfied. For each of the 22 distinct old→new class pairs I counted restamped rows that are not ctors/dtors and whose demangled member name and argument list are identical across the rename — minimum 1, no pair had zero. That is the #774 lesson honoured: a compiler-generated destructor body is byte-identical across unrelated classes, so a lone diverging dtor pair is not evidence of a rename. Every dtor-based rename here is backed by ordinary method rows. No old class maps to two new ones (RaycastLine appears twice only because the outer class became dBgCh_Lin and the nested RaycastLine::Line became dM3dGLin, verified equivalent after substitution renumbering S3_S2_).


Fixed — 1. The body's flagship signature-fix example did not exist

The body cited `AfterClsnEv` -> `AfterClsnEi` as the worked example for the category it also called "worth the closest read." That change is not in the diff:

0x020eff18  _ZN16dPathLiftActor_c9AfterClsnEi  function   <- 2038 base (mirror)
0x020eff18  _ZN16dPathLiftActor_c9AfterClsnEi  function   <- 2039 head (mirror), identical

The tree's only AfterClsn row is untouched and was never divergent, and not one of the 334 rows has the shape …Ev -> …Ei (I checked the whole diff for that pattern: zero). On a PR whose entire subject is naming discipline, an invented worked example is the one thing that gets quoted back later as fact.

I have edited the body to use 0x020156dc Clipper::Func_020156DC() -> (int, unsigned short, int, int) instead — a real row, and the cleanest of the eleven. I also folded in the three caveats below, because a reader who takes "worth the closest read" seriously deserves to know which rows are the soft ones.

Fixed — 2. The provenance note goes stale the moment this merges

notes/symbol-name-provenance.md:308-311 ships in #2038 and uses two addresses as its live examples of divergence. This PR restamps both:

0x0203562c   mirror @2038: _ZNK12WithMeshClsn8IsOnWallEv  ->  @2039: _ZNK10dBgCh_Actr8IsOnWallEv
0x0200651c   mirror @2038: _ZN8CapEnemyD2Ev               ->  @2039: _ZN11dCapEnemy_cD1Ev

After this lands the mirror agrees with config at both, so the paragraph describes a divergence that no longer exists. The note is correct as of #2038 and wrong as of #2039, so the fix belongs here, not there — an agent is swapping in two of the 27 held-back DIVERGED rows, which stay true permanently.

Instruction — 3. Merge #2038 with a merge commit, NOT a squash

Measured both ways:

#2038 merged as then #2039 retargeted
merge commit git merge-tree CLEAN
squash CONFLICT (add/add) on symbols/record_divergences.tsv

A squash creates that file under a new blob identity while #2039 also adds it. The repo's history is merge commits so the default is already right; this only matters if someone reaches for the squash button.

Your body's --delete-branch warning is separately correct and I am honouring it: GitHub closes the child PR rather than retargeting it, so #2039 gets retargeted to main before #2038 is merged, and #2038 will not be merged with branch deletion.


Two findings that are not yours to fix, but that I am recording

tools/import_symbols.py silently reverts all 334 restamps — even on a dry run. On origin/main, lines 149-154 write the full file before the if not args.apply: return early exit:

with OUT.open("w", encoding="utf-8") as f:          # OUT = symbols/verified.tsv
    for addr in sorted(theirs): ...
print(f"wrote {OUT.relative_to(REPO)} ...")
if not args.apply:
    print("\n(dry run) re-run with --apply to write renames.")
    return

So python tools/import_symbols.py with no flags overwrites the mirror with raw import spellings. Exposure is limited — reference/ is gitignored, so most people cannot run it — but nothing documents it and no gate catches it. A dry run that writes is a bug worth its own PR; one line in the provenance note would do for now.

No CI gate can see a symbols/-only PR. No workflow on origin/main has a path filter naming symbols/**, which is why #2039 ran 3 checks where #2038 ran 6 — names, references and tools are all filtered to tools/** or **.py. The consequence that matters going forward: #2038's new test_symrecords.py::LiveTreeInvariants, which asserts on the live symbols/verified.tsv, will never run on a PR shaped like this one. Adding "symbols/**" to tool-tests.yml's paths closes it. Pre-existing, not introduced here, and the same family as this repo's other "gates blind to unenumerated files" incidents — a missing check reads exactly like a passing one.

One judgement call I am making knowingly

The pre-restamp spellings survive only in git historysymbols/actor_renames.tsv recovers 0 of 334. Your provenance note states this. I am accepting it: the old spellings were never authoritative, they are raw importer output, and config is the assertion. Recording it so the decision is visible rather than implicit.

Three of the eleven signature fixes rest on judgement, not on the function's own bytes

Stating these plainly because the ROM contains no mangled names, so nothing in the cartridge can prove Ei over Ev — what it can prove is the register/stack read, and for the other eight it does (ldr vs ldrh for SetBlendAlpha's 32-bit final argument; ldrb r0,[sp,#0x18] plus a byte-count argument for UntrackAndSpawnStar).

  • 0x020ada40 — the third Fix12<int> is admitted in the source comment as "genuinely unused; every caller materialises it and no body reads it." Argued from call sites.
  • 0x020ae2b8Fix12<int>int is a Fix12i typedef that mangles to i. ABI-identical; the ROM cannot arbitrate.
  • 0x0203c6ac — rests on a signed blt in a different function (0x0204eba4, rsb r2, r2, #0) plus family consistency with the other three Allocate variants.

None of these three decisions is made by this PR. All eleven already live in config/**/symbols.txt, backed by byte-matching src/ definitions (all 11 have a src/<exact-mangled-name>.cpp, all are enrolled in config/**/delinks.txt, all carry zero NONMATCHING banners). This PR only stops the mirror contradicting them. That is the right scope.


What I verified

Cross-overlay collision — no unflagged ambiguity. Exactly 3 of the 334 addresses are claimed by 2+ modules, and they are exactly the 3 you name. In each, the other module's entry is a machine-generated placeholder, and note the differing sizes and the function/data kind split — genuinely different code at an overlay-aliased address, which a flat mirror can only carry one of:

addr module A module B
0x020ada40 ov002 _ZN12dEnemyBase_c20KillByInvincibleChar… size 0x100 ov004 func_ov004_020ada40 size 0xbc
0x020aed98 ov002 _ZN12dEnemyBase_cC2Ev ov007 func_ov007_020aed98
0x020ee55c ov002 _ZN10dBgActor_c4KillEv kind:function ov007 data_ov007_020ee55c kind:data

Building (address → {module: name}) across all 106 config/**/symbols.txt: 2,341 addresses are multi-module, 243 have 2+ real names, and only one of those 243 (0x02110154) appears in verified.tsv at all — this PR does not touch it. Zero of the 334 new names fail to be asserted by config at their address, and the PR introduces 0 new duplicate names (it removes 1).

Nothing downstream breaks. Exactly two readers of symbols/verified.tsv exist on main: cluster_targets.py:31 reads only the address column (renames invisible), and cpp_rename.py layers verified.tsv under config/arm9/symbols.txt — so for arm9 config already wins, and for overlay addresses this PR does change its output, in the direction of agreeing with config. Not a gate; test_cpp_rename_verify passes on the merged tree. No by-name join exists anywhere — the two records are joined by address, never by name. check_dead_references cannot go red: the PR renames and deletes no file. Three src/ comments cite verified.tsv; only 0x02010ad8 is restamped, and that comment already spelled the new name, so the restamp fixes pre-existing staleness.

Merge-result gating — no gate differs. Both trees exported and re-git init/git add -A -f, with the export-completeness assertion passing on both (baseline ec83193c8: 12750/12750; combined 216aa258b: 12753/12753, adding exactly the 3 new files). tiers_ratchet --check, check_dead_references, langmode_audit --check, check_src_tu, check_duplicate_sources, port_refcheck, check_python_names, source_coverage, plus 12 unittest modules including test_cpp_rename_verifyall exit 0 on both trees, no verdict differs. Two reds (check_references, test_check_header_offsets) are byte-identical on both trees and are export artifacts, not attributable here. python -m unittest tools.test_symrecords → 21 tests OK.

No rombuild.py/eligible.py was run (shared build/ with live agents), so no byte-level ROM claim above is my own measurement — the "byte-matching" statements come from delinks enrollment plus the absence of NONMATCHING banners.


Body fixed, note fix in flight. Once #2038 clears its own review this merges right behind it — retarget first, merge commit, no branch deletion.

The paragraph cited 0x0203562c and 0x0200651c, both of which this branch
restamps -- the examples would read as agreeing rows the moment it lands.
Swap in two addresses the classifier holds back as needing a human, so they
stay divergent: 0x02043444 (fBase_c vs ActorBase) for the plain case and
0x02023598 (BootScene D1 vs Scene D2) for class-and-variant at once.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WdCK1xgrJdiJzPCh3bAJfQ
andrewboudreau added a commit that referenced this pull request Aug 30, 2026
… map

The 21 tests this PR added never ran. .github/workflows/tool-tests.yml
enumerates its modules on purpose -- a glob would silently absorb a
self-skipping toolchain suite -- and test_symrecords was not on the list,
so run 33329528573 printed "Ran 312 tests", exactly the count on
unmodified main. The path filter fired; there was just nothing new to run.

Adds tools.test_symrecords to the run: list and to the trailing table, and
moves the header's own count. That count is documented as "the number to
compare against if this job's total ever moves", so it is measured, not
computed: the enumerated invocation now reports "Ran 336 tests ... OK".
The module contributes no skips (it reads only tracked text and imports
nothing outside the stdlib), so the expected-skip paragraph goes 306/312
-> 330/336. The module census in the same header goes 57 -> 58, stale
since this PR added the file.

class_rename_map() iterated every config entry at an address with no
module filter, while compare() refuses to explain a row where two modules
both assert a real name ("ambiguous-module"). An address the report
refuses to draw a conclusion FOR should not quietly supply evidence to
conclusions ABOUT other addresses -- that evidence is invisible to a
reviewer reading the report, because the row it came from is refused.
Same floor now applies in both places. On the live tree this changes no
conclusion: the class map is 109 pairs before and after, byte-identical,
and symbols/record_divergences.tsv regenerates with zero lines of diff.

Adds SharedOverlaySlotIsNotAutomaticallyAmbiguous, which pins the guard
in BOTH directions so the next reader does not "fix" it. The tempting
change is to widen ambiguous-module to "more than one module has an
entry" -- that would refuse three provably right restamps (0x020ada40,
0x020aed98, 0x020ee55c), where the colliding overlay's slot is still a
func_/data_ placeholder and the mirror's own old name identifies the
class. Those three are recorded in the docstring rather than asserted,
because #2039 restamps them and the rows then stop existing; a pin on the
addresses would be a test that deletes itself.

Two doc corrections from review: notes/symbol-name-provenance.md now
carries the caveat symrecords.py:14-17 states properly, that a mirror row
keeps its import-time spelling when no correction touches it -- the table
row read as though every row tracks config. And the allowlist test's
docstring narrows "nothing new reads verified.tsv at all" to "nothing new
in tools/", since its glob is tools/*.py non-recursively and cannot see
tools/archive/*.py or any reader outside tools/.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WdCK1xgrJdiJzPCh3bAJfQ
@andrewboudreau
andrewboudreau changed the base branch from tools/symrecords to main August 30, 2026 20:13
@andrewboudreau

Copy link
Copy Markdown
Collaborator

Audited the 334-row restamp against the cartridge's own config. MERGE.

A name-only restamp of symbols/verified.tsv is exactly the change class that lands
tree-wide with every byte gate green, so I did not take the green table at face value. I
re-derived the whole delta and checked it against config/**/symbols.txt on origin/main
by address, which is the only key that cannot be argued with.

The delta is mechanically clean

property result
rows removed / added 334 / 334
addresses removed but not re-added none
addresses added but not removed none
new name matches config at that address 334 / 334
old name present in config at that address 0 / 334

So: no address moved, no record was added or dropped, every new spelling is the one
config already carries at that address, and every old spelling was stale. This PR is
repairing 334 orphaned records, not renaming anything.

It also recomputes the mangled length prefixes correctly — _ZN12MeshCollider…
_ZN7dBgW_Kc…, 12 → 7. That is the trap in this class of edit and it is handled.

What it actually is: coined → ROM normalization, 35 classes

Actor -> dActor_c (82)   WithMeshClsn -> dBgCh_Actr (27)   ActorBase -> fBase_c (25)
Scene -> dScene_c (20)   MeshColliderBase -> dBgW (19)     MeshCollider -> dBgW_Kc (16)
MovingMeshCollider -> dBgW_KcMbg (15)   CapEnemy -> dCapEnemy_c (13)
Enemy -> dEnemyBase_c (13)   Platform -> dBgActor_c (11)   ...

The 12 rows where the class name is unchanged (Player, Sound, Stage, Camera, …)
are parameter-type restamps — the class was already right, a RaycastLine argument
became dBgCh_Lin.

The one thing I want on the record: this swaps D1/D2, and that is correct

41 of the 334 rows are destructors, and several swap the destructor variant:

0x020112c8  _ZN5ActorD1Ev      -> _ZN8dActor_cD2Ev     <- D1 becomes D2
0x02011374  _ZN5ActorD2Ev      -> _ZN8dActor_cD1Ev     <- D2 becomes D1
0x02043d48  _ZN9ActorBaseD1Ev  -> _ZN7fBase_cD2Ev
0x02043dbc  _ZN9ActorBaseD2Ev  -> _ZN7fBase_cD1Ev
0x0200651c  _ZN8CapEnemyD2Ev   -> _ZN11dCapEnemy_cD1Ev

while others (dCcAc_c, dCc_c) keep D2→D2. So it is not a blanket convention flip,
which is what a mechanical find-and-replace bug would look like — and a D1/D2 swap in the
CapEnemy family specifically is a defect this tree has been bitten by before.

I checked each swapped address against config individually. Config already carries the
post-swap form at every one of them:

0x020112c8 -> _ZN8dActor_cD2Ev      0x02011374 -> _ZN8dActor_cD1Ev
0x02043d48 -> _ZN7fBase_cD2Ev       0x02043dbc -> _ZN7fBase_cD1Ev
0x0200651c -> _ZN11dCapEnemy_cD1Ev  0x020aedf4 -> _ZN11dCapEnemy_cD0Ev

The swap was settled on main some time ago; verified.tsv was simply still carrying the
pre-resolution spelling. The standing rule in this tree is that origin/main's naming
wins for dCapEnemy_c, and that is precisely what this PR adopts. Nothing to fix.

Blast radius

symbols/verified.tsv is read by cluster_targets.py, cpp_rename.py and
import_symbols.py. None of the three is a byte gate, and no address changed, so this
cannot move a verification verdict in either direction. What it does change is that 334
records now join against config by name as well as address, where before they silently
did not.

Stated gap

I did not run rombuild/eligible/romdata_check — shared build/ with other agents
in flight makes concurrent runs produce fake failures. I did not need to: the address set
is provably unchanged, so there is no byte claim here to audit. Merging as a merge commit.

One note for the reader, not a blocker

This PR adds tools/test_symrecords.py but touches no workflow file, and main's
tool-tests.yml does not enumerate it — so the new suite does not run in CI as of this
merge.
That is correct sequencing rather than an oversight: #2038 is the PR that wires
test_symrecords (along with test_match_provenance), and it is next in this lane. I am
landing #2039 first because it is CLEAN while #2038 carries a tool-tests.yml conflict;
serialising the clean one behind the conflicted one buys nothing.

Re-verified after main moved to 9f8b2f8c8 mid-review: still 334/334 against the new
config, merge tree still clean (141928b4f6e5a0b200276c0ed47dad0e1b423138).

@andrewboudreau
andrewboudreau merged commit a8577b5 into main Aug 30, 2026
4 checks passed
@andrewboudreau
andrewboudreau deleted the symbols/verified-restamp branch August 30, 2026 21:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants