Repository navigation
feat(common): let the exception directory refuse an interior candidate too - #329
Conversation
|
Blocking, and it is an interaction with #333 rather than anything wrong in the measurement. #333 merged an hour ago as Five of the eight pinned addresses stop appearing in The cause
The shared helper has one flag name for both, so on an ARM64 PE the analysis-time refusal answers to the x64 flag. Measured on
So the ARM64 flag no longer turns the ARM64 rule off, and the x64 flag would turn it off on an image it has nothing to do with. Default output on that fixture does not move — with both flags on the gap scan reaches all eight first, which is why 0 refusals — so this is flag semantics plus a broken test, not a behaviour regression. It is also why your corpus table could not see it: there is no ARM64 PE row in it, and at defaults there is nothing to see. The fix I would takeA backend hook rather than a flag name in shared code, mirroring the split the two gap scans already have: # common/FunctionCandidateManager.py
def _pdataInteriorRefusalEnabled(self):
"""Whether this backend's exception-directory interior rule is on. See the two overrides."""
return Falsewith I applied that here to check it: It does cost your PE unit cases, and in a way worth knowing before you write it. Worth a case pinning the architecture split as well: the ARM64 flag off, over the fixture #333 bundles, should produce #333's eight and nothing else — which is exactly the assertion that just failed. The rest of the review, which standsThe fragment decision is right and it is the interesting difference from the gap scan: a fragment's own start is not the function covering the address, so taking the gap scan's shortcut here would be using the weaker evidence at the point the stronger one is available. Declining rather than refusing is the conservative direction. The format-disjointness argument holds as written — but note the code makes it load-bearing in a way the gap scan does not: an Rebase over |
|
Reproduced and fixed, and your diagnosis is exactly right — Took your design. One correction to the note, and it was in my commit message too
That does not change the design — the base still has to answer, because the bare common class is what a unit helper builds and what any manager added later inherits — but the reason is "this is the answer for a manager that carves no directory", not "these two backends rely on it". Stated that way in the docstring, the test docstring and the commit body now. The unit casesExactly as you measured: left alone,
Verified by mutation — each of these fails the intended case and nothing else:
The format-disjointness noteWritten into the docstring, since you are right that it is now structural rather than incidental:
Left as an assumption rather than restructured — an arm that re-tries on a qualifying owner rather than any owner would be code for a case no image produces. SweepFor the class behind this — shared code gating on one backend's flag where a sibling reaches the same path — the whole tree has one other hit, VerificationFull suite on this tip: 2,136 passed, 1 skipped, 2,596 subtests. Master is merged in rather than rebased on both this and #327 — I would rather not force-push over branches you have already reviewed, so taking the rebase at merge is yours. #327 carries the docstring correction from your review there, and this still stacks on it. |
7b47997 to
7cad9ff
Compare
|
Rebased over Once #327 merges this reduces to its own two commits. |
|
The backend hook is the fix, and You went further than the fix in the one place it mattered. Making the PE cases build the real backend rather than a bare common manager is what stops five of the seven passing vacuously — they were getting their
The widened Verified stacked over No further review notes. One thing before it merges: with #336 and #337 in, this needs its own |
959a471 to
48ae03a
Compare
|
Two things: a merge of master is needed, and there is an entry missing. The merge is mine, not yoursYou based this on Resolved and verified locally: 2155 passed, 2 skipped, 2596 subtests, The entry for this PR's actual change is missingThe So it needs its own bullet under Worth saying plainly because it is the exact failure the format was adopted to prevent, and it has appeared on the first engine PR after it landed: at release time a missing entry is indistinguishable from a change that needed none, and only you can tell the difference while the PR is open. The The figures are already written in #328, so this is transcription rather than measurement. Once it is in and master is merged, this is ready and v4.7.0 is next. |
|
#327 is merged as Your branch still carries #327's three commits as their original objects ( Two things then, and the second is the one that matters:
The shape to match is what #327 just landed with: a Everything else is verified and stands — I ran the merged tree at This is the last thing before v4.7.0. It closes #328 when it lands. |
|
Both done, and the missing entry turned up something in the config comment. The mergeMerged rather than rebased — the duplicate-add conflicts resolve without rewriting history, since my side of each block is That check earned its keep. My earlier "keep both sides" resolution had silently duplicated The missing entryYou are right that It did not. Every figure reproduces exactly:
One correction, and it is to a line you quoted approvinglyThe config comment said "none of the 57 declares an exception directory at all, so this evidence does not reach a mapped image." Three of them do — The conclusion survives; the reason was wrong, and the truth is the better control. Instrumented on all three, the rule is consulted and refuses nothing — every address its guards see is declined. Reachable-and-declining says the guards hold on mapped dumps; an absence says only that the corpus never asked. The comment now says that instead. Suite on the merged tree 2,156 passed, 1 skipped, 2,596 subtests; |
…e too Third of the three rules that read declared evidence only at the gap pointer, and the last one with anything left to reach. danielplohmann#327 widened the .eh_frame arm; this does the same for the PE exception directory, under the guards that arm established: the record's own function has to be recovered, and its recovered extent has to surround the address. A fragment record is not the shortcut it is in the gap scan. Its own start is not the function covering the address, so it fails the recovered-owner test and declines rather than refusing, which is the conservative reading at a point where a better one is available. It is worth much less than the .eh_frame arm, which is why it is a change of its own rather than part of danielplohmann#327: 120 MinGW PE x64 cells -47 FP at identical TP and FN 2 Rust windows-gnu-x64 -2 FP at identical TP and FN Nothing else moves. The 140 x86-64 ELF cells, 72 AArch64 ELF cells, 23 Go cells, 11 ARM64 Mach-O cells, the 32-bit and ELF Rust cells and all 57 malpedia dumps are bit-identical, and only PE x64 cells differ in any corpus. The dumps are the reading worth keeping: none of the 57 declares an exception directory at all, so this evidence does not reach a mapped image, which is the corpus this project cares most about.
Both native backends fill _pdata_ranges -- intel from an x64 exception directory, aarch64 from an ARM64 PE's -- and each gap scan gates on its own architecture's flag. The shared analysis-time rule named only USE_PE_X64_PDATA_INTERIOR_GAPS, so on an ARM64 PE the refusal answered to a switch for a format it never sees: USE_PE_ARM64_PDATA_INTERIOR_GAPS no longer turned the ARM64 rule off, and the x64 one would. Default output does not move -- with both flags on, the gap scan reaches those addresses first -- but testArm64PeInteriorGapFixture fails, because five of its eight pinned addresses stop appearing with the ARM64 flag off. Ask a backend hook instead, mirroring the split the two gap scans already have. Those two managers are the only ones this engine builds, so the inherited answer is what one carving no directory should give. The unit helper builds a bare common manager, which now inherits that no, so the PE cases build the backend that fills the ranges; left as it was, two of them would fail and the other five would pass vacuously, getting their None from the gate rather than from the condition each one is about. An ARM64 subclass of the same cases pins the split from the other side. Also widen the summary line of declaredInteriorOwner, which still named only .eh_frame after this rule learned to read an exception directory.
Fixed rather than Changed, and it says so: default output does not move, because with both flags on the gap scan reaches those addresses first. What moved is which switch controls the rule.
The Unreleased section carried only the backend-hook fix, which is the flag-semantics correction and moves no default output. The change this PR is named after -- a PE exception directory refusing an interior candidate at analysis time -- had no Changed bullet and no Compatibility line, which is the entry a consumer holding stored reports actually needs. Re-measured on fb510c4 rather than carried forward, since both danielplohmann#327 and danielplohmann#339 have landed under this since the figures were taken: 120 MinGW PE x64 cells 93.191 -> 93.235 PPV at -47 false positives and 109/120 bit-identical, 2 Rust windows-gnu-x64 cells at -2, neither losing a true positive, and the 140 C/C++ ELF cells and all 57 malpedia dumps bit-identical. Every figure reproduces. Also correct the malpedia reading in the config comment. It claimed none of the 57 dumps declares an exception directory, so the evidence could not reach a mapped image. Three of them do, carrying 282 to 430 records, and the rule is consulted on all three and refuses nothing -- its guards decline every address they see. The conclusion stands and the reason was wrong, and reachable-and-declining is the stronger control of the two: an absence shows nothing about whether the rule would have been safe.
Four times today the question 'what is left before v4.7.0' was answered by re-reading the issue list and the PR list and working out which of them mattered. The scope was spread across an issue acting as an ad-hoc tracker (#322), the handover sections of out-of-repo session notes, and PR bodies. None of those is queryable and the first two are maintained by hand. The Unreleased section #337 introduced covers the other half well -- it is the record of what landed and what it cost -- but it is silent on what has not landed, which is the half the question is actually about. A milestone per release is the thin version of the missing piece: one bucket holding issues and PRs, a closed/total count, and a filter. It does not order work and nothing assigns it automatically, so a PR joins when it is accepted for the release rather than when it is opened -- otherwise it accumulates into a wishlist and the count stops meaning anything. v4.7.0 exists and carries #328 and #329. No CHANGELOG entry: this changes how the repository is organised and nothing a consumer of the package can observe. Raised in the PR, since it is the second change today where 'every PR writes a bullet' meets a PR with nothing user-facing to say. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
8245f63 to
024f4bc
Compare
|
Rebased onto Zero merge commits, and the tree is unchanged by the rebase — Two conflicts came up in the replay rather than disappearing, both additive and both because the commits predate what landed under them: The diff against master is the six files this PR owns and nothing else. |
|
Merged as The backend hook is the right shape and it clears what I blocked on: Verified on saturn against master at Two notes, neither blocking:
This closes #328 and empties the v4.7.0 milestone. Cutting the release next, by hand - #343 is not in this milestone's scope, so it is not gating it. I will come to #343 after the tag is out. |
Closes #328. Stacked on #327 — it extends the helper that PR introduces, so it must merge after it. The diff shown against
masterwill include #327's commit until then; this PR's own commit is2efa621.Third of the three rules named in the #324 sweep, and the last one with anything left to reach. #327 widened the
.eh_framearm to a candidate from any source; this does the same for the PE exception directory, under the guards that arm established.The rule
The same two conditions, unchanged: the
RUNTIME_FUNCTIONrecord's own function has to be recovered, and its recovered extent has to surround the address. The two evidence sources stay format-disjoint exactly as the gap scan's comment already notes —_pdata_rangesis only ever filled from a PE exception directory and the.eh_frameranges decode nothing unless lief reports an ELF — so no address is ever arbitrated between them.One deliberate difference from the gap scan. There, a fragment record is evidence on its own: it says the extent is part of some other function, and the scan needs nothing more. Here it is not taken as that. A fragment's own start is not the function covering the address, so it fails the recovered-owner test and declines rather than refusing. At a point where the better evidence — an actual recovered owner — is available, using the weaker form would be the wrong reading. No fragment record appeared among the interior false positives in the sample either way.
Measured
feat/fde-interior-analysis-gate(#327) → this branch, so the figures isolate this arm.windows-gnu-x64No corpus loses a true positive, and only PE x64 cells differ anywhere — which is the control this arm has to pass, the mirror of #327's ELF-only one.
The part worth reading before merging
On memory dumps this evidence does not exist. None of the 57 malpedia dumps declares an exception directory at all —
_pdata_rangesis empty, so the rule never reaches its first condition. That is the corpus SMDA cares most about, and it is the honest limit of this change.Between that and 47 false positives on one PE corpus, this is a small win, and #328 says so rather than selling it. It is here because it is free of recall risk under guards that are already measured, and because leaving one of three siblings unwidened without a reason is worse than either doing it or declining it on the record. If you would rather close #328 and keep the surface smaller, that is a reasonable call and the numbers are the argument either way.
One thing I had wrong
I filed #328 saying 14 of 1,038 on a 20-cell sample, and scaled that to roughly 80 across 120 cells. The real figure is 47. The sample over-predicted; the table above is the measurement.
I also assumed the Rust corpus was ELF, which is why the two moving cells looked like an anomaly at first. They are
windows-gnu-x64— PE x64, exactly where this applies. The corpus is mixed, and every one of its ELF and 32-bit cells is bit-identical.Tests
Seven cases in
tests/testFdeInteriorGaps.py::DeclaredInteriorOwnerPeTest, one condition each, exercising the arm through a stubbed exception directory so the ELF and PE arms are tested apart. Each was checked to fail when the guard it describes is removed: dropping the arm fails two, dropping its flag check fails one.python -m pytest tests/→ 2,076 passed, 1 skipped, 2,593 subtestsruff check .andruff format --check .→ cleanmake typecheck→ exit 0, 0 error-level diagnostics, no new diagnosticsNo bundled fixture baseline moves.
Fixed after review
The shared rule answered to the wrong architecture's flag. Both native backends fill
_pdata_ranges—intelfrom an x64 exception directory,aarch64from an ARM64 PE's — and each gap scan gates on its own flag, butdeclaredInteriorOwnernamed onlyUSE_PE_X64_PDATA_INTERIOR_GAPS. On an ARM64 PE that leftUSE_PE_ARM64_PDATA_INTERIOR_GAPSunable to turn the ARM64 rule off, and the x64 flag able to. Default output does not move — with both flags on the gap scan reaches those addresses first — buttestArm64PeInteriorGapFixturefails, five of its eight pinned addresses stopping short ofoff - on.Each backend now answers for itself through a
_pdataInteriorRefusalEnabled()hook, mirroring the split the two gap scans already have. The base returns the disabled answer, which is what a manager carving no exception directory should give.The PE unit cases had to move with it.
declaredOwnerOfbuilt a bare common manager, which now inherits that answer: two of the seven cases would fail outright and the other five would pass vacuously, taking theirNonefrom the gate rather than from the condition each is about. The helper takes amanager_classand the PE class builds the backend that fills the ranges.DeclaredInteriorOwnerArm64PeTestasks all seven of the same conditions ofaarch64's manager, each class asserting that the other architecture's flag does not gate it, andDeclaredInteriorOwnerUnbackedPeTestpins the inherited answer so a manager added without an override cannot start refusing on extents it never filled.Four mutations verified, each caught by the intended case and nothing else: either backend's override answering to the other's flag, the shared gate naming a flag directly, and the base returning
True.Also documented: disjointness between the two structures is load-bearing here in a way it is not in the gap scan — a declared FDE range whose start was never recovered sets
ownerand then fails the recovered-owner test, so on an image carrying both the exception directory would never be asked about that address.Full suite on the current tip: 2,136 passed, 1 skipped, 2,596 subtests,
ruffclean,make typecheckexit 0.