Skip to content

harden parse boundary, C13 diagnostics, and DBI/GSI error handling - #53

Merged
danielplohmann merged 13 commits into
danielplohmann:mainfrom
r0ny123:fix-correctness-audit
Sep 15, 2026
Merged

danielplohmann merged 13 commits into
danielplohmann:mainfrom
r0ny123:fix-correctness-audit

Conversation

@r0ny123

@r0ny123 r0ny123 commented Sep 3, 2026 •

Copy link
Copy Markdown
Contributor

This PR addresses the findings from the correctness audit across purepdb:

Summary of Changes

  • P1 — Missing PDB Info stream handling: Guarded PDB.named_streams() with self.msf.is_valid_stream(STREAM_PDB_INFO) and caught MsfError to return {} as documented, ensuring diagnose(), string_table(), and lines() produce empty results and clear diagnostic explanations rather than raising an unhandled exception.
  • P3-1 — PublicsStream.parse exception surface: Replaced ValueError with plain PdbError in PublicsStream.parse() when header or address map is truncated, preserving the invariant that PdbError is the only exception crossing public boundaries. Plain PdbError rather than MsfError: the publics stream is not an MSF container.
  • P3-2 — DBI substream size validation: Added checks in DbiStream.parse() that each substream size is non-negative and does not overrun the stream buffer, raising MsfError with descriptive context if corrupted.
  • P3-3 — C13 line-info truncation diagnostics: Added C13Truncation dataclass in purepdb.c13 (exported in __all__), collected truncations in iter_subsections and parse_lines, surfaced Diagnostics.c13_truncations, and added an explanatory warning to Diagnostics.warnings.
  • P3-4 — Fuzzer coverage for publics_stream: Added pdb.publics_stream() to exercise() in tools/fuzz.py.
  • P3-5 — Inline sites signature offset: dropped. Once strip a C7 or C11 module-stream signature, and read nothing after one unknown #65 landed, every path that returns module records strips exactly four bytes, so the fixed CV_SIGNATURE_SIZE already matches and the conditional helper had nothing left to correct.
  • Nits: Removed unused Reader.align dead code in purepdb/reader.py, and added docstrings to ContributionMap.__len__ and SectionContribution.contains in purepdb/dbi.py.
  • Changelog: Updated CHANGELOG.md under ## [Unreleased].

Design decisions (deliberate, and written down in the code)

  • A DBI header that lies about the bounds raises; in-substream damage degrades. The substream bound checks in DbiStream.parse() raise MsfError for a negative size (which would slice backwards, aliasing bytes that were never substream data) and for a size past the stream end (the header's arithmetic disagreeing with the container). The graceful paths — module_list_stopped_at, codeview.Truncation — still cover damage inside substreams whose bounds the header describes correctly. The rationale lives in a comment at _check_substream, and a test cuts a valid DBI stream short to pin the choice.
  • diagnose() re-decodes every DEBUG_S_LINES payload and discards the entries. Measured at 18 ms of the 180 ms diagnose() costs on the 3 MB sqlite fixture. Deliberate, for the same reason count_malformed_records runs twice per module: parse_lines is the only implementation of the block walk, and a header-only twin kept beside it would drift. lines() cannot be reused because it is a generator a caller may never consume, and it resolves file names this loop does not want. The reason and the number live in a comment at the call site.

Skipped Findings

  • None; all audit items were resolved.

Gate Verification

  • Tests: make test PYTHON=.venv/bin/python
    596 passed
    
    All groundtruth baselines in tests/test_groundtruth.py remain unchanged.
  • Lint: make lint PYTHON=.venv/bin/python
    All checks passed! (ruff check)
    All checks passed! (ty check)
    
  • Fuzz: make fuzz PYTHON=.venv/bin/python
    ok: 2000 inputs, seed 0, no exception escaped PdbError/MsfError/RecursionError
    

r0ny123 and others added 9 commits September 3, 2026 17:56
…s stream

Review follow-up on the audit fixes. PublicsStream.parse now raises plain
PdbError rather than MsfError -- the publics stream is not an MSF container,
and the exception should not claim to be one. The DBI substream bound checks
carry the reason a header that lies about the bounds raises outright while
in-substream damage degrades gracefully, pinned by a test that cuts a DBI
stream short. The C13 re-decode in diagnose() carries its measured cost
(18ms of 180ms on the 3 MB sqlite fixture) and why a header-only twin of the
block walk was rejected.
… padding as padding

Two of the checks this branch added were sharper than the contract
allows. A substream size past the end of the DBI stream raised MsfError
out of PDB.open(), which turned a file missing the last eight bytes of
its debug header -- every function recoverable -- into one that would
not open, with no diagnostic to say why; the negative size still raises,
because the next offset then goes backwards and every later slice
aliases bytes that were never substream data, but the overrun is now
read as far as the stream goes, recorded in DbiStream.substream_overrun,
and reported by diagnose() as Diagnostics.dbi_overrun with a warning
naming the substream. And a C13 region ending in a few zero bytes -- a
producer rounding it up -- was reported as lines going missing when none
did; fewer than eight bytes cannot be a header, and only bytes with
something in them are evidence of a cut.

The inline-site coordinate change is not observable on a well-formed
stream, and its test now says so and adds the one input where the two
spaces disagree, a damaged End two bytes past the site. The CLI's
diagnose gets a test for the C13 warning it prints.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017jQKALSgFN7VecWTqzWSmD
r0ny123 added a commit to r0ny123/purepdb that referenced this pull request Sep 12, 2026
…anielplohmann#59 and the dependabot bumps

Fill in the fuzz and timing results, note the follow-up branches, and
summarise the review in the log.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017jQKALSgFN7VecWTqzWSmD
@danielplohmann

Copy link
Copy Markdown
Owner

P3-5 holds up, and it should not block the stack

Checked this while sequencing the queue against #60/#61/#62, because the stack rewrites the same code. Two separate answers.

The fix is correct

module_symbol_bytes strips the signature conditionally, so iter_records offsets are body-relative. They are then compared against proc.end, whose docstring in codeview.py says what that field is measured from:

Byte offset of the record that closes this procedure's scope, in the module stream as stored — signature included, like S_PROCREF's.

So the compensation has to match the strip: 4 when a signature was removed, 0 when it was not. A conditional strip paired with an unconditional + CV_SIGNATURE_SIZE is inconsistent on its face, and _module_symbol_bytes_and_offset is the right shape for it.

Confirmed against the 46-file bag (39 readable, 23478 module symbol streams). For every proc record I tested both readings of pEnd against where scope-closing records actually sit:

=== what pEnd points at ===
   757682  stream-relative only (signature counted)

757682 procs, no ambiguous cases, no misses. The docstring is right.

But the branch it guards never fires on real input

=== module stream signature shapes ===
    23478  0x4 (C13, stripped)

modules NOT stripped that still carried records: 0

Every module stream in the corpus is C13-signed. And when the unstripped path is taken, the body it returns begins with a non-C13 signature word that iter_records then parses as a record — so the offsets are the smaller of that path's problems. This makes a broken path consistent rather than correct.

That reads as a fuzzer-facing robustness fix, which fits: tools/fuzz.py mutates exactly that word, and this PR extends its coverage. Worth keeping, not worth blocking on.

Sequencing

#60 ⊂ #61 ⊂ #62 is one stack; this PR is independent and the only one with real source overlap. Test-merging it onto #62 conflicts in c13.py, pdb.py, test_diagnose.py and test_inline.py.

Nothing here is superseded — C13Truncation, c13_truncations and _module_symbol_bytes_and_offset appear nowhere in the stack. But #62 rewrote the inline-site placement and now carries the unconditional add at three sites (pdb.py:1059, :1093, :1408) instead of two. So P3-5 lands as a rewrite against that shape, not a merge.

Suggested order: #60 → #61 → #62 → this. Details and the conflict inventory in the plan on #62.

Adjacent, not covered by this PR

A C7 (0) or C11 (1) signature is neither stripped nor recognised, leaving the signature word to be parsed as a record. Filed separately — it is a different bug that happens to live on the same line.

# Conflicts:
#	CHANGELOG.md
#	purepdb/c13.py
#	purepdb/pdb.py
#	tests/test_diagnose.py
#	tests/test_inline.py
danielplohmann pushed a commit that referenced this pull request Sep 15, 2026
… unknown

Fixes #64.

`PDB.module_symbol_bytes` stripped the CodeView signature only when it was
`CV_SIGNATURE_C13`. A C7 (`0`) or C11 (`1`) signature stayed on the front of the
buffer, and `codeview.iter_records` read that word as a record header - for C11,
length 1 and kind 0 - so every record after it was walked at the wrong
alignment, with no error and no warning.

- All three signatures are recognised and stripped, as `llvm-pdbutil`'s module
  stream reader does.
- A first word that is none of the three yields no symbols: the layout after an
  unknown signature is not known, and reading the word as a record is what
  produced the silent misparse. `Diagnostics.unrecognised_signatures` counts
  those modules by the value found, and `diagnose()` warns with it.
- A stream too short to hold a signature is unchanged: its bytes go to the
  record walk, which reports them as a truncation.

Two tests, both verified to fail under a mutation of the line they cover:
`test_every_codeview_signature_is_stripped` (C7 and C11 fail when only C13 is
recognised) and `test_an_unrecognised_signature_is_not_parsed_as_a_record`,
whose unknown word is chosen so that, read as a header, it is a 16-byte
`S_GPROC32`. Every golden fixture is C13-signed, so no golden count moves.

This also settles #53's P3-5. The strip is no longer conditional on any path
that returns records - a body is either the stream minus exactly four bytes, or
empty - so the fixed `+ CV_SIGNATURE_SIZE` compensation already matches it. The
one path returning an unstripped non-empty body is a stream shorter than four
bytes, which cannot hold a record header and so yields no offsets at all.

Merged with main carrying #60, #61 and #62; 636 passed, ruff and ty clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@danielplohmann

Copy link
Copy Markdown
Owner

Status, since we crossed in flight: #60 (cf5e4ac), #61 (840d367), #62 (8a949f8) and #65 (5ad39e3) are all on main now, in that order. #52 is closed as superseded by #61. This is the last of the queue.

Your 19:20 push picked up the stack but not #65, which landed a couple of minutes after it, so the pdb.py conflict you're about to hit is P3-5 against #65's _module_symbols.

P3-5 is now a no-op, as you predicted

Your call on #62 was right, and it holds on the merged code. After #65, _module_symbols returns a non-empty body on exactly two paths: the recognised-signature path, which strips exactly four bytes, and len(raw) < 4, which returns 1-3 bytes unstripped. A CodeView header is u16 length + u16 kind, so the short path cannot produce a record - checked directly against the merged parser, bodies of length 0 through 5 all yield zero records. Every path that yields records has stripped exactly four, so the fixed + CV_SIGNATURE_SIZE already matches and there is nothing left for a conditional offset to correct.

So the resolution is to drop P3-5 rather than port it: delete _module_symbol_bytes_and_offset, keep #65's _module_symbols, and put the two call sites at pdb.py:1060 and :1446 back on module_symbol_bytes with the unconditional CV_SIGNATURE_SIZE. That should also take tests/test_inline.py out of the conflict set.

For what it is worth, the corpus said the same before #65 landed: across 23478 module streams every one is C13-signed, so the branch P3-5 guarded never fired on real input. It was correct and unreachable; #65 made it unnecessary.

What is left conflicting

AGENTS.md, CHANGELOG.md, purepdb/pdb.py, tests/test_diagnose.py. The first is the clean-tree test count, now 636 on main. The rest of #53 - the PDB Info guard, the PublicsStream.parse exception surface, the DBI substream bounds, C13Truncation, the fuzzer's publics_stream coverage - is untouched by any of this and none of it appears anywhere in the four that landed.

Happy to do the rebase myself if you would rather hand it over, but you are clearly mid-flight on it, so I will keep off the branch unless you say otherwise.

With the C7/C11 signature change on main, every path that returns
module records has stripped exactly four bytes, so the conditional
signature offset has nothing left to correct. _module_symbol_bytes_and_offset
and its test are removed, and the inline-site helpers keep main's fixed
CV_SIGNATURE_SIZE. The rest of the branch is unchanged.
@r0ny123

r0ny123 commented Sep 15, 2026

Copy link
Copy Markdown
Contributor Author

Merged main in and dropped P3-5 like you said: _module_symbol_bytes_and_offset and its test are gone, the inline helpers and diagnose() are back on module_symbol_bytes with the fixed offset, and test_inline.py is identical to main now. The rest of the branch is unchanged. 645 tests, lint and fuzz clean. I also trimmed a comment in diagnose() that still talked about count_malformed_records running twice per module, which stopped being true after #61. Thanks for keeping off the branch, it's all yours to review.

@danielplohmann
danielplohmann merged commit 918d114 into danielplohmann:main Sep 15, 2026
11 checks passed
danielplohmann pushed a commit that referenced this pull request Sep 15, 2026
Dev-group pin only; `ruff` governs the `lint and type check` CI job and
`make lint`. No `.pre-commit-config.yaml` in this repository, so there is no
`ruff-pre-commit` rev that has to move with it.

Verified against this main rather than the one the PR was opened on: 0.16.6
reports `All checks passed!` over the whole tree, including the roughly 4400
lines that landed in #60, #61, #62, #65 and #53 after this PR was raised.
`target-version` stays `py311`.
danielplohmann pushed a commit that referenced this pull request Sep 15, 2026
Dev-group pin only; `ty` governs the type half of the `lint and type check` CI
job and `make lint`.

Dependabot rebased this onto the ruff bump and re-resolved the target from
0.0.79 to 0.0.80 in the process, so it was re-verified at the version actually
pinned here: 0.0.80 reports `All checks passed!` over the whole tree, including
the roughly 4400 lines that landed in #60, #61, #62, #65 and #53 after this PR
was raised.

`ty` is still 0.0.x, so it is the pin in this group most likely to surface a new
diagnostic on a bump.
danielplohmann added a commit that referenced this pull request Sep 15, 2026
Both version strings to 0.6.0, `## [Unreleased]` renamed to
`## [0.6.0] - 2026-09-15` with a fresh empty one above it, and the compare links
moved on. No source changes.

First release cut through `publish-release.yml`, which has existed since #59
without a tag ever exercising it.

23 entries from five merged PRs plus the release machinery and three dependency
bumps: the corpus-audit fixes (#60), the single-pass record walk (#61), the
mapped `PDB.open` and streamed inline-site placement (#62), the C7/C11 signature
handling (#65) and the parse-boundary hardening (#53).

Minor rather than patch: the `Fixed` entry for the fused
`ChangeCodeLengthAndCodeOffset` annotation says **This moves addresses** -- the
second and later ranges of an inline site built from that opcode now start
earlier than 0.5.0 reported, on every rust-lld and clang output. The first
range, and every single-range site, are unchanged. `CHANGELOG.md`'s preamble
reserves that for a breaking change and requires it to be stated.

Verified before tagging: `release_guard.py --tag v0.6.0` extracts the section
and reports prerelease=false, and refuses v0.7.0 naming both version strings;
645 passed, ruff and ty clean on the new pins; `python -m build` plus
`twine check --strict` pass on both artefacts; and the wheel, installed into a
clean venv and imported with `python -P`, reports 0.6.0 from site-packages with
`py.typed` present and runs the entry point.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@r0ny123
r0ny123 deleted the fix-correctness-audit branch September 30, 2026 18:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants