Repository navigation
map PDB.open and place inline sites without holding a module's worth - #62
danielplohmann merged 3 commits into
Conversation
Merge plan for the queue, and one thing this stack carries forwardPosting here because this is the tip of the stack. Status as of 2026-09-15, after the release workflow landed (#59, #63). ShapeFour open PRs, but three are one stack —
GitHub shows #61 and #62 as +3606 and +4413 because those are cumulative diffs. The stack as a whole is +4413/−389 over 35 files. ConflictsAgainst The squash problemThe repo squash-merges. Squashing #60 collapses its 15 commits into one new commit, after which #61 still carries the 15 originals and they are no longer ancestors of
What this stack carries forwardThe inline-site rewrite here preserves an inconsistency from if len(raw) >= 4 and struct.unpack_from("<I", raw, 0)[0] == CV_SIGNATURE_C13:
return raw[4:end]
return raw[:end]while record offsets get #53's P3-5 is the fix. I verified it against the corpus and it is correct — 757682 proc records, For whoever cuts 0.6.0#60 states it moves the second-and-later ranges of every inline site built from the fused opcode. Also note all four PRs currently show 10 checks rather than 11: they predate the |
|
Thanks for mapping this out, that saved me a lot of guessing. Went ahead with the first part: #60 is rebased onto main (only CHANGELOG/AGENTS conflicted, as you said) and #61 is rebased on top of the new #60. Both are green locally: 627 tests, lint and fuzz clean. #62 still has a merge from main on it, so I'll rebase it properly once #60 and #61 land, same for #53 after that. Also put up #65 for #64. It strips C7/C11 the same as C13 and stops reading a module whose signature it doesn't know, with a diagnose() warning. Heads up that it touches the same lines as P3-5 in #53, so one of them will need a small rebase. P3-5 may end up a no-op if #65 goes in first. |
755fb07 to
81e51e8
Compare
…ation cursor, stripped files, and the large block sizes Fifteen commits from a corpus audit of 421 real PDBs (python.org 2.7-3.14 on x64/x86/arm64, node, Firefox's xul.pdb, Microsoft symbol-server files for Win7/10/11 and XP, self-built clang-cl/lld-link and rustc variants, 168 corrupted derivatives). Every commit was gated on the full suite, `ruff check`, `ty check` and a 2000-input fuzz pass; the corpus is rebuildable with the `tools/fetch_corpus.py` this adds. - **Inline sites on MSVC output.** MSVC writes `S_INLINESITE2` (48608 of the 48642 sites in python312.pdb); 0.5.0 saw 34. The length fused into `ChangeCodeLengthAndCodeOffset` does not move the cursor - the old rule put 5582 of 79187 ranges past their procedure or cold chunk, the corrected rule none - and `ChangeCodeOffsetBase n` names the procedure's n'th `S_SEPCODE` chunk. This moves the second-and-later ranges of every site built from the fused opcode, on rust-lld and clang output too. - **Stripped PDBs produced no `diagnose()` warning**, which is the shape of every Microsoft public symbol file. DBI Flags and BuildNumber are read, three warnings cover the stripped / unflagged / empty-module-list cases, and the CLI opens with a `linker` line. - **MSF block sizes 8192, 16384 and 32768** are accepted; link.exe writes them for PDBs past a few GB and they were refused outright. - **A directory whose stream sizes sum past the file is rejected.** Each block list passed the per-stream check while sizing a multi-gigabyte allocation from a 4 MB file. - `S_LPROC32_DPC`/`_DPC_ID` are procedures; `S_COMPILE2` feeds `compile_info()`; `DEBUG_S_LINES` BlockSize is floored at the bytes read. - **Unnamed inline sites are counted and explained**: VS2015 wrote compiler-internal ids its linker never remapped, 5802 of 6554 sites in a python 3.5 `_hashlib.pdb`, and a missing IPI stream was silent too. First of the queued audit stack: #61 and #62 build on this. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
open() maps the file by default so a 1.9 GB PDB is not copied into bytes; close() and the context manager own that mapping, and copy=True is the previous read. diagnose() and inline_sites() collect procs and S_SEPCODE first, then place each site as it is parsed, because MSVC writes the chunks after the procedure's scope and xul's largest modules carry millions of sites. A stripped PDB that still has procedure records (Win10/11 public files) now gets a Diagnostics warning rather than leaving the caller to read the flag. tools/fetch_corpus.py copies PUREPDB_EXTRA_PDBS if the caller has FASTLINK/_ST/managed files this box cannot produce. Co-authored-by: basak.rittick8274 <basak.rittick8274@gmail.com>
The corpus is on this box now, so the open/functions/diagnose figures that perf.md still had as "not re-run" are from this tree: mapped open is 264 MB against a 1.9 GB copy, and diagnose peaks at 4.4 GB instead of 7.0 GB because sites are not held for the module. Co-authored-by: basak.rittick8274 <basak.rittick8274@gmail.com>
AGENTS.md said 614 tests; this branch is 619. The README still opened without close, which now holds a mapping. _place_module_sites had no caller. RecordSurvey.malformed's total is only count_malformed_records when parse= is omitted. Also: Windows cannot replace a mapped PDB, rustc provenance comes from rustc --version, and the same-interpreter node numbers from the review go in perf.md. Co-authored-by: basak.rittick8274 <basak.rittick8274@gmail.com>
81e51e8 to
e17bc89
Compare
… unknown Fixes #64. `PDB.module_symbol_bytes` stripped the CodeView signature only when it was `CV_SIGNATURE_C13`. A C7 (`0`) or C11 (`1`) signature stayed on the front of the buffer, and `codeview.iter_records` read that word as a record header - for C11, length 1 and kind 0 - so every record after it was walked at the wrong alignment, with no error and no warning. - All three signatures are recognised and stripped, as `llvm-pdbutil`'s module stream reader does. - A first word that is none of the three yields no symbols: the layout after an unknown signature is not known, and reading the word as a record is what produced the silent misparse. `Diagnostics.unrecognised_signatures` counts those modules by the value found, and `diagnose()` warns with it. - A stream too short to hold a signature is unchanged: its bytes go to the record walk, which reports them as a truncation. Two tests, both verified to fail under a mutation of the line they cover: `test_every_codeview_signature_is_stripped` (C7 and C11 fail when only C13 is recognised) and `test_an_unrecognised_signature_is_not_parsed_as_a_record`, whose unknown word is chosen so that, read as a header, it is a 16-byte `S_GPROC32`. Every golden fixture is C13-signed, so no golden count moves. This also settles #53's P3-5. The strip is no longer conditional on any path that returns records - a body is either the stream minus exactly four bytes, or empty - so the fixed `+ CV_SIGNATURE_SIZE` compensation already matches it. The one path returning an unstripped non-empty body is a stream shorter than four bytes, which cannot hold a record header and so yields no offsets at all. Merged with main carrying #60, #61 and #62; 636 passed, ruff and ty clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Findings from the correctness audit, each one a case where a damaged or unusual
file produced an unhandled exception or a silent gap rather than a `PdbError`
and a diagnostic.
- **A missing PDB Info stream** no longer escapes. `PDB.named_streams()` is
guarded by `is_valid_stream` and catches `MsfError`, returning `{}` as
documented, so `diagnose()`, `string_table()` and `lines()` produce empty
results with a clear explanation instead of raising.
- **`PublicsStream.parse`** raises plain `PdbError` rather than `ValueError`
when the header or address map is truncated, preserving the invariant that
`PdbError` is the only exception crossing a public boundary. Plain `PdbError`
and not `MsfError`: the publics stream is not an MSF container.
- **DBI substream sizes are bounded.** A negative size would slice backwards,
aliasing bytes that were never substream data; a size overrunning the buffer
is a header that lies about its own layout. Both raise `MsfError` with
context, while damage *inside* a substream still degrades rather than raising.
- **C13 line-info truncations are reported.** `C13Truncation` records them,
`iter_subsections` and `parse_lines` collect them, and
`Diagnostics.c13_truncations` plus a warning surface them.
- **The fuzzer covers `publics_stream()`**, which was the one public listing it
never exercised.
- Dead `Reader.align` removed; docstrings added to `ContributionMap.__len__`
and `SectionContribution.contains`.
P3-5 (the conditional signature offset) is **not** in this PR. #65 made the
strip unconditional on every path that returns records, so the fixed
`+ CV_SIGNATURE_SIZE` already matches and the conditional helper had nothing
left to correct; `tests/test_inline.py` is identical to main as a result.
Last of the audit queue, after #60, #61, #62 and #65. 645 passed, ruff and ty
clean, fuzz clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Dev-group pin only; `ruff` governs the `lint and type check` CI job and `make lint`. No `.pre-commit-config.yaml` in this repository, so there is no `ruff-pre-commit` rev that has to move with it. Verified against this main rather than the one the PR was opened on: 0.16.6 reports `All checks passed!` over the whole tree, including the roughly 4400 lines that landed in #60, #61, #62, #65 and #53 after this PR was raised. `target-version` stays `py311`.
Dev-group pin only; `ty` governs the type half of the `lint and type check` CI job and `make lint`. Dependabot rebased this onto the ruff bump and re-resolved the target from 0.0.79 to 0.0.80 in the process, so it was re-verified at the version actually pinned here: 0.0.80 reports `All checks passed!` over the whole tree, including the roughly 4400 lines that landed in #60, #61, #62, #65 and #53 after this PR was raised. `ty` is still 0.0.x, so it is the pin in this group most likely to surface a new diagnostic on a bump.
Both version strings to 0.6.0, `## [Unreleased]` renamed to `## [0.6.0] - 2026-09-15` with a fresh empty one above it, and the compare links moved on. No source changes. First release cut through `publish-release.yml`, which has existed since #59 without a tag ever exercising it. 23 entries from five merged PRs plus the release machinery and three dependency bumps: the corpus-audit fixes (#60), the single-pass record walk (#61), the mapped `PDB.open` and streamed inline-site placement (#62), the C7/C11 signature handling (#65) and the parse-boundary hardening (#53). Minor rather than patch: the `Fixed` entry for the fused `ChangeCodeLengthAndCodeOffset` annotation says **This moves addresses** -- the second and later ranges of an inline site built from that opcode now start earlier than 0.5.0 reported, on every rust-lld and clang output. The first range, and every single-range site, are unchanged. `CHANGELOG.md`'s preamble reserves that for a breaking change and requires it to be stated. Verified before tagging: `release_guard.py --tag v0.6.0` extracts the section and reports prerelease=false, and refuses v0.7.0 naming both version strings; 645 passed, ruff and ty clean on the new pins; `python -m build` plus `twine check --strict` pass on both artefacts; and the wheel, installed into a clean venv and imported with `python -P`, reports 0.6.0 from site-packages with `py.typed` present and runs the entry point. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Depends on #61, which depends on #60. Land those first; until then this diff also shows their commits.
What
PDB.openmaps the file instead of reading it intobytes. The object owns the mapping (close(), context manager).copy=Trueis the old behaviour.from_bytesis unchanged. Untilclose(), Windows cannot replace or delete the PDB.diagnose()/inline_sites()no longer hold a module's decoded sites. Procs andS_SEPCODEchunks are collected first (MSVC writes those after the procedure's scope), then each site is placed as it is parsed. Listing output is unchanged except for the new stripped-with-procs warning.Diagnostics.warningssentence.tools/fetch_corpus.pycopiesPUREPDB_EXTRA_PDBSintocorpus/extra/for FASTLINK /_ST/ managed files that cannot be produced locally. Rust provenance isrustc --version, not a pinned 1.94.1.Also
AGENTS.md 619; README uses
with PDB.open;_place_module_sitesdeleted;RecordSurvey.malformedqualified forparse=.Measured
Corpus snapshot vs #61 on 432 files under 50 MB plus fixtures: 12 diffs, all six Microsoft stripped files (each stored twice), solely the new warning. Inline-site rewrite is byte-identical on real MSVC/PGO output.
xul.pdb(CPython 3.12.3), open + functions + diagnose:Same-interpreter vs #61 (CPython 3.11.15, warm cache),
node.pdb: diagnose 21.92 → 12.06 s, inline_sites 20.81 → 16.09 s, open RSS 425 → 101 MB.ruff checkandty checkclean; 619 tests passed; fuzz 2000 / seed 0 on the code commit.