Skip to content

map PDB.open and place inline sites without holding a module's worth - #62

Merged
danielplohmann merged 3 commits into
danielplohmann:mainfrom
r0ny123:cursor/xul-memory-mmap-stream-cee3
Sep 15, 2026
Merged

danielplohmann merged 3 commits into
danielplohmann:mainfrom
r0ny123:cursor/xul-memory-mmap-stream-cee3

Conversation

@r0ny123

@r0ny123 r0ny123 commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

Depends on #61, which depends on #60. Land those first; until then this diff also shows their commits.

What

  • PDB.open maps the file instead of reading it into bytes. The object owns the mapping (close(), context manager). copy=True is the old behaviour. from_bytes is unchanged. Until close(), Windows cannot replace or delete the PDB.
  • diagnose() / inline_sites() no longer hold a module's decoded sites. Procs and S_SEPCODE chunks are collected first (MSVC writes those after the procedure's scope), then each site is placed as it is parsed. Listing output is unchanged except for the new stripped-with-procs warning.
  • Stripped-but-proc-bearing files (Win10/11 public symbol PDBs) get a Diagnostics.warnings sentence.
  • tools/fetch_corpus.py copies PUREPDB_EXTRA_PDBS into corpus/extra/ for FASTLINK / _ST / managed files that cannot be produced locally. Rust provenance is rustc --version, not a pinned 1.94.1.

Also

AGENTS.md 619; README uses with PDB.open; _place_module_sites deleted; RecordSurvey.malformed qualified for parse=.

Measured

Corpus snapshot vs #61 on 432 files under 50 MB plus fixtures: 12 diffs, all six Microsoft stripped files (each stored twice), solely the new warning. Inline-site rewrite is byte-identical on real MSVC/PGO output.

xul.pdb (CPython 3.12.3), open + functions + diagnose:

op mmap copy=True
open 0.77 s, 264 MB 2.63 s, 2086 MB
functions (265345) 12.45 s, 3207 MB 11.21 s, 4282 MB
diagnose 71.67 s, 4362 MB 70.67 s, 4925 MB

Same-interpreter vs #61 (CPython 3.11.15, warm cache), node.pdb: diagnose 21.92 → 12.06 s, inline_sites 20.81 → 16.09 s, open RSS 425 → 101 MB.

  • ruff check and ty check clean; 619 tests passed; fuzz 2000 / seed 0 on the code commit.

@danielplohmann

Copy link
Copy Markdown
Owner

Merge plan for the queue, and one thing this stack carries forward

Posting here because this is the tip of the stack. Status as of 2026-09-15, after the release workflow landed (#59, #63).

Shape

Four open PRs, but three are one stack — #60 ⊂ #61 ⊂ #62, confirmed by ancestry and stated in the bodies. #53 is independent. All four branch from e978f3a, before the release workflow.

PR own commits own diff what it is
#60 15 +2597/−164 corpus-audit fixes: S_INLINESITE2, separated code, stripped PDBs, MSF block sizes 8192–32768
#61 +11 +1082/−280 perf: single-pass stream walks, one struct per record
#62 +3 +930/−141 PDB.open maps the file; inline sites placed without holding a module's worth
#53 9 +436/−29 correctness hardening: parse boundary, C13 diagnostics, DBI/GSI errors

GitHub shows #61 and #62 as +3606 and +4413 because those are cumulative diffs. The stack as a whole is +4413/−389 over 35 files.

Conflicts

Against main, all four collide in CHANGELOG.md and AGENTS.md only, and mechanically: main gained an ## [Unreleased] block from #59, and #59 rewrote the same AGENTS.md rows these PRs edit — the workflow list (release.yml is now publish-release.yml), the make publish row, and the clean-tree test count. No source conflicts with main at all.

The squash problem

The repo squash-merges. Squashing #60 collapses its 15 commits into one new commit, after which #61 still carries the 15 originals and they are no longer ancestors of main. Each layer needs a rebase onto the new main before it merges:

  1. read what MSVC writes: S_INLINESITE2, separated code, the fused annotation cursor, stripped files, and the large block sizes #60 — rebase, resolve the two doc files, squash-merge
  2. walk each stream once, decode each record with one struct: 3–8x on every listing, output unchanged #61 — rebase onto the new main (drops read what MSVC writes: S_INLINESITE2, separated code, the fused annotation cursor, stripped files, and the large block sizes #60's redundant commits), force-push, squash-merge
  3. map PDB.open and place inline sites without holding a module's worth #62 — rebase onto the new main, force-push, squash-merge
  4. harden parse boundary, C13 diagnostics, and DBI/GSI error handling #53 — last; real conflicts in c13.py, pdb.py, test_diagnose.py, test_inline.py

What this stack carries forward

The inline-site rewrite here preserves an inconsistency from main and widens it. module_symbol_bytes strips the CodeView signature conditionally:

if len(raw) >= 4 and struct.unpack_from("<I", raw, 0)[0] == CV_SIGNATURE_C13:
    return raw[4:end]
return raw[:end]

while record offsets get + CV_SIGNATURE_SIZE unconditionally — at pdb.py:1059, :1093 and :1408, where main had two such sites. ProcSymbol.end is stream-relative ("signature included", per its docstring, unchanged here), so the compensation has to match the strip.

#53's P3-5 is the fix. I verified it against the corpus and it is correct — 757682 proc records, pEnd stream-relative in every one — but also that the branch never fires on real input: all 23478 module streams in the bag are C13-signed. So it is a robustness fix, not a live bug, and it is not a reason to hold this stack. Full evidence in the comment on #53.

For whoever cuts 0.6.0

#60 states it moves the second-and-later ranges of every inline site built from the fused opcode. CHANGELOG.md's preamble says a release that makes a previously-resolved address resolve differently is breaking and has to say so — that sentence was written for this case.

Also note all four PRs currently show 10 checks rather than 11: they predate the Changelog check. It runs on the next push, and all four touch CHANGELOG.md, so it should pass.

@r0ny123

r0ny123 commented Sep 15, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for mapping this out, that saved me a lot of guessing.

Went ahead with the first part: #60 is rebased onto main (only CHANGELOG/AGENTS conflicted, as you said) and #61 is rebased on top of the new #60. Both are green locally: 627 tests, lint and fuzz clean. #62 still has a merge from main on it, so I'll rebase it properly once #60 and #61 land, same for #53 after that.

Also put up #65 for #64. It strips C7/C11 the same as C13 and stops reading a module whose signature it doesn't know, with a diagnose() warning. Heads up that it touches the same lines as P3-5 in #53, so one of them will need a small rebase. P3-5 may end up a no-op if #65 goes in first.

@r0ny123
r0ny123 force-pushed the cursor/xul-memory-mmap-stream-cee3 branch from 755fb07 to 81e51e8 Compare September 15, 2026 07:47
danielplohmann pushed a commit that referenced this pull request Sep 15, 2026
…ation cursor, stripped files, and the large block sizes

Fifteen commits from a corpus audit of 421 real PDBs (python.org 2.7-3.14 on
x64/x86/arm64, node, Firefox's xul.pdb, Microsoft symbol-server files for
Win7/10/11 and XP, self-built clang-cl/lld-link and rustc variants, 168
corrupted derivatives). Every commit was gated on the full suite, `ruff check`,
`ty check` and a 2000-input fuzz pass; the corpus is rebuildable with the
`tools/fetch_corpus.py` this adds.

- **Inline sites on MSVC output.** MSVC writes `S_INLINESITE2` (48608 of the
  48642 sites in python312.pdb); 0.5.0 saw 34. The length fused into
  `ChangeCodeLengthAndCodeOffset` does not move the cursor - the old rule put
  5582 of 79187 ranges past their procedure or cold chunk, the corrected rule
  none - and `ChangeCodeOffsetBase n` names the procedure's n'th `S_SEPCODE`
  chunk. This moves the second-and-later ranges of every site built from the
  fused opcode, on rust-lld and clang output too.
- **Stripped PDBs produced no `diagnose()` warning**, which is the shape of
  every Microsoft public symbol file. DBI Flags and BuildNumber are read, three
  warnings cover the stripped / unflagged / empty-module-list cases, and the CLI
  opens with a `linker` line.
- **MSF block sizes 8192, 16384 and 32768** are accepted; link.exe writes them
  for PDBs past a few GB and they were refused outright.
- **A directory whose stream sizes sum past the file is rejected.** Each block
  list passed the per-stream check while sizing a multi-gigabyte allocation from
  a 4 MB file.
- `S_LPROC32_DPC`/`_DPC_ID` are procedures; `S_COMPILE2` feeds `compile_info()`;
  `DEBUG_S_LINES` BlockSize is floored at the bytes read.
- **Unnamed inline sites are counted and explained**: VS2015 wrote
  compiler-internal ids its linker never remapped, 5802 of 6554 sites in a
  python 3.5 `_hashlib.pdb`, and a missing IPI stream was silent too.

First of the queued audit stack: #61 and #62 build on this.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
r0ny123 and others added 3 commits September 15, 2026 21:15
open() maps the file by default so a 1.9 GB PDB is not copied into
bytes; close() and the context manager own that mapping, and copy=True
is the previous read. diagnose() and inline_sites() collect procs and
S_SEPCODE first, then place each site as it is parsed, because MSVC
writes the chunks after the procedure's scope and xul's largest modules
carry millions of sites.

A stripped PDB that still has procedure records (Win10/11 public files)
now gets a Diagnostics warning rather than leaving the caller to read
the flag. tools/fetch_corpus.py copies PUREPDB_EXTRA_PDBS if the caller
has FASTLINK/_ST/managed files this box cannot produce.

Co-authored-by: basak.rittick8274 <basak.rittick8274@gmail.com>
The corpus is on this box now, so the open/functions/diagnose figures
that perf.md still had as "not re-run" are from this tree: mapped open
is 264 MB against a 1.9 GB copy, and diagnose peaks at 4.4 GB instead
of 7.0 GB because sites are not held for the module.

Co-authored-by: basak.rittick8274 <basak.rittick8274@gmail.com>
AGENTS.md said 614 tests; this branch is 619. The README still opened
without close, which now holds a mapping. _place_module_sites had no
caller. RecordSurvey.malformed's total is only count_malformed_records
when parse= is omitted. Also: Windows cannot replace a mapped PDB,
rustc provenance comes from rustc --version, and the same-interpreter
node numbers from the review go in perf.md.

Co-authored-by: basak.rittick8274 <basak.rittick8274@gmail.com>
@danielplohmann
danielplohmann force-pushed the cursor/xul-memory-mmap-stream-cee3 branch from 81e51e8 to e17bc89 Compare September 15, 2026 19:15
@danielplohmann
danielplohmann merged commit 8a949f8 into danielplohmann:main Sep 15, 2026
11 checks passed
danielplohmann pushed a commit that referenced this pull request Sep 15, 2026
… unknown

Fixes #64.

`PDB.module_symbol_bytes` stripped the CodeView signature only when it was
`CV_SIGNATURE_C13`. A C7 (`0`) or C11 (`1`) signature stayed on the front of the
buffer, and `codeview.iter_records` read that word as a record header - for C11,
length 1 and kind 0 - so every record after it was walked at the wrong
alignment, with no error and no warning.

- All three signatures are recognised and stripped, as `llvm-pdbutil`'s module
  stream reader does.
- A first word that is none of the three yields no symbols: the layout after an
  unknown signature is not known, and reading the word as a record is what
  produced the silent misparse. `Diagnostics.unrecognised_signatures` counts
  those modules by the value found, and `diagnose()` warns with it.
- A stream too short to hold a signature is unchanged: its bytes go to the
  record walk, which reports them as a truncation.

Two tests, both verified to fail under a mutation of the line they cover:
`test_every_codeview_signature_is_stripped` (C7 and C11 fail when only C13 is
recognised) and `test_an_unrecognised_signature_is_not_parsed_as_a_record`,
whose unknown word is chosen so that, read as a header, it is a 16-byte
`S_GPROC32`. Every golden fixture is C13-signed, so no golden count moves.

This also settles #53's P3-5. The strip is no longer conditional on any path
that returns records - a body is either the stream minus exactly four bytes, or
empty - so the fixed `+ CV_SIGNATURE_SIZE` compensation already matches it. The
one path returning an unstripped non-empty body is a stream shorter than four
bytes, which cannot hold a record header and so yields no offsets at all.

Merged with main carrying #60, #61 and #62; 636 passed, ruff and ty clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
danielplohmann pushed a commit that referenced this pull request Sep 15, 2026
Findings from the correctness audit, each one a case where a damaged or unusual
file produced an unhandled exception or a silent gap rather than a `PdbError`
and a diagnostic.

- **A missing PDB Info stream** no longer escapes. `PDB.named_streams()` is
  guarded by `is_valid_stream` and catches `MsfError`, returning `{}` as
  documented, so `diagnose()`, `string_table()` and `lines()` produce empty
  results with a clear explanation instead of raising.
- **`PublicsStream.parse`** raises plain `PdbError` rather than `ValueError`
  when the header or address map is truncated, preserving the invariant that
  `PdbError` is the only exception crossing a public boundary. Plain `PdbError`
  and not `MsfError`: the publics stream is not an MSF container.
- **DBI substream sizes are bounded.** A negative size would slice backwards,
  aliasing bytes that were never substream data; a size overrunning the buffer
  is a header that lies about its own layout. Both raise `MsfError` with
  context, while damage *inside* a substream still degrades rather than raising.
- **C13 line-info truncations are reported.** `C13Truncation` records them,
  `iter_subsections` and `parse_lines` collect them, and
  `Diagnostics.c13_truncations` plus a warning surface them.
- **The fuzzer covers `publics_stream()`**, which was the one public listing it
  never exercised.
- Dead `Reader.align` removed; docstrings added to `ContributionMap.__len__`
  and `SectionContribution.contains`.

P3-5 (the conditional signature offset) is **not** in this PR. #65 made the
strip unconditional on every path that returns records, so the fixed
`+ CV_SIGNATURE_SIZE` already matches and the conditional helper had nothing
left to correct; `tests/test_inline.py` is identical to main as a result.

Last of the audit queue, after #60, #61, #62 and #65. 645 passed, ruff and ty
clean, fuzz clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
danielplohmann pushed a commit that referenced this pull request Sep 15, 2026
Dev-group pin only; `ruff` governs the `lint and type check` CI job and
`make lint`. No `.pre-commit-config.yaml` in this repository, so there is no
`ruff-pre-commit` rev that has to move with it.

Verified against this main rather than the one the PR was opened on: 0.16.6
reports `All checks passed!` over the whole tree, including the roughly 4400
lines that landed in #60, #61, #62, #65 and #53 after this PR was raised.
`target-version` stays `py311`.
danielplohmann pushed a commit that referenced this pull request Sep 15, 2026
Dev-group pin only; `ty` governs the type half of the `lint and type check` CI
job and `make lint`.

Dependabot rebased this onto the ruff bump and re-resolved the target from
0.0.79 to 0.0.80 in the process, so it was re-verified at the version actually
pinned here: 0.0.80 reports `All checks passed!` over the whole tree, including
the roughly 4400 lines that landed in #60, #61, #62, #65 and #53 after this PR
was raised.

`ty` is still 0.0.x, so it is the pin in this group most likely to surface a new
diagnostic on a bump.
danielplohmann added a commit that referenced this pull request Sep 15, 2026
Both version strings to 0.6.0, `## [Unreleased]` renamed to
`## [0.6.0] - 2026-09-15` with a fresh empty one above it, and the compare links
moved on. No source changes.

First release cut through `publish-release.yml`, which has existed since #59
without a tag ever exercising it.

23 entries from five merged PRs plus the release machinery and three dependency
bumps: the corpus-audit fixes (#60), the single-pass record walk (#61), the
mapped `PDB.open` and streamed inline-site placement (#62), the C7/C11 signature
handling (#65) and the parse-boundary hardening (#53).

Minor rather than patch: the `Fixed` entry for the fused
`ChangeCodeLengthAndCodeOffset` annotation says **This moves addresses** -- the
second and later ranges of an inline site built from that opcode now start
earlier than 0.5.0 reported, on every rust-lld and clang output. The first
range, and every single-range site, are unchanged. `CHANGELOG.md`'s preamble
reserves that for a breaking change and requires it to be stated.

Verified before tagging: `release_guard.py --tag v0.6.0` extracts the section
and reports prerelease=false, and refuses v0.7.0 naming both version strings;
645 passed, ruff and ty clean on the new pins; `python -m build` plus
`twine check --strict` pass on both artefacts; and the wheel, installed into a
clean venv and imported with `python -P`, reports 0.6.0 from site-packages with
`py.typed` present and runs the entry point.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@r0ny123
r0ny123 deleted the cursor/xul-memory-mmap-stream-cee3 branch September 30, 2026 18:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants