Skip to content

count the procs and publics diagnose() has already read - #52

Closed
r0ny123 wants to merge 1 commit into
danielplohmann:mainfrom
r0ny123:diagnose-single-pass
Closed

r0ny123 wants to merge 1 commit into
danielplohmann:mainfrom
r0ny123:diagnose-single-pass

Conversation

@r0ny123

@r0ny123 r0ny123 commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Summary

diagnose() walks every module stream to count record kinds, then called
module_procs(), which re-read and re-sliced each of those same streams to
extract the procedures it counts. Publics were worse: the symbol-record stream
is already read for the kind counts and the truncation check, and
public_symbols() read it a second time and then ordered the result through
the publics hash — an ordering the len() around it discards.

Both counts now come from bytes the function is already holding. The totals
are unchanged on all seven fixtures.

Not a measurable speedup, and worth saying so rather than leaving the next
reader to assume one: diagnose() is dominated by the record walks
themselves — count_malformed_records alone runs twice per module, once for
the general count and once for S_INLINESITE — against which the duplicated
reads removed here do not register.

Testing

  • pytest: 588 passed
  • ruff check and ty check: clean
  • tools/fuzz.py --iterations 2000 --seed 0: no exception escaped

diagnose() walks every module stream to count record kinds, then called
module_procs(), which re-reads and re-slices each of those same streams
to extract the procedures it counts. Publics were worse: the
symbol-record stream is already read for the kind counts and the
truncation check, and public_symbols() read it a second time and then
ordered the result through the publics hash -- an ordering the len()
around it discards.

Both counts now come from bytes the function is already holding. The
totals are unchanged on all seven fixtures.

Not a measurable speedup, and worth saying so rather than leaving the
next reader to assume one: diagnose() is dominated by the record walks
themselves -- count_malformed_records alone runs twice per module, once
for the general count and once for S_INLINESITE -- against which the
duplicated reads removed here do not register.
r0ny123 added a commit to r0ny123/purepdb that referenced this pull request Sep 12, 2026
…anielplohmann#59 and the dependabot bumps

Fill in the fuzz and timing results, note the follow-up branches, and
summarise the review in the log.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017jQKALSgFN7VecWTqzWSmD
@danielplohmann

Copy link
Copy Markdown
Owner

Superseded by #61, which landed the same ground with measurements behind it. Suggest closing.

This PR's contribution is removing len(self.module_procs()) and len(self.public_symbols()) from diagnose() so the counts come from bytes already held. On main before #61 both calls were there:

proc_records=len(self.module_procs()),
public_records=len(self.public_symbols()),

In #61 neither call exists. codeview.survey_records replaced that whole region, and the surviving comments mention module_procs() only to describe what the survey returns rather than to call it. The duplicate reads this removes are gone by construction there, along with the second count_malformed_records pass that this PR's own description names as the thing which actually dominates.

The framing here was right, and honest about its limits - "not a measurable speedup, and worth saying so rather than leaving the next reader to assume one". #61 measured the same walks at 3-8x once they were done once instead of three and four times.

Nothing is lost by closing: the behaviour is in #61, and this branch is still on the pre-release-workflow base and conflicts with both #62 and #53 in pdb.py precisely because it touches lines they rewrote. Worth a second look first if you see something here the survey does not cover.

danielplohmann pushed a commit that referenced this pull request Sep 15, 2026
…ery listing, output unchanged

Eleven commits on top of #60. Output snapshots are byte-identical to the fixes
branch on all 7 fixtures, 439 corpus files and node.pdb; every commit passed the
suite, `ruff`, `ty` and a 2000-input fuzz pass. Full tables and dead ends in
`docs/audit/perf.md`.

Wall seconds, best of N, 4-core Xeon, CPython 3.11:

| file | op | before | after | speedup |
|---|---|---:|---:|---:|
| sqlite x64 (3 MB) | all listings | 1.19 | 0.30 | 4.0x |
| | diagnose | 0.55 | 0.07 | 8.1x |
| python314.pdb (21 MB, MSVC PGO) | functions | 1.77 | 0.27 | 6.6x |
| | diagnose | 6.69 | 0.88 | 7.6x |
| ntkrnlmp.pdb (8.5 MB, stripped) | diagnose | 1.36 | 0.22 | 6.3x |
| node.pdb (355 MB) | functions | 21.6 | 3.5 | 6.2x |
| | all listings | 193 | 61 | 3.2x |
| xul.pdb (1.93 GB) | diagnose | 954 (11.4 GB RSS) | 152 (7.0 GB) | 6.3x |

What changed:

- `iter_records`: one `unpack_from` per header and a `kinds=` filter, so an
  extractor after one kind pays a slice and a `RawRecord` only for the records
  it wants. This loop was 73% of all time.
- `Reader.u16/u32` via `Struct.unpack_from` at the cursor; a fixed-portion
  `Struct` per named record kind, with one `find` for the name.
- `codeview.survey_records`: `diagnose()` and `functions()` walk each module
  stream once instead of three and four times, and `diagnose()` never builds the
  inline listing. This also subsumes #52, which is closed.
- `msf._read_blocks` reads contiguous block runs as single slices.
- Inline-site annotations walked with an integer cursor; the IPI counted
  positionally without a `RawRecord` per record; `Line`, `LineEntry`,
  `RawRecord`, `InlineFunction` slotted; per-file name and per-segment base
  hoisted out of the line loop.

Public API unchanged.

Rebased onto main after #60 squash-landed; the rebased tree is byte-identical to
the pre-rebase branch, so the squash carried #60 verbatim. 627 passed, ruff clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants