layer_numerics: add address capture, bad-value locate, allocator snap… - #267
Open
qianghan-amd wants to merge 3 commits into
Open
layer_numerics: add address capture, bad-value locate, allocator snap…#267qianghan-amd wants to merge 3 commits into
qianghan-amd wants to merge 3 commits into
Conversation
…shot, and tensor dump Add v2 diagnostic capabilities to the NaN logger for tracing memory-aliasing corruption to its producer buffer: - NANLOG_ADDR: record GPU data_ptr + storage extent per tensor (default ON, sync-free) - NANLOG_LOCATE: count bad rows per tensor to distinguish tile-sized late writes from numeric blowup - NANLOG_BAD_VALUES: record first bad element position and value (GPU-side reductions, no host sync) - NANLOG_ALLOC_SNAPSHOT: enable PyTorch allocator event recorder, dump on first NaN detection - NANLOG_DUMP_TENSOR: save full corrupted tensor to disk on first detection Also adds LOGGER_REFERENCE.md (full technical reference) and updates README.md with streamlined run instructions. Co-authored-by: Cursor <cursoragent@cursor.com>
qianghan-amd
force-pushed
the
users/qianghan/layer-numerics-v2
branch
from
July 1, 2026 21:36
d6f3178 to
cc402e9
Compare
… test suite _fwd_hook stashed a layer's output (act) before its input, so the one-shot NANLOG_DUMP_TENSOR captured the output instead of the corrupt input -- the aliased 8 MiB block that is the whole point of the emb_proj aliasing workflow. Stash the input before the output so the input wins when both go bad in the same step; layers where only the output is bad still dump the output, so nothing is lost. Also correct the NANLOG_DUMP_TENSOR docstring (one-shot, input-preferred). Add test_logger.py: 215-assertion pre-delivery suite covering all channels, BAD_VALUES, ADDR, LOCATE, ALLOC_SNAPSHOT, and DUMP_TENSOR. Test 12 asserts the dumped tensor is the input ([256,8192]), which is the regression guard for this fix. Co-authored-by: Cursor <cursoragent@cursor.com>
A customer-facing guide that stages the diagnostic runs from least to most perturbation, one new capability per round (BAD_VALUES + wide watch scope -> ALLOC_SNAPSHOT -> DUMP_TENSOR), so a timing-sensitive aliasing NaN is confirmed and its donor named before the heavier, race-perturbing flags are enabled. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…shot, and tensor dump
Add v2 diagnostic capabilities to the NaN logger for tracing memory-aliasing corruption to its producer buffer:
Also adds LOGGER_REFERENCE.md (full technical reference) and updates README.md with streamlined run instructions.
Motivation
Technical Details
Test Plan
Test Result
Submission Checklist