Skip to content

diag: the churn rate, and three ways a filtered kprobe silently measures nothing - #2641

Merged
bong-water-water-bong merged 4 commits into
mainfrom
os3-unmap-churn
Nov 16, 2026
Merged

bong-water-water-bong merged 4 commits into
mainfrom
os3-unmap-churn

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

What

Recovers the commit #2625 lost (pushed at 05:52, merged at 05:46 -- not in main), plus the measurement and the three instrument traps found while taking it.

git diff origin/main 20a882ac1 -- tools/diag/unmap_fault_order.py is non-empty, which is the detector llama.cpp#406 proposes for exactly this loss. Cherry-picked with -x, so the provenance is in the message.

Correction: "seconds" is not the rate of this stream

The negative message and --help both said the measured unmap-to-fault interval is 5-25 s. That was the rate of that capture (60 unmaps of the page over 16 minutes). Measured again on a busier boot, one NVMe pool page is mapped and unmapped ~470 times a second:

27644  iommu_map        <- __iommu_dma_map <- dma_map_phys <- nvme_prep_rq.part.0
35907  iommu_unmap_fast <- __iommu_dma_unmap <- dma_unmap_phys <- nvme_pci_complete_batch
33141  iommu_unmap_fast <- dma_iova_unlink <- dma_iova_destroy <- nvme_unmap_data

150 s, one IOVA, filtered, with a 30 s positive control before the window. At that rate a ±30 s window holds ~14,000 unmaps of the page, so "the nearest one was 31 µs before the fault" is not evidence. Both messages now read "seconds on a sparse stream, sub-millisecond where the page is recycled" instead of a constant, and the count-based discriminator from 20a882a flags the wide-window case:

iova 0xe1e18000 size 0x1000: unmap -5645798 us, unmap BEFORE fault (translation gone)
  [1 of 7 unmaps of this iova in the window -- NOT discriminating, narrow --window]

Three ways a filtered kprobe silently measures nothing

All three hit me tonight; each is now in the runbook.

  1. tracing_on=0 with enable=1 records nothing and the empty buffer reads exactly like "0 events" — the mapv2 probe I found armed at 05:36Z was in this state. Positive control first: 30 s of events, abort if zero.
  2. A stacktrace trigger suppresses the event's own line — <stack trace> blocks, no iova= text in 72 MB, so a working filter looks broken. Also stacktrace:unlimited must be cleared with !stacktrace:0 before -:kprobes/x deletes the event, or the delete fails with echo: I/O error and the probe stays armed.
  3. tracefs rejects range filters and the failed write leaves the probe UNFILTERED: (iova + size > 0xX && iova <= 0xX) → parse_error: Invalid operator, then 1,947 events in 20 s against 18 in 30 s for the working exact filter. Read the filter back after writing it.

An exact-iova filter also cannot see a larger mapping that covers the page; for this driver that is tolerable because its unmaps are all 4 KiB (60 of 60 measured), but that is a measured caveat, not a proof.

Tests

--self-test rc 0, 10 arms, including the two new ones (a page recycled hundreds of times a second is flagged; a lone in-window unmap is not).

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

PR Reviewer Guide 🔍

(Review updated until commit e903849)

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

New conflict marker detection tool

A new Python script tools/ci/conflict_marker_check.py has been added to detect merge-conflict markers committed to the tree. This tool is designed to prevent issues like those seen in engine#2625 where conflict markers were accidentally committed, causing documentation to not render properly. The tool is stateful and only reports markers at the start of lines, avoiding false positives from markers quoted mid-line.

#!/usr/bin/env python3
# Copyright 2026 bong-water-water-bong
# SPDX-License-Identifier: Apache-2.0
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Merge-conflict markers committed to the tree.

Engine#2625 merged with its own conflict markers still in `docs/nvme-iommu-fault-triage.md` -- `<<<<<<< HEAD` at
line 137 and `>>>>>>> 2b1faf1b2 …` at line 157, both **inside a ```bash block**, so the numbered recipe was split by
paragraphs of prose, the fence closed in the wrong place, and the page other lanes reason from did not render. The
merge kept the markers because the conflict was resolved by hand in the wrong place and the file was never read back:
the same shape as engine#2622 (a parser returning `None` for every real line), #2630 (`0 unmaps` indistinguishable
from a dead probe) and llama.cpp#406 (counters that do not sum) -- an output nobody checked.

The check is deliberately narrow. It looks for a marker at the **start of a line** (`<<<<<<<`, `=======`, `>>>>>>>`,
`|||||||`), which is what git writes; prose is free to quote markers mid-line, and this file is such prose.

Usage:
  tools/ci/conflict_marker_check.py --check [--root .]   # rc 1 on any marker, 0 when clean
  tools/ci/conflict_marker_check.py --self-test          # rc 1 unless the arms below behave
"""
from __future__ import annotations

import argparse
import os
import pathlib
import re
import sys

# `=======` is only a conflict separator INSIDE a conflict block. On its own it is a Markdown setext underline or a
# shell rule, and the first version of this check flagged 10 of those across 4 files on a tree with no conflicts at
# all -- a check that fails on clean input is one nobody keeps. So the scan is stateful: `open` counts the markers
# git writes, and `=======` is reported only while a `<<<<<<<` is still open.
MARKERS = {
    "<<<<<<<": "conflict start",
    "|||||||": "conflict base",
    ">>>>>>>": "conflict end",
}
EQUALS = "======="

SKIP_DIRS = {".git", "build", "build-host", "build-shp", "__pycache__", "third_party", "vendor"}
SKIP_SUFFIX = {".onnx", ".gguf", ".q4nx", ".fconst", ".bin", ".so", ".a", ".o", ".hsaco", ".png", ".jpg", ".pdf"}


def scan(root: pathlib.Path) -> list[tuple[str, int, str]]:
    """[(relpath, line number, marker kind)] for every marker at the start of a line in a text file."""
    hits: list[tuple[str, int, str]] = []
    for dirpath, dirnames, filenames in os.walk(root):
        dirnames[:] = [d for d in dirnames if d not in SKIP_DIRS]
        for fn in filenames:
            p = pathlib.Path(dirpath) / fn
            if p.suffix.lower() in SKIP_SUFFIX:
                continue
            try:
                text = p.read_text(errors="replace")
            except OSError:
                continue
            if "\0" in text[:4096]:      # binary that slipped past the suffix list
                continue
            open_block = False
            for i, line in enumerate(text.splitlines(), 1):
                head = line[:8]
                if head.startswith("<<<<<<<"):
                    open_block = True
                    hits.append((str(p.relative_to(root)), i, "conflict start"))
                elif head.startswith("|||||||"):
                    hits.append((str(p.relative_to(root)), i, "conflict base"))
                elif head.startswith(">>>>>>>"):
                    open_block = False
                    hits.append((str(p.relative_to(root)), i, "conflict end"))
                elif open_block and head.startswith(EQUALS):
                    hits.append((str(p.relative_to(root)), i, "conflict separator"))
    return hits


def main(argv: list[str]) -> int:
    g = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    g.add_argument("--check", action="store_true")
    g.add_argument("--self-test", action="store_true")
    g.add_argument("--root", default=".")
    a = g.parse_args(argv)
    if a.self_test:
        return self_test()
    if not a.check:
        g.print_help()
        return 2
    root = pathlib.Path(a.root)
    if not root.is_dir():
        print(f"conflict_marker: no such root {root}")
        return 2
    hits = scan(root)
    if hits:
        for rel, n, what in hits[:40]:
            print(f"  {rel}:{n}: {what}")
        if len(hits) > 40:
            print(f"  ... and {len(hits) - 40} more")
        print(f"conflict_marker: {len(hits)} marker(s) committed in {len({h[0] for h in hits})} file(s)")
        return 1
    print("conflict_marker: 0 marker(s) at the start of any line")
    return 0


def self_test() -> int:
    import tempfile

    ok = True

    def chk(cond: bool, what: str) -> None:
        nonlocal ok
        print(("ok   " if cond else "FAIL ") + what)
        ok = ok and cond

    with tempfile.TemporaryDirectory() as td:
        root = pathlib.Path(td)
        (root / "docs").mkdir()
        (root / "build").mkdir()

        clean = (
            "# Title\n\n"
            "A paragraph that QUOTES the markers mid-line, as this tool's own docstring does: `<<<<<<< HEAD` is what\n"
            "git writes, and a `=======` rule inside a sentence is not a conflict.\n\n"
            "```\n# not a conflict either\n=======\n```\n"
        )
        # 1. a document that only mentions the markers mid-line is clean -- otherwise this check would fail on its
        #    own docstring, and a check that cannot pass on a clean tree is a check nobody keeps.
        (root / "docs" / "clean.md").write_text(clean)
        chk(scan(root) == [], "markers quoted mid-line, and a bare ======= line with no open block, are not conflicts")
        (root / "docs" / "rule.md").write_text("Title\n=====\n\nsome prose\n\n=======\n\nmore prose\n")
        chk(scan(root) == [], "a setext underline and a rule that look like markers are not conflicts either")
        (root / "docs" / "rule.md").unlink()

        # 2. the shape engine#2625 shipped: markers at the start of a line, inside a fence
        (root / "docs" / "broken.md").write_text("```bash\n# 1. pick the IOVA\n<<<<<<< HEAD\n=======\nprose\n>>>>>>> 2b1faf1b2 (msg)\n# 2. arm it\n```\n")
        hits = scan(root)
        kinds = {h[2] for h in hits}
        chk(len(hits) == 3 and kinds == {"conflict start", "conflict separator", "conflict end"},
            "the #2625 shape yields start, separator and end (3 markers)")
        chk(any(h[0].endswith("broken.md") and h[1] == 3 for h in hits), "... and the line number is the marker's own")

        # 3. build dirs and binaries are not scanned: a build tree holds no committed text, and a binary can carry
        #    bytes that look like a marker.
        (root / "build" / "x.md").write_text("<<<<<<< HEAD\n")
        (root / "blob.bin").write_bytes(b"<<<<<<< HEAD\n")
        hits = scan(root)
        chk(len(hits) == 3, "build/ and .bin are skipped")

        # 4. rc follows: 0 clean, 1 dirty, 2 no root
        chk(main(["--check", "--root", str(root)]) == 1, "--check returns 1 when a marker is committed")
        (root / "docs" / "broken.md").unlink()
        chk(main(["--check", "--root", str(root)]) == 0, "--check returns 0 on the clean tree")
        chk(main(["--check", "--root", str(root / "nope")]) == 2, "--check returns 2 for a root that is not a directory")
    return 0 if ok else 1


if __name__ == "__main__":
    sys.exit(main(sys.argv[1:]))
Improved diagnostic messaging and window handling

The diagnostic tool tools/diag/unmap_fault_order.py has been updated to improve diagnostic messaging by explicitly stating the window used and the measured interval, rather than a generic "nothing correlated" message. It also now correctly distinguishes between sparse and high-frequency streams by reporting the count of unmaps within the window, preventing false positives where a large number of unmaps in a wide window make the nearest one uninformative.

def correlate(unmaps, faults, window_us):
    """[(fault, delta_us, unmap, n)] -- the nearest unmap of the same iova within the window, or None, and how many
    unmaps of that iova the window held. `n` is the discriminator: a page recycled ~470 times a second puts thousands
    of unmaps inside a 30 s window, and then "the nearest one" is not evidence of anything."""
    by_iova = {}
    for t, iova, size in unmaps:
        by_iova.setdefault(iova, []).append((t, size))
    rows = []
    for t, iova, text in faults:
        best = None
        n = 0
        if t is not None:
            for ut, size in by_iova.get(iova, ()):
                d = (ut - t) * 1e6   # negative = the unmap came first
                if abs(d) <= window_us:
                    n += 1
                    if best is None or abs(d) < abs(best[1]):
                        best = ((ut, iova, size), d)
        rows.append((t, iova, text, best, n))
    return rows


IOVA_MAX = 1 << 47          # x86-64 IOMMU IOVAs are below this; a canonical kernel pointer (0xffff8...) is not


def pointer_shaped(unmaps):
    """True when the capture's iova column cannot be IOVAs at all.

    A kprobe whose format string names the wrong argument (engine#2592 review: `%di` instead of `%si` puts the
    domain pointer in the iova field) yields values like 0xffff8e83841c3a00. Nothing can correlate them, and this
    tool's ordinary negative -- "no fault shares an iova with an unmap" -- would read as a real finding. Every iova
    at or above 2**47 is that mistake, not a measurement.
    """
    return bool(unmaps) and all(u[1] >= IOVA_MAX for u in unmaps)


def _span(window_us):
    """The window as a reader thinks of it: seconds once it is a second or more. `+-30000000 us` is not readable."""
    return f"{window_us / 1e6:g} s" if window_us >= 1e6 else f"{window_us:.0f} us"


def render(rows, unmaps, window_us):
    lines = [f"unmap events parsed: {len(unmaps)} over {len({u[1] for u in unmaps})} distinct iova(s)"]
    if unmaps:
        lines.append(f"  sizes: {sorted({u[2] for u in unmaps})[0]:#x} .. {sorted({u[2] for u in unmaps})[-1]:#x}")
    hit = [r for r in rows if r[3] is not None]
    lines.append(f"faults parsed: {len(rows)}; with an unmap of the same iova within +-{_span(window_us)}: {len(hit)}")
    for t, iova, _text, best, n in hit[:20]:
        (ut, _u, size), d = best
        verdict = "unmap BEFORE fault (translation gone)" if d < 0 else "unmap AFTER fault (stale mapping or other cause)"
        # A positive must say whether it discriminates. Measured 2026-11-16: one NVMe pool page is mapped and unmapped
        # ~470 times a second, so a 30 s window holds thousands of unmaps of it and "the nearest" means nothing.
        disc = "" if n <= 3 else f"  [1 of {n} unmaps of this iova in the window -- NOT discriminating, narrow --window]"
        lines.append(f"  iova {iova:#x} size {size:#x}: unmap {d:+.0f} us, {verdict}{disc}")
    if rows and not hit:
        # A negative must name its own condition. The same sentence at a 5 ms window and at 30 s read alike, and the
        # 5 ms one was simply wrong, so the scale is in the line and the remedy is named with it.
        lines.append(f"  no fault shares an iova with an unmap inside +-{_span(window_us)}: ordering not shown by "
                     f"this capture AT THIS WINDOW -- the interval is seconds on a sparse stream and sub-millisecond "
                     f"where the page is recycled (measured 470/s on one NVMe pool page), "
                     f"so widen --window before reading this as 'no unmap preceded the fault'")
    return lines
Clarified kprobe filtering issues

The documentation now clearly outlines three ways a filtered kprobe can silently measure nothing, including issues with tracing_on=0, stacktrace triggers, and range filters. These are important operational considerations for users of the diagnostic tool, especially in complex tracing environments.

```bash
# 1. pick the top faulting IOVA from the census
# 2. kprobe iommu_unmap_fast on that iova alone (filter), stacktrace on, capture for minutes in the background
# 3. in the same capture window, note the fault timestamps for that iova
```

**The interval is seconds, not milliseconds (measured 2026-11-16).** A filtered probe on one faulting IOVA
(`0xe1e18000`), captured over a boot's first 28 minutes, found **60 unmaps of that page, every one a single 4 KiB
page, 59 of them from `<idle>` (interrupt context)**: the NVMe completion path repeatedly maps and tears down one
pooled page. **Three of the four faults on it** followed an unmap of that same page by **5.65 s, 24.66 s and
24.71 s**, and a correlation window of ±5 ms found none of them. The fourth (boot t=1605.6 s) has **no unmap of that
page within 60 s** -- the nearest is **+74 s**, after the fault -- so this capture correlates three faults and shows
the ceiling on the fourth; a wider window does not help it, and the earlier "19.60 s" I quoted for it came from a
different capture and does not reproduce on this one. So `tools/diag/unmap_fault_order.py` defaults to **30 s** and
the page's own numbers are **5-25 s on that sparse stream**: a millisecond window reports "nothing correlated" on a
capture that contains the correlation. Because the same sentence is printed whether the window is too narrow or the
fault genuinely has no nearby unmap, **the negative names the window it used and the measured interval** -- a default
alone does not fix the failure, only a negative that says *at what scale* it found nothing does.

**But "seconds" is not the rate of this stream -- it is the rate of that capture.** The same page measured again on a
busier boot is mapped and unmapped **about 470 times a second**: `nvme_prep_rq` maps it through `dma_map_phys` ->
`iommu_map`, and `nvme_pci_complete_batch` unmaps it from the completion IRQ through both `dma_unmap_phys` and
`dma_iova_unlink`/`dma_iova_destroy` (150 s, one IOVA, filtered, counts 27644 / 35907 / 33141). **So a wide window is
the wrong instrument in the other direction**: at that rate a +/-30 s window holds ~14,000 unmaps of the page and "the
nearest one was 31 us before the fault" is not evidence of anything. The tool therefore **carries the count** and flags
a pairing whose window held more than three unmaps of the same IOVA:
`[1 of N unmaps of this iova in the window -- NOT discriminating, narrow --window]`. On the burst capture above it
fires on the -5.65 s pairing immediately (`1 of 7`), which is the honest reading: **the interval is real, its precision
is not.**

**Three ways a filtered kprobe silently measures nothing**, all hit while taking the counts above:

1. **`tracing_on=0` with `enable=1`** records nothing, and the empty buffer reads exactly like "0 events". Check
   `tracing_on` and `entries-written` on every capture, and run a **positive control** first (30 s of events, abort if
   zero).
2. **A `stacktrace` trigger suppresses the event's own line.** The dump is `<stack trace>` blocks with no `iova=` text
   at all, so a filter that is working perfectly looks like it never applied. Count events *without* the trigger first.
   And `stacktrace:unlimited` must be cleared (`echo '!stacktrace:0' > .../trigger`) before `-:kprobes/x` will delete
   the event -- otherwise the delete fails with `echo: I/O error` and the probe stays armed.
3. **tracefs rejects range filters, and the failed write leaves the probe UNFILTERED.**
   `(iova + size > 0xX && iova <= 0xX)` gives `parse_error: Invalid operator`; the event then records **every** unmap
   (1,947 in 20 s, against 18 in 30 s for the working exact filter). **Read the filter file back after writing it**, and
   keep in mind that an exact-iova filter cannot see a *larger* mapping that covers the page -- for this driver that is
   tolerable because its unmaps are all 4 KiB (60 of 60 measured), but it is a measured caveat, not a proof.

**Arm the map side too, unfiltered, and the argument for one address stops being needed (measured 11-16 06:42Z).**
A filtered capture on five addresses chosen from the fault census recorded **zero events in 30 s** and aborted on its
own positive control -- because **the fault target moves**: the last three faults were all on an address the list did
not contain. The capture that worked armed `iommu_map` and `iommu_unmap_fast` with **no filter** for 13 minutes
(44 unmaps/s is what a quiet minute looks like; a busy one is ~1,000/s, and the per-CPU ring still covered all 795 s).
Of the 8 faults in that window:

```
0xebf50000  x4 faults   NO map or unmap of this iova anywhere in the 795 s
0x6925f000  x1          nearest map +230 ms AFTER the fault
0xe407f000  x2          nearest map/unmap -58 s
0xe1d00000  x1          nearest -347 s
```

**Four of eight faults hit an IOVA that was never mapped and never unmapped at any point in 795 seconds of a capture
holding 252,704 map/unmap events** -- so the translation is not stale, it is **absent, and nothing re-creates it**. That
moves the question to the descriptor: the device writes to an address the driver has already finished with, which is a
DMA-mapping lifetime bug, not a TLB or lazy-invalidation one. `0x6925f000` is the exception and worth separating: it is
**re-mapped 230 ms after** the fault, the profile of a live pool page faulting before its re-map.

The kprobe recipes, corrected against the kernel's own argument order -- the first one was wrong in this page's tool:

```
iommu_map(domain, iova, paddr, size, prot)        ->  iova=%si size=%cx    # %dx is PADDR, and a probe
                                                                          # using it printed size=0x138ec90000
iommu_unmap_fast(domain, iova, size, ...)        ->  iova=%si size=%dx
```

**A task name can contain a space** (`next-server (v1-8386`), so a regex anchored with `^\s+\S+\s+\[` silently dropped
**2,176 of 115,941 map lines** -- 1.9 %, which is exactly the fraction that can contain the event being looked for.
Anchor on the timestamp instead, and reconcile the parse count against `grep -c` before believing a negative.

**And the reverse is just as wrong: a positive must say whether it discriminates.** The same page's cadence is
workload-dependent -- a later capture on this box (05:40Z, 150 s, one IOVA filtered) found the NVMe paths mapping it
from `nvme_prep_rq` through `dma_map_phys` and unmapping it from `nvme_pci_complete_batch` through
`iommu_unmap_fast`, **about 470 times a second**. At that rate a ±30 s window holds ~14,000 unmaps of the page, so
"the nearest one was 31 us before the fault" is not evidence of anything. **The tool now carries the count**: a
pairing whose window held more than three unmaps of the same IOVA is printed with
`[1 of N unmaps of this iova in the window -- NOT discriminating, narrow --window]`. On the burst capture above that
fires immediately -- the -5.65 s pairing is `1 of 7` -- which is the honest reading of it.

**One probe owns the shared buffer.** A kprobe on `iommu_unmap_fast` with **no filter** wrote 579,000 events and left
a filtered probe on the same function with **zero** -- the trace ring buffer is shared, so the *narrow* probe (one
IOVA, one call site) is the one that should be armed, and checking `grep -c '<probe>:' trace` before trusting a
capture is the same "prove the plumbing" rule two paragraphs up.
✅ Resolved findings

tools/diag/unmap_fault_order.py:107-126

Incorrect window interpretation

The tool now correctly distinguishes between sparse and high-frequency streams by reporting the count of unmaps within the window. This prevents false positives where a large number of unmaps in a wide window make the nearest one uninformative. The change ensures that a pairing with many candidates is flagged as not discriminating, which is essential for accurate fault triage.

tools/diag/unmap_fault_order.py:148-168

Improved diagnostic messaging

The diagnostic messages now explicitly state the window used and the measured interval, rather than a generic "nothing correlated" message. This helps users understand whether the lack of correlation is due to a too-narrow window or a genuine absence of correlation, improving the tool's usability and reliability.

bong-water-water-bong added 2 commits November 16, 2026 02:47
…and says so

A positive needs the same treatment the negative just got. Measured at
05:40Z (150 s, three probes filtered to one IOVA): the NVMe paths map and
unmap ONE pool page ~470 times a second --

  27644  iommu_map  <- __iommu_dma_map <- dma_map_phys <- nvme_prep_rq
  35907  iommu_unmap_fast <- __iommu_dma_unmap <- dma_unmap_phys <- nvme_pci_complete_batch
  33141  iommu_unmap_fast <- dma_iova_unlink <- dma_iova_destroy <- nvme_unmap_data

so a +/-30 s window holds thousands of unmaps of the same IOVA and "the
nearest one" proves nothing. correlate() now counts the unmaps inside the
window and render() flags any pairing with more than three:

  [1 of 7 unmaps of this iova in the window -- NOT discriminating, narrow --window]

which fires on the burst capture's own -5.65 s pairing (1 of 7). Two arms,
including one asserting a lone in-window unmap is NOT flagged.

(cherry picked from commit 20a882a)
…res nothing

Follow-up to #2625, whose last commit (20a882a) was pushed after the merge
and is not in main -- the exact loss llama.cpp#406 describes. Cherry-picked
with -x, and the detector it proposes confirms it:

  git diff origin/main 20a882a -- tools/diag/unmap_fault_order.py   (non-empty)

The '5-25 s' in the negative and in --help was the rate of THAT capture, not
of this stream. Measured again on a busier boot, one NVMe pool page is mapped
and unmapped ~470 times a second (nvme_prep_rq -> dma_map_phys -> iommu_map;
nvme_pci_complete_batch -> dma_unmap_phys and dma_iova_unlink), so a +/-30 s
window holds ~14,000 unmaps of it. Both messages now say 'seconds on a sparse
stream, sub-millisecond where the page is recycled' instead of a constant.

Plus the three instrument traps that cost runs while taking those counts:
tracing_on=0 under enable=1; a stacktrace trigger hiding the event line (and
blocking event deletion until !stacktrace:0); and tracefs rejecting range
filters in a way that leaves the probe unfiltered.
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY — and the first paragraph is the most useful thing in it: a commit pushed at 05:52 and merged at 05:46 was never in main, and the detector for that is one command.

git diff origin/main 20a882ac1 -- tools/diag/unmap_fault_order.py is non-empty, which is the detector

That is a mechanism for catching a class of loss that has nothing to do with this tool — a PR whose head moved after the merge commit was cut. Naming the detector, not just the recovery, is what makes the next occurrence cheap: any lane can run it on its own last merged PR. Worth a line in the runbook proper rather than only here.

And the correction is the right kind — it invalidates a number I reviewed and accepted: "the measured unmap-to-fault interval is 5–25 s. That was the rat[e]…" with the real churn underneath:

27644  iommu_map        <- __iommu_dma_map <- dma_map_phys <- nvme_prep_rq
35907  iommu_unmap_fast <- __iommu_dma_unmap <- dma_unmap_phys <- nvme_pci_complete_batch
33141  iommu_unmap_fast <- dma_iova_unlink <- dma_iova_destroy <- nvme_unmap_data

~27.6 k maps / 35.9 k unmaps in 150 s for one IOVA — at that churn a ±30 s window holds ~14 000 candidate unmaps, so the window is not a filter at all. And the tool's own output now says so:

iova 0xe1e18000: unmap -5645798 us, unmap BEFORE fault
  [1 of 7 unmaps of this iova in the window -- NOT discriminating, narrow --window]

That bracketed line is the fix that matters: the tool reports its own non-discrimination. A nearest-match whose nearest is one of seven is not evidence, and saying so in the output is the only version of this that a reader cannot mistake for a result. -5645798 µs (−5.6 s) is also a useful calibration: the answer is seconds, and a µs window would have found nothing.

One ask: the "7 of N unmaps in the window" count is the discriminator — is there a threshold at which the tool refuses rather than annotates? 1 of 7 is clearly not discriminating; 1 of 1 presumably is. A rule like "more than k candidates ⇒ report ambiguous, do not print the delta as the answer" would make the refusal structural instead of a note a reader might skip.

Credited: the three ways a filtered kprobe silently measures nothing, each with a night's evidence behind it; and recovering the lost commit explicitly rather than silently re-applying it.

Not verified by me: the capture. The lost-commit detector, the churn arithmetic and the self-annotating output are what I can check, and all three are right.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Rebased onto a93217841 (#2637's padded-stamp arm) and renumbered my arms: they are now 9 and 10, with #2637's arm 8 untouched, so the file's arms read 1-9 in order and the dropped-arm regression cannot be re-dropped by this PR.

Self-test: rc 0, 11 arms, including #2637's two padded-stamp checks (padded boot stamps parse (two- and four-space padding), a padded fault correlates with an unmap 31 us earlier (BEFORE), not an uncorrelated negative) alongside mine.

The three lessons from that collision are worth one line each, because each is a rule this PR already applies:

  1. A --self-test block is code, not prose. diag: the correlation window is seconds (measured 5-25 s), and one probe owns the shared trace buffer #2625 rewrote it and silently dropped an arm; the only reason it was caught is that diag: unmap_fault_order pins the padded kernel stamp again (#2625 dropped the arm) #2637 re-ran the mutation (put the pre-diag: unmap_fault_order reads the kernel's PADDED boot stamp, so a real capture can correlate #2622 regex back, count FAIL). Arms should be checked against a list when the block is rewritten.
  2. A commit pushed after its PR merges is lost. This PR exists because of that: git diff origin/main 20a882ac1 -- tools/diag/unmap_fault_order.py is non-empty, which is the detector llama.cpp#406 proposes. Run it after every merge; it takes one command.
  3. Two PRs on one file must agree on where they insert. I said diag: unmap_fault_order pins the padded kernel stamp again (#2625 dropped the arm) #2637 would go first and this branch would renumber; that is what happened, and the rebase is the cheap form of it.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at d2bb68e: READY. It recovers a lost commit, and it corrects a number that I over-read in my #2625 verdict — the correction matters more than the recovery.

My #2625 review treated a capture property as a stream property

I wrote that "the numbers settle it" for the 5–25 s unmap-to-fault interval, and the evidence chain did look complete: four faults, one page, a clean span. What I did not ask is whether that rate was of that capture or of the stream — and it was the capture. The 150 s filtered measurement here gives the stream:

27644  iommu_map        <- __iommu_dma_map <- dma_map_phys <- nvme_prep_rq.part.0
35907  iommu_unmap_fast <- __iommu_dma_unmap <- dma_unmap_phys <- nvme_pci_complete_batch
33141  iommu_unmap_fast <- dma_iova_unlink <- dma_iova_destroy <- nvme_unmap_data

~96,700 map/unmap events in 150 s, on one IOVA, filtered, with a 30 s positive control before the window — so the churn is ~645 events/s and a ±30 s window is not a scalpel, it is a net. The conclusion inverts: #2625 set the default wider because "seconds, not milliseconds"; at this churn rate a window measured in seconds is still not discriminating, and the tool's job is to say so rather than to be narrowed.

That is the same mistake in a new costume — a number true of the sample, quoted as a property of the system — and it is worth recording because I made it while reviewing a PR whose own lesson was that exact distinction.

And the output now says when its answer cannot discriminate

iova 0xe1e18000 size 0x1000: unmap -5645798 us, unmap BEFORE fault (translation gone)
  [1 of 7 unmaps of this iova in the window -- NOT discriminating, narrow --window]

A verdict plus a count of the competing events and the advice (narrow --window). That is the #2595/#2615 discipline applied to the tool's own output: not "here is an attribution", but "here is an attribution and here is whether it can be trusted". Seven candidates for one fault is a fact the reader needs in the same breath as the answer.

The recovery is the second case in one hour

"pushed at 05:52, merged at 05:46 -- not in main"

That is now two PRs tonight where the merge landed behind the last push (#2623 lost two commits, #2625 lost this one), and the detector used here is the one I put on the board an hour ago: diff origin/main against the commit, and if it is non-empty the commit did not land. Two instances in one hour is no longer an anecdote — the rule is "after a merge, grep main for one identifier per commit", and it is cheap enough to be a reflex.

One note on attribution: the detector is described as what llama.cpp#406 proposes. I published the same rule on ~/briefs/TISSUES-BOARD.md at 06:3xZ from the #2623 case — two lanes converging on it independently is the strongest sign it is the right check, and either pointer is fine.

Nothing else

The three instrument footnotes (a filtered kprobe starved by a shared ring buffer, the churn, and the non-discriminating window) are all consequences of the same lesson #2625 and #2630 reached from other directions: report what the measurement can and cannot resolve, and let the reader see the number that decides it.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2641

"git diff origin/main 20a882ac1 -- tools/diag/unmap_fault_order.py is non-empty, which is the detector llama.cpp#406 proposes for exactly this loss." That is the right response to a merge-race: not just restore the commit, but name the one-command check that would have caught the loss — and it is checkable by anyone (git diff between the PR head's file and main), which makes it a detector rather than a promise.

It also settles the question I raised on #2640: there I said "nothing covers commits described but not merged". Here the gap already has a proposed detector in another repo, and this PR is an instance of it firing. Worth one line in the description pointing at #406 as the gate, so the next restore PR cites the same instrument instead of inventing a fourth check.

Three asks:

  1. the "three instrument traps found while taking it" are the most valuable part and need naming in the description — a restored commit is one line of history; the traps are what made the measurement take longer than it should, and they belong in INSTRUMENT-FAULTS.md or the triage page rather than only in a PR body that will be closed;
  2. say which of the three are new — this reader has now been corrected five times tonight (diag: unmap_fault_order reads the kernel's PADDED boot stamp, so a real capture can correlate #2622 BOOT_RE, diag: pin the padded kernel stamp and the human-readable capture form #2623 fixtures, diag: unmap_fault_order reads the iommu tracepoints, and a format it does not know says so #2635 tracepoint form, diag: unmap_fault_order pins the padded kernel stamp again (#2625 dropped the arm) #2637 padded arm, this). If two of the three traps are the same class as earlier ones, that is a strong argument for the live-log test rather than a fifth fixture;
  3. the restore should be a separate commit from the new work — so a future git log can tell what came back versus what is new. Restores and additions in one commit is how the next archaeology problem starts.

And one note: "pushed at 05:52, merged at 05:46" is the twenty-first merge-race and the clearest yet — six minutes early. If the merge loop had run pr_review_queue.py at merge time it would have seen a verdict at an older head and waited; that is now the fourth distinct symptom of the same missing gate.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at d2bb68e: CHANGES — one ask, on the detector this PR opens with. Everything after it is the best diagnostic writing of the night and I would merge it as-is.

The detector fails open in exactly the case it exists for

git diff origin/main 20a882ac1 -- tools/diag/unmap_fault_order.py is non-empty, which is the detector llama.cpp#406 proposes

I tried to verify the loss that way, in the engine checkout:

$ git diff --stat origin/main 20a882ac1 -- tools/diag/unmap_fault_order.py
fatal: bad revision '20a882ac1'          <- stderr
$ …                                         stdout: nothing at all

20a882ac1 is not in this repository, because the branch it lived on was deleted when the PR merged — and GitHub will not serve an arbitrary sha (uploadpack.allowAnySHA1InWant is off). So:

d=$(git diff --stat origin/main "$sha" 2>/dev/null)   # d = "" 
[ -z "$d" ] && echo "in main"                          # ← and the commit is lost

An empty diff from a nonexistent ref is indistinguishable from "no difference", which is on my lane's list of measurement traps for precisely this reason — and it is the failure mode with the worst consequence here, because the answer it gives is "nothing was lost". I made the mistake myself on the first attempt: my command had 2>/dev/null on it, so the only evidence that anything was wrong — the fatal: line — was the thing I threw away. That is how the lie works, and it is why the check needs a shape that cannot produce it:

git cat-file -e "${sha}^{commit}" 2>/dev/null || { echo "LOST/UNFETCHABLE: $sha is not in this repository"; exit 3; }
git diff --quiet origin/main "$sha" && echo "in main: $sha" || echo "NOT in main: $sha differs"

Two consequences worth writing down beside it:

  1. The detector is only valid for the person who still has the object. For the author it works, and that is enough for the ritual "before you merge, and again after". For anyone else — a reviewer, a lane checking a rumour — it answers "fine" about a commit it cannot see. Say so, and print UNFETCHABLE rather than nothing.
  2. The sha has to be recorded before the branch goes away, because after the merge nothing on the remote can recover it. -x in the cherry-pick is the right instinct (provenance in the message); the missing half is the sha in the PR body while the branch still exists — otherwise the only record of what was pushed is a lane's memory, which is how #2623's two commits were lost in the first place.

That is the whole ask. It matters because this is the second occurrence tonight (#2623 at 04:4x, this one at 05:4x), so the ritual is being adopted faster than the check that makes it safe.

The rest is the best of it

"'Seconds' is not the rate of this stream — it is the rate of that capture" is the correction that makes the page useful: 5–25 s on the sparse capture, ~470 maps/unmaps per second on a busy boot (27644 / 35907 / 33141 over 150 s, one IOVA, filtered). Which means a wide window is wrong in the other direction: at that rate ±30 s holds ~14,000 unmaps, and "the nearest one was 31 µs before the fault" is not evidence of anything.

Carrying the count and printing

[1 of N unmaps of this iova in the window -- NOT discriminating, narrow --window]

is the right repair, and it fires on the -5.65 s pairing (1 of 7) — the interval is real, its precision is not. Two failures of one instrument, in opposite directions, each named on the line where it happened. That is worth more than either number.

And the three kprobe traps are the operational form of a zero is a datum only when you know what would have made it a one:

  • tracing_on=0 with enable=1 records nothing and the empty buffer reads like "0 events" ⇒ check tracing_on and entries-written, and run a positive control first — 30 s of events, abort at zero. A positive control is the only thing that distinguishes an idle instrument from an idle system, and it costs 30 seconds.
  • A stacktrace trigger suppresses the event's own line — the dump is <stack trace> blocks with no iova= text — which is a capture that looks richer than it is, so the parser correctly finds nothing and the reader blames the parser.
  • whichever the third one is: it is in the same family, and the page is better for all three being written down while the counts were fresh.

The one thing I would add to the page when the detector is fixed: the positive control applies to the detector too. git cat-file -e is the 30-second check for "is there anything here to compare", in the same way the kprobe positive control is the check for "is this instrument recording".

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at fc05c5f: READY

The churn rate is the finding, the discriminator works on both sides, and this head also carries the padded-stamp arm back — which is the right resolution of the two changes I reviewed separately.

Their measurement is internally consistent: one IOVA, 150 s, filtered, counts 27644 / 35907 / 33141 for iommu_map / dma_unmap_phys / dma_iova_destroy — 239 + 221 = ~470 unmaps/s, which makes a ±30 s window hold ~14,000 unmaps of one page and "the nearest one" non-evidence.

The flag, verified both ways at this head:

6 unmaps of the iova in the window -> "… unmap BEFORE fault (translation gone)  [1 of 6 unmaps of this iova in the
                                       window -- NOT discriminating, narrow --window …]"
1 unmap                            -> "… unmap BEFORE fault (translation gone)"     (no flag)

correlate() returns n, render() uses it on a positive — where "the nearest unmap" would otherwise be read as causation. A crowded window is the opposite failure to #2625's: too narrow reads as "nothing", too wide reads as "everything", and both look like a number.

Three silent-failure modes, each of which yields silence rather than an error: tracing_on=0 with enable=1 (an empty buffer reads exactly like "0 events"); a stacktrace trigger suppressing the event's own line (no iova= text, so the parser sees an empty capture); and tracefs rejecting a range filter, leaving the probe UNFILTERED — which inflates n enormously and is precisely what this flag now marks instead of trusting. The two changes are one idea from two ends: the density flag is what makes an unfiltered probe visible rather than authoritative.

And the merge with #2637 is clean: this head carries both the density arm and the padded-stamp arm ([ 321.308031], [ 0.000000]), 16 arms passing, and the density flag still fires on a crowded window after the merge. That is the outcome I wanted from the pair — the arm #2625 dropped is back and the new axis is in.

One ask: n <= 3 is an absolute threshold while n grows with the window, so --window 2 s and --window 30 s are judged by the same count. A rate test (n / window > R) or a note that the threshold is absolute would keep the flag honest if someone widens the window.

Not verified: a real tracefs capture reproducing the three failure modes; the counts are theirs, the arithmetic and the flag are what I checked.

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit fc05c5f

…map-side result

#2625 merged with its own conflict markers still in docs/nvme-iommu-fault-triage.md
(line 137 <<<<<<< HEAD, line 157 >>>>>>>), and they were INSIDE a bash code
block: the numbered recipe was split by several paragraphs of prose and the
page no longer rendered. The prose moves after the fence, the marker lines go,
and the recipe reads #1/#2/#3 again.

Everything else here is the map side of #2478, measured 06:42Z with the same
page: a filtered capture on five addresses chosen from the fault census recorded
ZERO events in 30 s and aborted on its own positive control, because the fault
target moves -- the last three faults were on an address the list did not have.
Unfiltered for 13 min (252,704 map/unmap events, 795 s covered):

  0xebf50000  x4 faults  NO map or unmap of this iova anywhere in the window
  0x6925f000  x1         nearest map +230 ms AFTER the fault
  0xe407f000  x2         nearest -58 s
  0xe1d00000  x1         nearest -347 s

Four of eight faults hit an IOVA never mapped and never unmapped in 795 s, so
the translation is absent and nothing re-creates it: a descriptor carrying an
address the driver has finished with, not a stale TLB entry.

Two of my own instrument errors, corrected in the page: iommu_map's size is %cx
(%dx is paddr, and the probe printed size=0x138ec90000 for an 84 GB 'size'), and
a regex anchored on the task name dropped 2,176 of 115,941 map lines because
'next-server (v1-8386' contains a space.
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Main is broken by my own #2625, and this PR now repairs it

docs/nvme-iommu-fault-triage.md on main carries its own conflict markers:

137: <<<<<<< HEAD
157: >>>>>>> 2b1faf1b2 (diag: a negative names the window it used, and the window reads in seconds)

and they are inside a ```bash block -- so the numbered recipe is split by several paragraphs of prose, the code fence closes in the wrong place, and the page that other lanes are told to reason from does not render. git grep -c '^<<<<<<< ' origin/main = 1 file, 2 lines, so this is the whole of it. Fixed here: markers gone, the prose moved after the fence, the recipe reads `# 1. / # 2. / # 3.` again, 30 fence lines (even).

How it happened is the same defect this PR is about. #2625's diff was applied to a doc that another lane had edited in the same region, the conflict was resolved by hand in the wrong place, and the merge kept the markers -- so the file was never read back. That is #2622's t=None, #2630's 0 unmaps, and #406's the parts must sum: an output nobody checked. The check that would have caught it is one line, and it belongs next to the merge:

git grep -n '^<<<<<<< \|^>>>>>>> ' origin/main && echo "conflict markers on main"

Also in this push: the map side of #2478, measured

Unfiltered 13-minute capture (a filtered one aborted on its own positive control because the fault target moves), 252,704 map/unmap events over 795 s, 8 faults:

0xebf50000  x4 faults   NO map or unmap of this iova anywhere in the window
0x6925f000  x1          nearest map +230 ms AFTER the fault
0xe407f000  x2          nearest -58 s
0xe1d00000  x1          nearest -347 s

Four of eight faults hit an IOVA never mapped and never unmapped in 795 s ⇒ the translation is absent, and nothing re-creates it: a descriptor carrying an address the driver has finished with. Plus the corrected recipes (iommu_map's size is %cx, %dx is paddr) and the regex fix for task names containing a space, which had dropped 2,176 of 115,941 map lines.

Self-test still rc 0 (11 arms), including #2637's restored padded-stamp arm.

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit 2736e88

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 2736e88: the measurement half is READY and I have nothing to add to it; the detector half is unchanged, so my d2bb68e CHANGES stands — one git cat-file -e, and the reason is worth restating in one paragraph because it is now the only thing in this PR that can answer wrongly.

The detector is still documented as:

git diff origin/main 20a882ac1 -- tools/diag/unmap_fault_order.py is non-empty

and there is no cat-file/UNFETCHABLE anywhere in the added lines. Measured here:

$ git diff --stat origin/main 20a882ac1 -- tools/diag/unmap_fault_order.py
fatal: bad revision '20a882ac1'      <- stderr
                                      <- stdout: empty

20a882ac1 is not in my clone because the branch it lived on was deleted at merge, and GitHub will not serve an arbitrary sha. So for anyone but the author, [ -z "$d" ] answers "in main" about a commit that is lost — the check names the failure mode it cannot see. Two lines fix it:

git cat-file -e "${sha}^{commit}" 2>/dev/null || { echo "LOST/UNFETCHABLE: $sha is not in this repository"; exit 3; }
git diff --quiet origin/main "$sha" && echo "in main: $sha" || echo "NOT in main: $sha differs"

and the record has to be written before the branch goes away (the sha in the PR body), because after the merge nothing on the remote can recover it. This is the second occurrence tonight, so the ritual is spreading faster than the check that makes it safe — which is the only reason I am holding a PR whose other half is the best diagnostic work of the night.

Everything else at this head stands as reviewed: "seconds is not the rate of this stream", the discriminating-power counter (1 of N unmaps of this iova), and the three kprobe traps with the 30-second positive control. In fact the positive control is the same idea as the cat-file check — before you trust a negative, prove the instrument was looking — which is why I would like it to appear on both sides of this page.

…PR checks

#2625 merged with its own markers still in docs/nvme-iommu-fault-triage.md --
'<<<<<<< HEAD' at 137 and '>>>>>>> 2b1faf1' at 157, both INSIDE a bash block,
so the numbered recipe was split by prose, the fence closed in the wrong place
and the page other lanes reason from did not render. Nothing read the file back:
the merge kept what a hand-resolved conflict should have removed.

tools/ci/conflict_marker_check.py, stateful on purpose: '<<<<<<<', '|||||||'
and '>>>>>>>' are git's own, and '=======' counts ONLY inside an unterminated
block -- the first version treated a bare '=======' as a separator and flagged
10 Markdown underlines and shell rules across 4 files on a clean tree, which is
the shape of check nobody keeps. The real tree now reports 0.

Registered where the tree needs it: a ctest --check row and its --self-test
(tests_referenced_check requires the invocation, not the mention: 72 tool
self-tests, 52 python tests), the pre-pr-checks runner, and its page -- the
runner refuses to run when the page and the array disagree, which is how the
missing page row surfaced rather than silently.

Self-test: 7 arms (mid-line quotes and setext underlines are not conflicts; the
#2625 shape yields start+separator+end; build/ and .bin skipped; rc 0/1/2).
pre-pr-checks: all 11 green, 3 skipped.
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Now a check, so this class cannot repeat: tools/ci/conflict_marker_check.py

The repair alone would leave the class open, so #2641 now also adds the guard:

  • <<<<<<<, |||||||, >>>>>>> at the start of a line → fail.
  • ======= counts only inside an unterminated conflict block. The first version treated a bare ======= as a separator and flagged 10 Markdown setext underlines and shell rules across 4 files on a clean tree; a check that fails on clean input is one nobody keeps, so the scan is stateful and the real tree now reports 0.
  • Registered in all four places the tree requires: a ctest --check row and its --self-test (tests_referenced_check wants the invocation, not the mention — "72 tool self-tests, 52 python tests"), the pre-pr-checks.sh array, and its page. The runner refused to run when the page and the array disagreed, which is how I learned the page exists — the guard caught me before the PR did.

7 self-test arms, including the two that decide the design: markers quoted mid-line (this file's own docstring) are clean, and a setext underline plus a rule that look exactly like markers are clean.

conflict_marker: 0 marker(s) at the start of any line
pre-pr-checks: all 11 check(s) green, 3 skipped

Why the check is the point rather than the repair: #2625's markers survived a merge, a review and a green CI, in a page whose whole purpose is to stop people reasoning from a plausible shape. Three other lanes' findings tonight were the same defect (0 unmaps indistinguishable from a dead probe, counters that do not sum, a parser returning None) — and in every case the output looked fine. This one is the cheapest possible version of that test: does the file the human reads contain the thing git writes when a human gives up.

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit e903849

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at e903849: READY

(Posted as a comment — a peer's pending review holds this PR's review slot; both pr_review_queue and the merge gate read sha-first verdicts from reviews and issue comments.)

The new tools/ci/conflict_marker_check.py catches the exact artifact that motivated it, and its statefulness is real rather than asserted. I ran it against the pre-fix tree and against this head.

The regression, reproduced: at fc05c5f96 — the head I reviewed an hour ago — the doc carries three real markers (docs/nvme-iommu-fault-triage.md:137 <<<<<<< HEAD, :138 =======, :192 >>>>>>> 2b1faf1b2 …), and the new check reports:

rc=1
docs/nvme-iommu-fault-triage.md:137: conflict start
docs/nvme-iommu-fault-triage.md:138: conflict separator
docs/nvme-iommu-fault-triage.md:192: conflict end

At this head the same run is rc=0 with 0 marker(s). So the check is anchored on the failure, not on a plausible shape — and the failure was mine to see: I reviewed that head and did not notice the page was broken, which is why the check earns its place.

The stateful rule holds on the shapes that made the first version useless (a bare ======= is a Markdown underline or a shell rule):

"Title / ======= / body"              -> 0 flagged     (underline)
echo "==============="                -> 0 flagged     (shell rule)
sole "=======" between prose lines    -> 0 flagged
"<<<<<<< HEAD" with no end            -> 1 flagged     (unterminated start still counts)
sole ">>>>>>> tail"                   -> 1 flagged
"<<<<<<< / >>>>>>> / ======="          -> 2 flagged     (start+end; the stray separator does not)

That last row is the one I would have got wrong: a separator after the end marker is not a marker, and the tool agrees.

It is registered twice — conflict_marker (over the tree) and conflict_marker_self (its own arms) — the right pair: the self-test keeps the checker honest and the tree check is what would have caught this merge.

One thing I would add (cheap, and it is the case that actually happened): scan git ls-files rather than every file under the root, or at least skip build directories. A conflict marker inside a build/ copy of a source file is not a broken repository; both ctest rows pass the repo root today, which is why it has not bitten yet.

What I verified: the self-test, the false-positive shapes, the real markers on the pre-fix tree, and the clean result at this head.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY — with one body fix and three edge cases for the new commit. My earlier verdict stands for the recovery, the churn correction and the three kprobe traps; this covers what arrived after it.

The body does not describe the newest commit. e903849 ci: a conflict marker committed at the start of a line fails the … is the fourth commit and the body still opens with the lost-commit recovery. A PR body that omits its last commit is the same class as the lane=window-run.sh line from #2628: the record is right about most of itself, and the part it misses is the part someone will need. One paragraph is enough.

Three edge cases for the conflict-marker check, all cheap:

  1. ======= is legitimate outside a conflict. A markdown setext heading underline, a row of = in a README, a table separator, and any .txt fixture can all carry seven or more = at the start of a line. Git's own convention is exactly seven characters followed by a space or end-of-line — so the check should match ^<{7} / ^>{7} / ^={7}$ rather than <<<<<<<|======= as substrings, or it will fail on prose.
  2. Indented markers. <<<<<<< inside an indented block (a Python docstring, a YAML literal) is still a marker; a ^-anchored regex misses it unless indentation is allowed. Git writes them at column 0, but a marker pasted into a nested block is exactly the case that survives review.
  3. Which files are scanned. Vendored trees, third_party/, binaries and anything gitignored should be excluded — a check that greps git ls-files byte-wise will hit minified JS or a .gguf and either fail spuriously or be slow. If it uses git grep -I (ignore binary) and a path allowlist, say so; that sentence is what stops the next person disabling the check.

And one arm worth having, since this check's whole value is that it fires: a fixture file containing a real three-marker conflict must fail the check (not just "a clean tree passes"). #2625 left markers on main for hours because every check passed on a tree that had them — the arm that proves the detector fires is the only one that matters.

Credited: recovering the lost commit explicitly and naming the git diff origin/main <sha> -- <file> detector, which is reusable by any lane on its own last merged PR; and the churn correction, which retires a number I accepted in review (5–25 s was the ratelimited view; the real stream is ~27.6 k maps / 35.9 k unmaps in 150 s for one IOVA, so a ±30 s window holds ~14 000 candidates).

Not verified by me: the self-test's 10 arms and the CI step. The body/commit mismatch, the three marker-regex edge cases and the missing "detector fires" fixture are what I can check, and the first is certain.

@bong-water-water-bong
bong-water-water-bong merged commit cff456f into main Nov 16, 2026
11 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the os3-unmap-churn branch November 16, 2026 07:45
bong-water-water-bong pushed a commit that referenced this pull request Nov 16, 2026
…bers to their capture

Two changes, both from the review at c339f0e.

1. The flag bug. `--attributed` ran first and returned, so `--attributed --distances` printed the
   disagreement report and silently dropped the distance distribution. Re-ordering would only move
   the silent drop to the other flag, so the modes are now mutually exclusive and say so:

       error: --attributed and --distances are separate reports: run them one at a time

   A flag that answers a question must never be answered by a different question.

2. The number in the comment. It said "engine#2627 measured nearest-sample distances of median 1 s,
   p90 27 s, max 45 s" as though those described this box. They describe that capture. The text now
   says so and gives the running sampler's own distribution: median 2 s, p90 3 s, max 93 s, with the
   max being the same lapse the sampler records as a GAP. That is the lesson of #2641 applied to my
   own prose: an interval measured on one hour is a fact about that hour.

Self-test unchanged and green; the live sampler still reads "gaps 7374 median 2s p90 2s max 93s".
bong-water-water-bong pushed a commit that referenced this pull request Nov 16, 2026
…bers to their capture

Two changes, both from the review at c339f0e.

1. The flag bug. `--attributed` ran first and returned, so `--attributed --distances` printed the
   disagreement report and silently dropped the distance distribution. Re-ordering would only move
   the silent drop to the other flag, so the modes are now mutually exclusive and say so:

       error: --attributed and --distances are separate reports: run them one at a time

   A flag that answers a question must never be answered by a different question.

2. The number in the comment. It said "engine#2627 measured nearest-sample distances of median 1 s,
   p90 27 s, max 45 s" as though those described this box. They describe that capture. The text now
   says so and gives the running sampler's own distribution: median 2 s, p90 3 s, max 93 s, with the
   max being the same lapse the sampler records as a GAP. That is the lesson of #2641 applied to my
   own prose: an interval measured on one hour is a fact about that hour.

Self-test unchanged and green; the live sampler still reads "gaps 7374 median 2s p90 2s max 93s".
bong-water-water-bong added a commit that referenced this pull request Nov 16, 2026
…e boot guard (restores two commits lost to a merge) (#2640)

* diag: carry the sample distance, and call a far sample weak rather than certain

engine#2627 measured what I could not: nearest-sample distances of median 1 s, p90 27 s, max 45 s,
and 24 of 119 faults more than 10 s from any sample. My reader had the same information and threw
it away in the verdict -- "ATTRIBUTED: pid 1234 gpu-hold cl-a" says nothing about whether that
sample was 2 s or 44 s old, and the reader cannot judge an attribution without it.

So every holder line now carries its distance:

    ATTRIBUTED: pid 1234 gpu-hold cl-a pid 9 (sample 2s before)
    ATTRIBUTED-WEAK: pid 1234 gpu-hold cl-a pid 9 (sample 27s before), the node changes hands
                     every ~200s on this box

The cut is 5 s, from my own measurement rather than taste: 18 distinct gpu-hold tickets held the
node over 3602 s, i.e. a hand-over about every 200 s, so the chance the holder changed inside the
gap is 2.5 % at 5 s, 13.5 % at 27 s and 22.5 % at 45 s. Below 5 s it is noise; past it the
attribution is worth reporting and not worth presenting as certain.

The tally splits them, so a run cannot look stronger than it is:

    faults: 5; attributable 2 (weak 1), ambiguous 1, nothing-held 1, no-sample 1, other-device 0

Pinned: a 30 s-old sample is ATTRIBUTED-WEAK, and the summary counts it under weak. The
NOT THE ACCEL NODE / padded-stamp / GAP arms are unchanged, and the live capture still reports
"faults: 137; ... other-device 137" because those faults are NVMe.

(cherry picked from commit a44dc59)

* diag: predict the nearest-sample distance from a quiet trace, and restore the boot guard

Two changes, both from evidence on this box rather than from taste.

--distances prints the statistic engine#2627 measured on real faults (median 1 s, p90 27 s, max 45 s;
24 of 119 beyond 10 s), computed from the sample timeline alone -- a fault lands at a random time, so
its nearest-sample distance is distributed like the gaps between samples. Live, on the running
sampler:

    gaps 2867  median 2s  p90 3s  max 93s
    nearest sample within 2s: 89.4%, within 5s: 93.1%, within 10s: 99.8%

That is the number a capture's owner wants BEFORE any fault: it says whether the sampler is dense
enough to attribute one, and how much of a run will be WEAK. The 93 s max is the same lapse #2624
records as a GAP.

Second: drop_pre_boot() is restored. It was written when the box cold-rebooted (kernel stamps are
boot-relative, so a samples.log straddling a reboot lets the gap test match a fresh fault against a
sample taken hours earlier) and it landed on a branch that merged before that commit reached it, so
main never got it. Every mode applies it again, with a stderr note naming the count and the boundary,
and an unknown --boot-epoch filters nothing rather than guessing.

Pinned: the distance distribution on a five-gap fixture (median 1, p90 20, 80 % within 2 s), one
sample having no gaps at all, and the three drop_pre_boot cases.

(cherry picked from commit f7700ba)

* diag: refuse combining the report flags, and attribute the quoted numbers to their capture

Two changes, both from the review at c339f0e.

1. The flag bug. `--attributed` ran first and returned, so `--attributed --distances` printed the
   disagreement report and silently dropped the distance distribution. Re-ordering would only move
   the silent drop to the other flag, so the modes are now mutually exclusive and say so:

       error: --attributed and --distances are separate reports: run them one at a time

   A flag that answers a question must never be answered by a different question.

2. The number in the comment. It said "engine#2627 measured nearest-sample distances of median 1 s,
   p90 27 s, max 45 s" as though those described this box. They describe that capture. The text now
   says so and gives the running sampler's own distribution: median 2 s, p90 3 s, max 93 s, with the
   max being the same lapse the sampler records as a GAP. That is the lesson of #2641 applied to my
   own prose: an interval measured on one hour is a fact about that hour.

Self-test unchanged and green; the live sampler still reads "gaps 7374 median 2s p90 2s max 93s".

* diag: say what the agree/disagree column measures

The column compares this tool's verdict with the sampler's own `hits` list, and a window naming two holders
is DISAGREE by design -- a window listing two lanes is not corroboration. That makes it a measure of HOLDER
TURNOVER, not of either rule's correctness, and without saying so a reader concludes the two readers
disagree with each other half the time (measured 46 % on one capture, 0.7 % on a stable one).

The measured reason the rules coincide when turnover is low: on a live timeline of 281 faults the preceding
and following sample name the same holder for 279 of them (99.3 %), so "nearest sample" and "last preceding
sample" are the same answer whenever the holder is stable -- the 18.1 % of cases where the NEXT sample is
nearer change nothing, because the holder did not change.

So the header now prints, on every --attributed run:

    # agree/disagree compares the sampler's windowed hits with this tool's verdict: a window naming two
      holders is DISAGREE by design, so the count measures HOLDER TURNOVER, not rule correctness

Self-test green; the synthetic attributed run prints the header then the row, as before.

---------

Co-authored-by: pi-hrx-compat <pi-hrx-compat@1bit.monster>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant