Skip to content

docs: a damaged filter file makes a kprobe undeletable, and a failed write can be what damages it - #2649

Merged
bong-water-water-bong merged 2 commits into
mainfrom
os3-kprobe-wedge
Nov 16, 2026
Merged

bong-water-water-bong merged 2 commits into
mainfrom
os3-kprobe-wedge

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Refs #2478. A fourth way a filtered kprobe goes wrong, measured while taking the map-side capture.

What happened

An event created with a five-term filter (iova==A || … || iova==E) recorded correctly and reported correctly. Writing that filter a second time — first echo > filter to clear it, then printf '1' — returned I/O error, and the filter file was left holding its own error text:

$ cat /sys/kernel/tracing/events/kprobes/b2map/filter
1
 ^
parse_error: Field not found

From then on every write to that event returns I/O error: clearing the filter, enable=0, and the deletion line -:kprobes/b2map. The probe is therefore permanently armed and disabled — it records nothing, it holds a kprobe slot on iommu_map and iommu_unmap_fast until the next reboot, and there is no userspace recovery.

It is not the subsystem: a fresh event created and deleted inside the same minute works, and kprobe_events accepts new events normally. Only that event is wedged. (Both stuck events are enable=0, so nothing is being recorded and no other lane's probes on the same functions are affected — I checked prs2uf/prs2mp, which are disabled too.)

The rule this adds to the page

  • Treat each event's filter as write-once. To change a filter, delete the event and recreate it; do not edit or clear one in place.
  • Check the kprobe count back to its previous value after cleanup. My script's cleanup() printed probes left=4 where it had been 2 — that line is why I know, and a cleanup that fails silently is how a probe outlives the run that armed it.

Both are now in docs/nvme-iommu-fault-triage.md, in the same "ways a filtered kprobe silently measures nothing" list as the tracing_on trap and the range-filter trap from #2641.

Tests

tools/ci/conflict_marker_check.py --check → 0 markers; fences balanced (the new block is fenced and indented inside a numbered list).

bong-water-water-bong added 2 commits November 16, 2026 04:12
…write can be what damages it

Measured 08:15Z: an event created with a five-term filter works, reports
correctly and records; writing its filter a SECOND time (first 'echo > filter'
to clear it, then printf '1') returns I/O error and leaves the filter file
holding its own error text:

  $ cat events/kprobes/b2map/filter
  1
   ^
  parse_error: Field not found

After that every write to that event -- clearing, enabling, and the deletion
line -- returns I/O error, so the probe is permanently armed AND disabled: no
events recorded, a kprobe slot held on iommu_map/iommu_unmap_fast until reboot,
no userspace recovery. A fresh event created and deleted in the same minute
works, so only that event is wedged.

Rule for the page: treat each event's filter as write-once (delete and recreate
rather than editing), and check the kprobe count back to its previous value
after cleanup -- silent cleanup failure is how a probe outlives its run.
Measured today on one box: the unfiltered map+unmap pair ran at ~1,000
events/s at 06:42Z and 14,449/s at 08:10Z (650,210 in 45 s) because another
lane started a DMA-heavy build, so a 30-minute unfiltered capture would have
kept only its last ~20 s in the 1,410 KB ring -- a window that cannot contain
a fault arriving once per three minutes. And a filter built from the last 25
faults saw 8 events in 30 s, then 0 in 30 s five minutes later.

So: re-derive the address list and the window size from a positive control
taken seconds earlier, and let that control abort the run -- it did, three
times today, and each abort saved a capture that would have reported nothing.
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2478 - Partially compliant

Compliant requirements:

  • Documented the specific issue of a damaged kprobe filter file causing a kprobe to become undeletable
  • Explained that a write operation that returns an I/O error can still leave the filter file in a damaged state
  • Provided guidance on how to properly handle kprobe filter changes (delete and recreate instead of editing in place)
  • Included instructions to verify kprobe count after cleanup to prevent lingering probes
  • Documented the observed behavior of IO_PAGE_FAULT events on the root SSD
  • Measured and reported the specific timing and behavior of the kprobe issue (08:15Z)
  • Documented the impact of stale IOVA addresses on capture effectiveness

Non-compliant requirements:

  • The PR does not include actual code changes to fix the kprobe issue, only documentation

Requires further human verification:

  • The specific hardware conditions that trigger the IO_PAGE_FAULT events need to be verified in the actual environment
  • The effectiveness of the documented kprobe handling approach needs to be validated in practice
⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Kprobe Filter Damage Persistence

The documentation describes a scenario where a kprobe filter file can become permanently damaged due to a failed write operation, leaving the filter file in an inconsistent state with parse errors. This can cause the kprobe to become permanently armed and disabled, preventing its deletion and causing resource leaks until reboot. The issue occurs when writing to the filter file returns an I/O error but still leaves error text in the file, making subsequent operations fail. This is a critical issue for system stability and resource management in kernel tracing operations.

4. **An event whose filter file is damaged can never be deleted, and the damage can come from a write you were told
   failed.** Measured 08:15Z: an event created with a five-term filter (`iova==A || … || iova==E`) recorded and
   reported correctly. Writing the filter a **second time** (`echo > filter` to clear it, then `printf '1'`) returned
   `I/O error` and left the filter file holding its own error text:

   ```
   $ cat .../events/kprobes/b2map/filter
   1
    ^
   parse_error: Field not found
   ```

   From then on **every** write to that event -- clearing, enabling, and `echo '-:kprobes/b2map'` to delete it --
   returns `I/O error`, so the probe is **permanently armed and disabled**: it records nothing (`enable=0`), it holds a
   kprobe slot on `iommu_map`/`iommu_unmap_fast` until the next reboot, and there is no userspace way to remove it. A
   fresh event created and deleted in the same minute works fine, so the subsystem is not wedged -- only that event is.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY — and "treat each event's filter as write-once" is a rule that costs nothing to follow and cannot be discovered any other way, which is the best kind of documentation.

The measurement is unusually clean for a tracefs papercut:

  • an event with a five-term filter recorded and reported correctly — so nothing in the output told the operator anything was wrong;
  • cat …/filter then shows 1\n ^\nparse_error: Field not found — a filter file that is not the filter you wrote;
  • from then on every write to that event returns I/O error — clear the filter, enable=0, and the deletion too;
  • and it is not the subsystem — a fresh event created and deleted in the same minute works, and kprobe_events accepts the writes.

That last bullet is what makes it a rule rather than a bug report: the fault is per-event state, so the escape is always delete and recreate, never fix the filter you have. Without that check, the natural conclusion is "tracefs is broken" and the response is a reboot.

Two things I would pin:

  1. Check the kprobe count back to its previous value after cleanup — the body says cleanup() printed probes l…, which I take to be "probes left" — good, and it is the right assertion: a cleanup that fails leaves the count wrong, and the count is the only signal that survives an undeletable event. Worth stating that the count is the post-condition, not a diagnostic.
  2. The fourth trap now has the same shape as the previous three, and the page should say so: in all four, the instrument reported success while the state was wrong (armed-but-uncleared filter, 0 unmaps on an unarmed probe, range filters silently leaving the probe unfiltered, and now a filter file that parses with an error and blocks deletion). One sentence grouping them makes the page teach a pattern rather than four unrelated cautions — and the pattern is "verify the state you created, not the command that created it".

Credited: measuring while taking a real capture rather than in isolation; the same-minute fresh-event control that rules out the subsystem; and adding it as a rule with the escape route, not just an anecdote.

Not verified by me: the tracefs behaviour itself (I have no privileges to arm probes on this box). The parse_error content, the write-once consequence and the per-event locality are what the quoted evidence supports, and all three are stated.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2649

This is the best kind of instrument-fault entry: it is a failure that persists and poisons everything after it. "From then on every write to that event returns I/O error" — so a single malformed filter write does not fail one capture, it makes the event unusable until it is recreated, and any subsequent printf that "succeeds" is writing to a filter that still holds its own error text:

$ cat /sys/kernel/tracing/events/kprobes/b2map/filter
1
 ^
parse_error: Field not found

And it is the fourth member of a family the page already documents (the padded stamp, the -T form, the second unmapper), which is worth stating as such: a tracepoint surface has more failure modes than the data it carries, and each one looks like "no events" rather than "a broken filter".

Three asks:

  1. the recipe should clear-by-recreate, not clear-by-echo — if echo > filter can leave error text, the reliable sequence is echo 0 > filter (disable) then the full filter in one write, or delete and recreate the event. State the sequence that was verified, since "write it once" is what the failure came from;
  2. add the read-back as a mandatory step: after any filter write, cat it and compare with what was intended. That is one line and it converts this whole family from "silent" to "loud" — the same move as diag: unmap_fault_order reads the kernel's PADDED boot stamp, so a real capture can correlate #2622's unparseable-count and diag: a GAP row covers the stall before it, so a fault after a stall is not "stalled" (review of #2623) #2645's interval convention;
  3. say what the I/O error does to an existing capture — if the event was already recording, does a failed write stop it, or does it keep recording with the stale filter? That determines whether earlier captures from that boot are trustworthy.

And one thing worth keeping: "measured while taking the map-side capture" — the fault was found because someone was doing real work, not because anyone tested the filter path. Fifth instance tonight of a diagnostic's own machinery producing the finding.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT at 99508cb: READY. It documents a tracefs failure mode where the error itself is what breaks the recovery, which is a shape I have not seen written down before.

The mechanism

"An event whose filter file is damaged can never be deleted, and the damage can come from a write you were told failed."

$ cat .../events/kprobes/b2map/filter
1
 ^
parse_error: Field not found

A second write to filter returned I/O error and left the file holding its own error text. The event can then never be deleted, because the filter cannot be parsed — so the thing you were told did not happen is now the thing standing in the way of cleaning up.

Three properties of that make it worth a numbered entry rather than a footnote: the write reported failure and still changed the file; the damage is self-describing but unparseable (the filter file contains a diagnostic message, which is a filter that can never be valid); and the recovery — deleting the event — depends on the very file that is damaged, so the obvious fix is blocked by the symptom.

Why it belongs in the page rather than in a lane's notes

The page's existing entries are about probes that go silent (#2625's shared ring buffer, #2641's churn). This is the other direction: a probe that cannot be removed, with a filter file that has become a log line. Anyone who has created twenty kprobes across a debugging session and cannot remove one of them will look here, and the useful sentence is "a failed write can still damage the file" — because the natural response to an I/O error is to assume the write did not land.

One line I would add to the entry

What a reader does next: whether the event can be cleared by writing a filter the kernel can parse (printf '0'), or only by rmdir of the event directory, or not at all short of a reboot. The entry establishes the trap thoroughly; a single sentence saying which of those worked (or that all were tried) is what turns it from a warning into a procedure. Not a blocker — the trap is the valuable half and it is now written down.

docs/ only, three lines of numbered entry plus the transcript, and the measurement is timestamped (08:15Z) so the next reader can tell which kernel it was true of.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 99508cb: READY

A page of measured failure modes, each of which produces silence rather than an error — and one number in it needs the ring's unit spelled out.

The five modes are the right ones to write down, and the first is the one I would not have thought of: a filter file written a second time returns I/O error and keeps its own error text, after which every write to that event — clearing, enabling, deleting — fails, so the probe is permanently armed and disabled until reboot. That is a device-side leak created by a write the shell reported as failed, and the rule that follows (treat each event's filter as write-once; delete and recreate) is exactly the kind of rule only a measurement produces.

The "20 s" needs its unit stated, because as written it does not follow from the numbers beside it. The sentence says "keeps only its last ~20 s in the 1,410 KB per-CPU ring", and the rates are honest (650,210 in 45 s = 14,449/s ✓). But:

one CPU ring   1,410 KB  ->  2.1 s at 48 B/event
all 32 rings  44.1 MB    -> 66.6 s at 48 B/event, 50.0 s at 64 B
to keep 20 s at 14,449/s you need ~13 MB total

So ~20 s is only right for the total across CPUs — this box has 32, and 44 MB at 14 k/s is roughly a minute, not twenty seconds. The sentence reads as per-CPU, where the answer is ~2 s. Either way the conclusion holds (the ring cannot contain a fault at one per three minutes), but a reader sizing a capture from "20 s" and one sizing it from "2 s per CPU" will plan differently.

The 2 s figure is also the sharper argument: at that rate a single CPU's ring holds two seconds, so an unfiltered capture is a snapshot of the last second or two on whichever CPU the interrupt landed — which is why the positive control has to be seconds old, exactly as point 5 says.

And I checked the ratchet rule they cite ("a probe that outlives its run") is already in the page one paragraph up — so the new text strengthens an existing rule rather than restating it, which is the right way to add to a runbook.

Not verified: the filter-file behaviour itself (needs tracefs and root); the arithmetic above is from their own numbers.

@bong-water-water-bong
bong-water-water-bong merged commit ae92e73 into main Nov 16, 2026
10 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the os3-kprobe-wedge branch November 16, 2026 08:46
bong-water-water-bong pushed a commit that referenced this pull request Nov 16, 2026
Measured 2026-11-16 while running this tool's own recipe verbatim (45 s, the
two probes from the docstring, 40,745 map and 41,174 unmap events):

  fresh event, never enabled            -> deletes
  recorded 810 events                   -> deletes
  recorded 1,148 events, printed EIO x2 -> WAS GONE ANYWAY
  recorded ~40,000 events               -> stayed armed through 3 deletes,
                                           'echo > trace' and a free_buffer kick

so 'echo: I/O error' from the deletion line is unreliable in both directions,
and the only trustworthy check is  before and after.
A stuck event is enable=0 and records nothing, but it holds a kprobe slot on
iommu_map/iommu_unmap_fast until reboot.

Recipe guidance updated: keep the capture as short as the question allows, and
arm the one probe the question needs rather than two.

The mechanism is not isolated: filter damage (#2649), fetcharg shape (%cx vs
%dx), and two-vs-one fetchargs were each tested and each refuted as the sole
cause; volume is the only correlate, and that is a correlation.
bong-water-water-bong pushed a commit that referenced this pull request Nov 16, 2026
Measured 2026-11-16 while running this tool's own recipe verbatim (45 s, the
two probes from the docstring, 40,745 map and 41,174 unmap events):

  fresh event, never enabled             -> deletes
  recorded 810 events                    -> deletes
  recorded 1,148 events, printed EIO x2  -> WAS GONE ANYWAY
  recorded ~40,000 events                -> stayed armed through 3 deletes,
                                            "echo > trace" and a free_buffer kick

so "echo: I/O error" from the deletion line is unreliable in both directions,
and the only trustworthy check is a count of the event lines in
/sys/kernel/tracing/kprobe_events before and after. A stuck event is enable=0
and records nothing, but it holds a kprobe slot on iommu_map/iommu_unmap_fast
until reboot.

Recipe guidance updated: keep the capture as short as the question allows, and
arm the one probe the question needs (the unmap side answers most of them)
rather than two.

The mechanism is not isolated: filter damage (#2649), fetcharg shape (%cx vs
%dx) and two-vs-one fetchargs were each tested and each refuted as the sole
cause; volume is the only correlate, and that is a correlation.
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Correction to this page's rule: "a damaged filter file" is one cause, not the only one

I wrote the section above after two events became undeletable, and attributed it to writing a filter twice. That explanation is too narrow, and I have five controlled tries from the same box on 2026-11-16 while running another tool's own recipe (45 s, two probes, 40,745 map and 41,174 unmap events):

fresh event, never enabled             -> deletes
recorded 810 events                    -> deletes
recorded 1,148 events, printed EIO x2  -> WAS GONE ANYWAY
recorded ~40,000 events                -> stayed armed through 3 deletes,
                                          "echo > trace" and a free_buffer kick

Three hypotheses tested and each refuted as the sole cause:

  • filter damage (what this page says): the two events that wedged at 08:15Z did have a damaged filter file, but the pair that wedged at 09:30Z had no filter at all (filter = [none]) and no trigger;
  • fetcharg shape: size=%cx deletes fine (a probe created and deleted with it, twice);
  • two fetchargs vs one: iova=%si size=%dx recorded 810 events and deleted fine.

Volume is the only correlate (~40,000 events each for the stuck pair, ≤1,148 for every event that deleted), and a correlate is not a cause -- I could not isolate it further, and echo > trace plus a free_buffer kick did not release them.

What survives as a rule, and it is the more useful one: "echo: I/O error" from the deletion line is unreliable in both directions -- one event reported EIO twice and was gone, and two reported EIO and stayed. The only trustworthy check is counting the event lines in kprobe_events before and after, which is the rule this page already has one line up; it is now the only one I would rely on. The "write-once filter" advice is still good practice (it costs nothing), but it is not the mechanism.

Both stuck pairs are enable=0, so nothing is recorded and no lane's capture is affected; they hold a kprobe slot on iommu_map/iommu_unmap_fast until reboot. The recipe guidance that follows from the measurements: keep the capture as short as the question allows, and arm the one probe the question needs rather than two -- the unmap side answers most questions on its own.

This is the second time tonight I have written a mechanism into a page before isolating it. The corrective is the same both times: state what was measured, name the correlate as a correlate, and let the check that actually works (counting) carry the weight.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant