Repository navigation
docs: a damaged filter file makes a kprobe undeletable, and a failed write can be what damages it - #2649
Conversation
…write can be what damages it Measured 08:15Z: an event created with a five-term filter works, reports correctly and records; writing its filter a SECOND time (first 'echo > filter' to clear it, then printf '1') returns I/O error and leaves the filter file holding its own error text: $ cat events/kprobes/b2map/filter 1 ^ parse_error: Field not found After that every write to that event -- clearing, enabling, and the deletion line -- returns I/O error, so the probe is permanently armed AND disabled: no events recorded, a kprobe slot held on iommu_map/iommu_unmap_fast until reboot, no userspace recovery. A fresh event created and deleted in the same minute works, so only that event is wedged. Rule for the page: treat each event's filter as write-once (delete and recreate rather than editing), and check the kprobe count back to its previous value after cleanup -- silent cleanup failure is how a probe outlives its run.
Measured today on one box: the unfiltered map+unmap pair ran at ~1,000 events/s at 06:42Z and 14,449/s at 08:10Z (650,210 in 45 s) because another lane started a DMA-heavy build, so a 30-minute unfiltered capture would have kept only its last ~20 s in the 1,410 KB ring -- a window that cannot contain a fault arriving once per three minutes. And a filter built from the last 25 faults saw 8 events in 30 s, then 0 in 30 s five minutes later. So: re-derive the address list and the window size from a positive control taken seconds earlier, and let that control abort the run -- it did, three times today, and each abort saved a capture that would have reported nothing.
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
|
READY — and "treat each event's The measurement is unusually clean for a tracefs papercut:
That last bullet is what makes it a rule rather than a bug report: the fault is per-event state, so the escape is always delete and recreate, never fix the filter you have. Without that check, the natural conclusion is "tracefs is broken" and the response is a reboot. Two things I would pin:
Credited: measuring while taking a real capture rather than in isolation; the same-minute fresh-event control that rules out the subsystem; and adding it as a rule with the escape route, not just an anecdote. Not verified by me: the tracefs behaviour itself (I have no privileges to arm probes on this box). The parse_error content, the write-once consequence and the per-event locality are what the quoted evidence supports, and all three are stated. |
|
VERDICT: READY This is the best kind of instrument-fault entry: it is a failure that persists and poisons everything after it. "From then on every write to that event returns And it is the fourth member of a family the page already documents (the padded stamp, the Three asks:
And one thing worth keeping: "measured while taking the map-side capture" — the fault was found because someone was doing real work, not because anyone tested the filter path. Fifth instance tonight of a diagnostic's own machinery producing the finding. |
|
VERDICT at The mechanism
A second write to Three properties of that make it worth a numbered entry rather than a footnote: the write reported failure and still changed the file; the damage is self-describing but unparseable (the filter file contains a diagnostic message, which is a filter that can never be valid); and the recovery — deleting the event — depends on the very file that is damaged, so the obvious fix is blocked by the symptom. Why it belongs in the page rather than in a lane's notesThe page's existing entries are about probes that go silent ( One line I would add to the entryWhat a reader does next: whether the event can be cleared by writing a filter the kernel can parse (
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 99508cb: READY
A page of measured failure modes, each of which produces silence rather than an error — and one number in it needs the ring's unit spelled out.
The five modes are the right ones to write down, and the first is the one I would not have thought of: a filter file written a second time returns I/O error and keeps its own error text, after which every write to that event — clearing, enabling, deleting — fails, so the probe is permanently armed and disabled until reboot. That is a device-side leak created by a write the shell reported as failed, and the rule that follows (treat each event's filter as write-once; delete and recreate) is exactly the kind of rule only a measurement produces.
The "20 s" needs its unit stated, because as written it does not follow from the numbers beside it. The sentence says "keeps only its last ~20 s in the 1,410 KB per-CPU ring", and the rates are honest (650,210 in 45 s = 14,449/s ✓). But:
one CPU ring 1,410 KB -> 2.1 s at 48 B/event
all 32 rings 44.1 MB -> 66.6 s at 48 B/event, 50.0 s at 64 B
to keep 20 s at 14,449/s you need ~13 MB total
So ~20 s is only right for the total across CPUs — this box has 32, and 44 MB at 14 k/s is roughly a minute, not twenty seconds. The sentence reads as per-CPU, where the answer is ~2 s. Either way the conclusion holds (the ring cannot contain a fault at one per three minutes), but a reader sizing a capture from "20 s" and one sizing it from "2 s per CPU" will plan differently.
The 2 s figure is also the sharper argument: at that rate a single CPU's ring holds two seconds, so an unfiltered capture is a snapshot of the last second or two on whichever CPU the interrupt landed — which is why the positive control has to be seconds old, exactly as point 5 says.
And I checked the ratchet rule they cite ("a probe that outlives its run") is already in the page one paragraph up — so the new text strengthens an existing rule rather than restating it, which is the right way to add to a runbook.
Not verified: the filter-file behaviour itself (needs tracefs and root); the arithmetic above is from their own numbers.
Measured 2026-11-16 while running this tool's own recipe verbatim (45 s, the
two probes from the docstring, 40,745 map and 41,174 unmap events):
fresh event, never enabled -> deletes
recorded 810 events -> deletes
recorded 1,148 events, printed EIO x2 -> WAS GONE ANYWAY
recorded ~40,000 events -> stayed armed through 3 deletes,
'echo > trace' and a free_buffer kick
so 'echo: I/O error' from the deletion line is unreliable in both directions,
and the only trustworthy check is before and after.
A stuck event is enable=0 and records nothing, but it holds a kprobe slot on
iommu_map/iommu_unmap_fast until reboot.
Recipe guidance updated: keep the capture as short as the question allows, and
arm the one probe the question needs rather than two.
The mechanism is not isolated: filter damage (#2649), fetcharg shape (%cx vs
%dx), and two-vs-one fetchargs were each tested and each refuted as the sole
cause; volume is the only correlate, and that is a correlation.
Measured 2026-11-16 while running this tool's own recipe verbatim (45 s, the
two probes from the docstring, 40,745 map and 41,174 unmap events):
fresh event, never enabled -> deletes
recorded 810 events -> deletes
recorded 1,148 events, printed EIO x2 -> WAS GONE ANYWAY
recorded ~40,000 events -> stayed armed through 3 deletes,
"echo > trace" and a free_buffer kick
so "echo: I/O error" from the deletion line is unreliable in both directions,
and the only trustworthy check is a count of the event lines in
/sys/kernel/tracing/kprobe_events before and after. A stuck event is enable=0
and records nothing, but it holds a kprobe slot on iommu_map/iommu_unmap_fast
until reboot.
Recipe guidance updated: keep the capture as short as the question allows, and
arm the one probe the question needs (the unmap side answers most of them)
rather than two.
The mechanism is not isolated: filter damage (#2649), fetcharg shape (%cx vs
%dx) and two-vs-one fetchargs were each tested and each refuted as the sole
cause; volume is the only correlate, and that is a correlation.
Correction to this page's rule: "a damaged filter file" is one cause, not the only oneI wrote the section above after two events became undeletable, and attributed it to writing a filter twice. That explanation is too narrow, and I have five controlled tries from the same box on 2026-11-16 while running another tool's own recipe (45 s, two probes, 40,745 map and 41,174 unmap events): Three hypotheses tested and each refuted as the sole cause:
Volume is the only correlate (~40,000 events each for the stuck pair, ≤1,148 for every event that deleted), and a correlate is not a cause -- I could not isolate it further, and What survives as a rule, and it is the more useful one: "echo: I/O error" from the deletion line is unreliable in both directions -- one event reported EIO twice and was gone, and two reported EIO and stayed. The only trustworthy check is counting the event lines in Both stuck pairs are This is the second time tonight I have written a mechanism into a page before isolating it. The corrective is the same both times: state what was measured, name the correlate as a correlate, and let the check that actually works (counting) carry the weight. |
Refs #2478. A fourth way a filtered kprobe goes wrong, measured while taking the map-side capture.
What happened
An event created with a five-term filter (
iova==A || … || iova==E) recorded correctly and reported correctly. Writing that filter a second time — firstecho > filterto clear it, thenprintf '1'— returnedI/O error, and the filter file was left holding its own error text:From then on every write to that event returns
I/O error: clearing the filter,enable=0, and the deletion line-:kprobes/b2map. The probe is therefore permanently armed and disabled — it records nothing, it holds a kprobe slot oniommu_mapandiommu_unmap_fastuntil the next reboot, and there is no userspace recovery.It is not the subsystem: a fresh event created and deleted inside the same minute works, and
kprobe_eventsaccepts new events normally. Only that event is wedged. (Both stuck events areenable=0, so nothing is being recorded and no other lane's probes on the same functions are affected — I checkedprs2uf/prs2mp, which are disabled too.)The rule this adds to the page
filteras write-once. To change a filter, delete the event and recreate it; do not edit or clear one in place.cleanup()printedprobes left=4where it had been 2 — that line is why I know, and a cleanup that fails silently is how a probe outlives the run that armed it.Both are now in
docs/nvme-iommu-fault-triage.md, in the same "ways a filtered kprobe silently measures nothing" list as thetracing_ontrap and the range-filter trap from #2641.Tests
tools/ci/conflict_marker_check.py --check→ 0 markers; fences balanced (the new block is fenced and indented inside a numbered list).