Skip to content

The four ways a capture lied here today, as a pre-flight pattern (Refs #2478) - #2531

Merged
bong-water-water-bong merged 2 commits into
mainfrom
docs/2478-capture-pattern
Nov 15, 2026
Merged

bong-water-water-bong merged 2 commits into
mainfrom
docs/2478-capture-pattern

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Refs #2478: the four ways a trace capture lied here today, as a pre-flight pattern

Working the fault trace for #2478 I produced three invalid results, and each one looked exactly like a finding:

mistake symptom cost
echo > .../filter to clear a kprobe filter parse error, old filter stayed, matched nothing a 12-min capture reported "0 unmaps"
probing iommu_unmap from memory 0 hits in 5 s on a box doing heavy I/O a wrong conclusion about which unmap path exists
enabling a probe whose definition my own cleanup had removed silence a 20-min capture measured nothing
sudo cp output then grep 2>/dev/null silence — the file was root-0600 the data was there; only the read failed

All four present as an empty result, which is indistinguishable from the finding you might legitimately be looking
for — the reason this is a docs change and not just a lesson learned.

The page gains the ten-line pattern that catches all of them: prove the plumbing on a known-hot symbol before trusting
any other probe; pre-flight inside the long run and abort instead of sleeping; chmod the copied output; read it
back before analysing. Plus the table above, so the next reader recognises the symptom rather than rediscovering it.

No new tool: the repo has no capture helper and half of one at the end of a long session is worse than a documented
pattern. The runbook this extends is already on main and linked from tools/diag/iommu_fault_census.py.

Writing the fault-side trace for #2478 I produced four invalid results and one correct one? No -- three invalid results,
each of which looked exactly like a finding: a filter that never cleared, a symbol that is never called, a probe whose
definition had been removed by my own cleanup, and an output file made unreadable by 'sudo cp' and then grepped with
stderr suppressed. All four present as an empty result.

The page gains the ten-line pattern that catches all four (prove the plumbing on a known-hot symbol, pre-flight inside
the long run so it aborts instead of sleeping, chmod the output, read it back before analysing) and the table of what
each mistake looked like. The lesson is one sentence: an empty result is indistinguishable from the finding you were
looking for, so the only defence is to prove the instrument before trusting its silence.
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2531

"Three invalid results, and each one looked exactly like a finding" is the most useful sentence in this PR, and it is the reason this belongs in docs/ rather than in #2478's comment thread: an investigation that produces wrong answers in the shape of right ones is worse than one that produces none, because it consumes the next person's confidence as well as their time.

Three things that would make the pre-flight pattern complete:

  1. each of the four should name the tell — the specific thing in the capture that distinguishes "this trace is invalid" from "this trace shows X". "The capture lied" is a warning; "the first event's timestamp precedes the run start, so the buffer wrapped and the head is missing" is a check the reader can perform in one glance;
  2. state the cheapest guard for each, in the order a triager would run them — if one of the four can be excluded by a single command, that command should be the first line, so a reader who only has two minutes still avoids the commonest mistake;
  3. keep the three invalid results themselves — what each looked like, and what revealed it. That is the part that teaches the pattern; a list of traps without the near-misses is a list someone will believe themselves immune to.

And one thing specific to #2478: its evidence is IO_PAGE_FAULT counts against a baseline of 2.3/min (#2524), and the whole point of a rate is that it needs the same instrument on both sides. If any of the four ways a capture lied here changed the rate rather than the events, say so explicitly — a 2.3/min baseline measured through one of these four is exactly how #2524's "the HMB is a bystander" conclusion could be right for the wrong reason.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

PR Reviewer Guide 🔍

(Review updated until commit be8ca31)

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2478 - Partially compliant

Compliant requirements:

  • Document the four ways a trace capture can lie, as observed in the NVMe IOMMU fault investigation
  • Provide a pre-flight validation pattern to catch these issues before trusting trace data
  • Explain how each of the four mistakes resulted in empty results that are indistinguishable from legitimate findings
  • Include a practical bash script pattern to validate trace captures before analysis

Non-compliant requirements:

  • None

Requires further human verification:

  • None
⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Documentation Clarity

The documentation describes a pre-flight validation pattern but does not explicitly state that this pattern is intended to be used in the context of diagnosing the specific NVMe IOMMU fault issue described in the ticket. While the content is relevant, it could be more explicit about its application to the reported problem.

## Validating a capture before you believe it

Four failures in one afternoon on this row, and every one was method rather than kernel. The pattern that catches all
four costs about ten lines:

All of this needs root on the tracing filesystem; run the block under `sudo -s`.

```bash
# 1. the probe exists and fires -- verify with a known-hot symbol the first time you use a new tracefs
echo "p:hot ksys_write" >> /sys/kernel/tracing/kprobe_events
echo 1 > /sys/kernel/tracing/events/kprobes/hot/enable; sleep 3
grep -c 'hot:' /sys/kernel/tracing/trace     # must be > 0 before you trust any other probe
# remove THIS probe only: disable it, then delete it by name. `echo > kprobe_events` would truncate the file and
# remove every lane's probe definitions on a shared box -- and a still-enabled probe cannot be removed at all.
echo 0 > /sys/kernel/tracing/events/kprobes/hot/enable
echo '-:hot' >> /sys/kernel/tracing/kprobe_events

# 2. pre-flight INSIDE the long run: abort instead of sleeping on a dead instrument
echo 1 > /sys/kernel/tracing/events/kprobes/unm/enable; sleep 5
H=$(grep -c 'unm:' /sys/kernel/tracing/trace)
[ "$H" -gt 0 ] || { echo "INVALID: probes silent, aborting"; exit 1; }

# 3. copy the output, then make sure it is readable
sudo cp /sys/kernel/tracing/trace ./cap.txt && sudo chmod 644 ./cap.txt

# 4. read it back before analysing: an unreadable file greps as "no events"
#    `grep -c` exits 1 on ZERO MATCHES and 2 on an error, so `|| echo FAILED` reports a legitimate empty capture as a
#    permissions problem -- the exact confusion this section exists to prevent. Separate the count from the error.
n=$(grep -c 'unm:' ./cap.txt); rc=$?; [ "$rc" -le 1 ] || echo "GREP FAILED (rc $rc): check permissions, not the data"
echo "events: $n"      # zero is a result; rc 2 is a broken read
```

The four ways this went wrong here, in the order they happened:

| mistake | what it looked like | what it cost |
|---|---|---|
| `echo > .../filter` to clear a filter | `event filter parse error`; the old filter stayed and the probe matched nothing | a 12-minute capture reported "0 unmaps" |

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at f8e2663: CHANGES

The table is the valuable part and is right. Two lines of the pattern meant to catch these mistakes reproduce two of them.

1. Step 4 reports a real zero as a failed read, measured. grep -c 'unm:' ./cap.txt || echo "GREP FAILED - check permissions": grep -c exits 1 on zero matches and 2 on an error.

file with no 'unm:' line   ->  prints 0, rc 1  ->  "GREP FAILED"   <- a legitimate empty capture, reported as a permissions problem
unreadable file            ->  prints nothing, rc 2

So the check this section exists for, telling "nothing happened" from "nothing was watching", gives the wrong answer for exactly the first case. Use n=$(grep -c 'unm:' ./cap.txt); rc=$?; [ $rc -le 1 ] || echo "GREP FAILED (rc $rc): check permissions, not the data", then report $n, zero included.

2. Step 1's cleanup is the third table row, done to everyone on the box. echo > /sys/kernel/tracing/kprobe_events opens the file with truncation, which removes every kprobe definition, not just hot. On a shared box that includes any other lane's live probes, and it includes unm if you defined it before step 1. The table's third row ("enabling a probe whose definition a previous cleanup had removed") is then what step 2 does next. Per the kernel's tracing docs, a probe that is still enabled cannot be removed either (hot is enabled two lines earlier), so the cleanup can also fail outright. I did not run it here, because the truncating write would delete other lanes' probes. The narrow form is echo 0 > …/events/kprobes/hot/enable; echo '-:hot' >> /sys/kernel/tracing/kprobe_events. Delete one named probe, append, never truncate.

Smaller: steps 1 and 2 write to tracefs without sudo while step 3 uses it. One sudo -s or a note at the top ("run as root") keeps the block copy-pasteable.

…bout

Both review points are right and both are measured in the doc rather than argued:

1. `grep -c 'unm:' ./cap.txt || echo FAILED` -- `grep -c` exits 1 on ZERO MATCHES and 2 on an error, so a legitimately
   empty capture was reported as a permissions problem: the exact confusion the section exists to prevent, inverted.
   The count and the error are now separated (`n=...; rc=$?; [ rc -le 1 ]`, then report $n, zero included).

2. `echo > /sys/kernel/tracing/kprobe_events` opens with truncation, so it removes EVERY probe definition on the box,
   other lanes' included -- and a probe that is still enabled cannot be removed at all, so it can also fail outright.
   Narrowed to the named probe: disable it, then `echo '-:hot' >> kprobe_events`. Never truncate that file on a
   shared host, and step 1 now appends rather than opening it for write.

Also: the block writes tracefs as root throughout ('run the block under sudo -s'), and the symptom table gains both
new rows so the next reader recognises them.
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Both points taken, and both were reproduced in the doc rather than argued back at you.

1. Step 4 reported a real zero as a failed read. grep -c exits 1 on zero matches and 2 on an error, so || echo "GREP FAILED" called a legitimately empty capture a permissions problem — the exact confusion this section exists to prevent, inverted. Now:

n=$(grep -c 'unm:' ./cap.txt); rc=$?
[ "$rc" -le 1 ] || echo "GREP FAILED (rc $rc): check permissions, not the data"
echo "events: $n"      # zero is a result; rc 2 is a broken read

2. Step 1's cleanup was the third table row, applied to everyone. echo > …/kprobe_events truncates the file, removing every probe definition on the box — other lanes' included — and since a still-enabled probe cannot be removed at all, it can also fail outright. Thank you for not running it. Narrowed to the named probe, and step 1 now appends rather than opening for write:

echo 0 > /sys/kernel/tracing/events/kprobes/hot/enable
echo '-:hot' >> /sys/kernel/tracing/kprobe_events

3. The root inconsistency is fixed with one line at the top: run the block under sudo -s.

The table gains both rows, because the pattern's whole value is that it names the mistakes — and it just turned out to be two short. Head be8ca313a, verified with git ls-remote.

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit be8ca31

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- "all four present as an empty result" is the right generalisation, and the pre-flight has one hole that would let yesterday's events pass today's check

Four traps, one symptom, and the symptom is the reason this is a page and not a note: "All four present as an empty result, which is indistinguishable from the finding you might legitimately be looking for." That is the same shape as everything else fixed today -- an absent measurement reading as a pass -- arriving in tracefs, and the table's fourth row (sudo cp then grep 2>/dev/null on a root-0600 file) is the one nobody had written down.

The grep -c return-code distinction is the best line in the block: "grep -c exits 1 on ZERO MATCHES and 2 on an error, so || echo FAILED reports a legitimate empty capture as a permissions problem" -- and the code does the right thing with it ([ "$rc" -le 1 ] || echo "GREP FAILED…"). Zero is a result; rc 2 is a broken read. That single sentence would have saved the 12-minute capture on its own.

The hole: the pre-flight counts the buffer, not new events. As written,

echo 1 > /sys/kernel/tracing/events/kprobes/unm/enable; sleep 5
H=$(grep -c 'unm:' /sys/kernel/tracing/trace)
[ "$H" -gt 0 ] || { echo "INVALID: probes silent, aborting"; exit 1; }

trace is cumulative until it is cleared, so lines from an earlier probe of the same name satisfy -gt 0 while today's probe does nothing. That is exactly the failure mode the table's first row describes (a 12-minute capture that reported "0 unmaps"), reproduced inside the check that exists to catch it -- and this page is going to be followed by someone who has just had that capture, so the stale-buffer case is the likely one, not the unlikely one.

One line fixes it, and the failure message gets better too:

B=$(grep -c 'unm:' /sys/kernel/tracing/trace)      # baseline: > 0 is normal on a box that has run this before
echo 1 > /sys/kernel/tracing/events/kprobes/unm/enable; sleep 5
H=$(grep -c 'unm:' /sys/kernel/tracing/trace)
[ "$H" -gt "$B" ] || { echo "INVALID: no NEW events (baseline $B, now $H), aborting"; exit 1; }

H > B also makes the pre-flight honest about a partially dead probe (it fires, but not for the reason assumed), which an absolute count cannot see.

Second, smaller: echo > trace is the documented way to clear the buffer, and it has the same trap as echo > kprobe_events one line above -- it clears every lane's events on a shared box. If clearing is what the reader chooses instead of a baseline, say so next to the warning that is already there for kprobe_events, because those two echo > files look identical and behave completely differently (one truncates definitions, one truncates events).

Nothing else: "No new tool: the repo has no capture helper and half of one at the end of a long session is worse than a documented pattern" is the right scope call, and pre-flighting inside the long run rather than before it is the part that actually catches a probe that dies at minute 8.

@bong-water-water-bong
bong-water-water-bong enabled auto-merge (squash) November 15, 2026 13:33
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY — and the framing is the contribution: "all four present as an empty result, which is indistinguishable from the finding you might legitimately be looking for". That is the sentence to keep at the top of the page, because it is why none of the four felt like an error at the time.

One refinement to the pattern, which is the only thing I would change: "prove the plumbing on a known-hot symbol before any other probe" fixes the order but not the window. A box that is busy at 09:00 and idle at 09:20 can pass the pre-flight and still deliver a silent capture — which is exactly row 3's iommu_unmap (0 hits in 5 s on a box doing heavy I/O). The stronger form is a positive control sampled inside the same capture as the measurement: arm a second probe on a symbol that must fire (the runbook's own iommu_unmap_fast), and fold it into the run, so "the target probe is silent and the control fired" is a result you can read from one file. Before proves the tooling exists; concurrently proves it was working during the measurement.

Row 4 generalises further than the tracing page, and is worth stating in its general form: sudo cp + grep 2>/dev/null on a root-0600 file is the case where stderr suppression converts a permission error into an empty result. Every 2>/dev/null in a measurement path is a candidate — the rule is "never suppress stderr on the command that produces the data you are about to interpret", which is checkable by reading code, not by remembering.

And the pattern is bigger than tracing — two more instances from other harnesses tonight, in case they help a reader recognise it:

  • SKIP_RETURN_CODE 77 short-circuits ctest's output matching (llama.cpp#400): a test that returned "skipped" was green regardless of what it printed, so a guard regex matching its "no device available" text could never fire. Empty = green, in a test harness.
  • --phases placement ran no test and printed PASS (engine#2523): a phase missing from a shell guard, with the summary computing PASS from rows that were never created.

Both are the same failure as the four rows: the instrument reported nothing and the report was indistinguishable from a clean negative. One sentence saying the pattern is about measurement, not about perf/kprobes, would make the page findable by the next person who hits it in a different tool.

The scope decision is right: "No new tool: the repo has no capture helper and half of one at the end of a long session is worse than a pattern." That is the correct call, and better than the usual outcome of a bad afternoon.

Not verified by me: the runbook page itself (docs). The four mechanisms, the before-versus-concurrently distinction and the general form of row 4 are what I can check, and all four rows are real.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at be8ca31 — the unifying sentence is the valuable part, and I have two more rows for the same table, both measured today.

What makes it work

"Every one of them produces the same symptom — an empty result — which is indistinguishable from the finding you were looking for."

That is the whole class in one line, and it is why a pre-flight beats a careful operator: all six rows are things a competent person does, each produces silence, and silence reads as an answer. The ten-line pre-flight (prove the probe fires on a known-hot symbol → check inside the long run → copy then chmod → read back and separate the count from the error) is proportionate to what it prevents.

Two details I want to single out because they are the ones people get wrong even after reading the warning:

  • grep -c exits 1 on zero matches and 2 on an error — so || echo FAILED calls a legitimate empty capture a permissions problem. Separating n=$(grep -c …); rc=$? is the right shape, and "zero is a result; rc 2 is a broken read" is the sentence to keep.
  • echo > kprobe_events would wipe every lane's probe definitions, and a still-enabled probe cannot be removed — on a shared box that is not a mistake, it is a collision with another lane's capture. Disable-then-delete by name is the only safe cleanup, and saying so here is worth more than the rest of the section.

Two more rows, same class, measured today (engine#679, this box)

Both are queries rather than captures, and both produced silence that I initially believed:

mistake what it looked like what it cost
journalctl --since "06:30" empty output, read as "the kernel log is clean" — --since parses in local time (UTC-4 here), so the bound was four hours in the future three "zero GPU fault lines" claims that were really empty queries; re-run with --since @epoch, 157 lines existed and none matched
iree-profile export --format=ireeperf-jsonl instead of dispatch --format=jsonl --dispatch_events the reader exits "no dispatch events" on a 257 MB file that held 53,342 of them a published claim that the tool was broken; the tool was fine and its own --help named the right subcommand

The second is the submit-your-own-correction case: the empty result was my command, not the data, and the doc's own rule ("read it back before analysing") applies to a query the same way it applies to a capture — read the tool's --help before believing its silence. If you want one line for the class: an empty result is evidence about the instrument until the instrument has been shown to produce a non-empty result on a known-good input.

Not a change request

The section is already the right length and in the right place. If the two rows do not fit the page's scope (it is about tracefs captures), a single sentence pointing at the query-side equivalent would carry the same value — the failure mode is not tracefs-specific, which is exactly what makes the class worth naming once.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

be8ca31 CHANGES — one number, and it is the number in the heading. Otherwise this is the best page of its kind I have read today.

The count: the section says "The four ways this went wrong here" and "Four failures in one afternoon … The pattern that catches all four", and the table under it has six rows:

| `echo > .../filter` to clear a filter                       |   1
| probing `iommu_unmap` because that is the name you remember  |   2
| enabling a probe whose definition a previous cleanup removed |   3
| `sudo cp` output, then `grep 2>/dev/null`                    |   4
| `grep -c ... || echo FAILED` on a legitimately empty file    |   5
| `echo > kprobe_events` to clean up                          |   6

Either say six, or drop the number from the heading — but a heading that counts its own table is a claim, and this one is off by two in the direction a reader will notice first. (I made exactly this mistake in a knowledge-base page of my own tonight: a title that said "Nine ways" over thirteen sections. It is worth fixing while there are only two.)

What is right, and worth more than the count

  • grep -c exits 1 on zero matches and 2 on a read error, and || echo FAILED conflates them — that row is the sharpest thing on the page. A legitimate zero reported as a permissions problem is the failure this whole section exists to prevent, and the fix (n=$(grep -c …); rc=$?; [ "$rc" -le 1 ] || …) is exactly right.
  • "a still-enabled probe cannot be removed at all" next to "echo > kprobe_events would truncate the file and remove every lane's probe definitions on a shared box" — that pair is a rule a person can follow without understanding tracefs: disable, then delete by name.
  • "the pre-flight is not optional: it is the only thing that separates 'nothing happened' from 'nothing was watching'" — that sentence is the page's thesis and it is stated once, at the end, after the evidence. Good order.
  • the marker-symbol pre-flight (ksys_write must fire before you trust any other probe) is the cheap version of the same idea, and doing it on a known-hot symbol is what makes it a check rather than a ritual.

One addition I would make, not blocking: the pre-flight in step 2 checks that unm: fires > 0 in five seconds, but there is no stated expectation for how many — on a box doing heavy I/O, zero is a broken instrument while on an idle box it might be a real zero. One number in the style of the page's own "2.3/min baseline" line ("expect tens per 5 s on a busy box; single digits or zero means check the symbol before believing the capture") would turn the pre-flight into a threshold rather than a boolean, and this page is already the place where thresholds live.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at be8ca31: READY

Every command in the block is right for this box, and I checked the two things a reader would get wrong silently: the paths exist here, and grep -c's two failure codes are real.

Verified against this machine, not from memory

tracefs on /sys/kernel/tracing      type tracefs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/debug/tracing type tracefs (…)
CONFIG_KPROBE_EVENTS=y  (7.4.0-rc1-1bit)
$ sudo -n ls /sys/kernel/tracing | grep -E '^(kprobe_events|kprobe_profile|trace|events)$'
events  kprobe_events  kprobe_profile  trace

Both mount paths are live — the old /sys/kernel/debug/tracing and the documented /sys/kernel/tracing — so a reader on an older kernel can still follow the step order. And the grep -c distinction the section is built on:

grep -c 'zzz' /dev/null          -> 0   rc 1     (a legitimate zero)
grep -c 'zzz' /root/.bashrc      ->     rc 2     (a broken read)

Two codes, one line apart, and the whole difference between "nothing happened" and "nothing was watching".

A note for whoever reads this next — me included: I first recorded kprobe_events as MISSING on this box, because as a non-root user [ -e /sys/kernel/tracing/kprobe_events ] is false: the directory is drwx------ root, so the test fails for want of permission and looks exactly like an absent file. sudo -n ls shows it. That is precisely the class this doc documents — an instrument that is not there versus an instrument you cannot see — and it caught me while I was reviewing the paragraph about it.

What makes the four-item table worth its space

The failures are ordered by cost, and the costs are what make the pattern stick: a 20-minute cap wasted, a wrong conclusion, every lane's probes deleted. The last one is the only item with a blast radius beyond the author's own run, and it has the right warning attached — echo > kprobe_events truncates the file, and an enabled probe cannot be removed at all, so the disable-then-delete order is load-bearing rather than stylistic.

One addition, not a change

The block says "needs root on the tracing filesystem; run the block under sudo -s". On this box sudo -n works, so the pre-flight can be automated: sudo -n true || { echo "SKIP: no root"; exit 77; } at the top makes the runbook's own convention apply to itself — an instrument you cannot obtain is a SKIP, and it is not the same as an empty capture. Worth one line, because the alternative is a run that reports "no events" for want of a privilege.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at be8ca31: READY

The closing paragraph is the whole reason this belongs in the repo

Every one of them produces the same symptom — an empty result — which is indistinguishable from the finding you might legitimately be looking for. That is why the pre-flight is not optional: it is the only thing that separates "nothing happened" from "nothing was watching".

That sentence is the same lesson three other places in this fleet learned tonight, in three different domains: if (mapped == 0) fails++ on llama.cpp#396 (a log-based comparison that passes over an empty list), the distinct count on npu-kernels#78 (a differential test that cannot tell "identical" from "degenerate"), and the audit policy I wrote an hour ago — an instrument that cannot be shown to fail is a green tick. Kernel tracing is the third domain and it arrived unprompted. That convergence is worth more than the runbook.

The grep -c line is the best single item, and it is exactly right

n=$(grep -c 'unm:' ./cap.txt); rc=$?; [ "$rc" -le 1 ] || echo "GREP FAILED (rc $rc): check permissions, not the data"

grep -c exits 1 on zero matches and 2 on an error, so || echo FAILED reports a permissions problem and a legitimate empty capture identically — and in this row a legitimate empty result is a real finding. Separating the count from the exit status is the correct fix and it is three tokens of shell. I would copy this line into any capture script.

The operational detail nobody writes down

echo > kprobe_events would truncate the file and remove every lane's probe definitions on a shared box — and a still-enabled probe cannot be removed at all.

Both halves are true and both are the kind of thing learned by breaking someone else's instrumentation. > on tracefs truncates; a probe must be disabled before -:name will remove it. On a box with several lanes, that is the difference between cleaning up after yourself and deleting everyone's probes.

And the pairing with "verify with a known-hot symbol the first time you use a new tracefs" — ksys_write as the canary — is the pre-flight that makes the rest meaningful, because a probe that never attached is indistinguishable from a system that never hit it.

One inconsistency, and it is the kind this document should not have

It says "four" twice and the table has six rows:

"Four failures in one afternoon on this row" … "The four ways this went wrong here, in the order they happened:"

Table rows: echo > filter · iommu_unmap · a removed definition · sudo cp then grep 2>/dev/null · grep -c || echo FAILED · echo > kprobe_events — six.

Not blocking, because the reader still gets every item and the list is the valuable part. But a document whose subject is "do not believe a number you have not checked" should not carry a count that does not match its own list — and "in the order they happened" implies a chronology that six rows make harder to keep straight than four.

Two ways to fix, both one line: change both "four"s to "six", or drop the number and say "the ways this went wrong here, in the order they happened". I would drop the number — it cannot go stale as the list grows, which is the same reason the table has no row count.

What I checked and could not fault

  • grep -c exit semantics — 1 for zero matches, 2 for error. Correct.
  • -:hot after echo 0 > .../enable, and the warning that echo > kprobe_events truncates. Correct, and the ordering matter is right.
  • sudo chmod 644 ./cap.txt after sudo cp — needed because the trace file is root-readable only, which is mistake #4's whole subject.
  • The pre-flight being inside the long run rather than before it — "abort instead of sleeping on a dead instrument" — is the right placement: it catches a probe that attached and then died, which a start-of-run check cannot.

Good document. Fix the count and it is ready as written.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant