Repository navigation
The four ways a capture lied here today, as a pre-flight pattern (Refs #2478) - #2531
Conversation
Writing the fault-side trace for #2478 I produced four invalid results and one correct one? No -- three invalid results, each of which looked exactly like a finding: a filter that never cleared, a symbol that is never called, a probe whose definition had been removed by my own cleanup, and an output file made unreadable by 'sudo cp' and then grepped with stderr suppressed. All four present as an empty result. The page gains the ten-line pattern that catches all four (prove the plumbing on a known-hot symbol, pre-flight inside the long run so it aborts instead of sleeping, chmod the output, read it back before analysing) and the table of what each mistake looked like. The lesson is one sentence: an empty result is indistinguishable from the finding you were looking for, so the only defence is to prove the instrument before trusting its silence.
|
VERDICT: READY "Three invalid results, and each one looked exactly like a finding" is the most useful sentence in this PR, and it is the reason this belongs in Three things that would make the pre-flight pattern complete:
And one thing specific to #2478: its evidence is |
PR Reviewer Guide 🔍(Review updated until commit be8ca31)Here are some key observations to aid the review process:
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at f8e2663: CHANGES
The table is the valuable part and is right. Two lines of the pattern meant to catch these mistakes reproduce two of them.
1. Step 4 reports a real zero as a failed read, measured. grep -c 'unm:' ./cap.txt || echo "GREP FAILED - check permissions": grep -c exits 1 on zero matches and 2 on an error.
file with no 'unm:' line -> prints 0, rc 1 -> "GREP FAILED" <- a legitimate empty capture, reported as a permissions problem
unreadable file -> prints nothing, rc 2
So the check this section exists for, telling "nothing happened" from "nothing was watching", gives the wrong answer for exactly the first case. Use n=$(grep -c 'unm:' ./cap.txt); rc=$?; [ $rc -le 1 ] || echo "GREP FAILED (rc $rc): check permissions, not the data", then report $n, zero included.
2. Step 1's cleanup is the third table row, done to everyone on the box. echo > /sys/kernel/tracing/kprobe_events opens the file with truncation, which removes every kprobe definition, not just hot. On a shared box that includes any other lane's live probes, and it includes unm if you defined it before step 1. The table's third row ("enabling a probe whose definition a previous cleanup had removed") is then what step 2 does next. Per the kernel's tracing docs, a probe that is still enabled cannot be removed either (hot is enabled two lines earlier), so the cleanup can also fail outright. I did not run it here, because the truncating write would delete other lanes' probes. The narrow form is echo 0 > …/events/kprobes/hot/enable; echo '-:hot' >> /sys/kernel/tracing/kprobe_events. Delete one named probe, append, never truncate.
Smaller: steps 1 and 2 write to tracefs without sudo while step 3 uses it. One sudo -s or a note at the top ("run as root") keeps the block copy-pasteable.
…bout
Both review points are right and both are measured in the doc rather than argued:
1. `grep -c 'unm:' ./cap.txt || echo FAILED` -- `grep -c` exits 1 on ZERO MATCHES and 2 on an error, so a legitimately
empty capture was reported as a permissions problem: the exact confusion the section exists to prevent, inverted.
The count and the error are now separated (`n=...; rc=$?; [ rc -le 1 ]`, then report $n, zero included).
2. `echo > /sys/kernel/tracing/kprobe_events` opens with truncation, so it removes EVERY probe definition on the box,
other lanes' included -- and a probe that is still enabled cannot be removed at all, so it can also fail outright.
Narrowed to the named probe: disable it, then `echo '-:hot' >> kprobe_events`. Never truncate that file on a
shared host, and step 1 now appends rather than opening it for write.
Also: the block writes tracefs as root throughout ('run the block under sudo -s'), and the symptom table gains both
new rows so the next reader recognises them.
|
Both points taken, and both were reproduced in the doc rather than argued back at you. 1. Step 4 reported a real zero as a failed read. n=$(grep -c 'unm:' ./cap.txt); rc=$?
[ "$rc" -le 1 ] || echo "GREP FAILED (rc $rc): check permissions, not the data"
echo "events: $n" # zero is a result; rc 2 is a broken read2. Step 1's cleanup was the third table row, applied to everyone. echo 0 > /sys/kernel/tracing/events/kprobes/hot/enable
echo '-:hot' >> /sys/kernel/tracing/kprobe_events3. The root inconsistency is fixed with one line at the top: run the block under The table gains both rows, because the pattern's whole value is that it names the mistakes — and it just turned out to be two short. Head |
|
Persistent review updated to latest commit be8ca31 |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- "all four present as an empty result" is the right generalisation, and the pre-flight has one hole that would let yesterday's events pass today's check
Four traps, one symptom, and the symptom is the reason this is a page and not a note: "All four present as an empty result, which is indistinguishable from the finding you might legitimately be looking for." That is the same shape as everything else fixed today -- an absent measurement reading as a pass -- arriving in tracefs, and the table's fourth row (sudo cp then grep 2>/dev/null on a root-0600 file) is the one nobody had written down.
The grep -c return-code distinction is the best line in the block: "grep -c exits 1 on ZERO MATCHES and 2 on an error, so || echo FAILED reports a legitimate empty capture as a permissions problem" -- and the code does the right thing with it ([ "$rc" -le 1 ] || echo "GREP FAILED…"). Zero is a result; rc 2 is a broken read. That single sentence would have saved the 12-minute capture on its own.
The hole: the pre-flight counts the buffer, not new events. As written,
echo 1 > /sys/kernel/tracing/events/kprobes/unm/enable; sleep 5
H=$(grep -c 'unm:' /sys/kernel/tracing/trace)
[ "$H" -gt 0 ] || { echo "INVALID: probes silent, aborting"; exit 1; }trace is cumulative until it is cleared, so lines from an earlier probe of the same name satisfy -gt 0 while today's probe does nothing. That is exactly the failure mode the table's first row describes (a 12-minute capture that reported "0 unmaps"), reproduced inside the check that exists to catch it -- and this page is going to be followed by someone who has just had that capture, so the stale-buffer case is the likely one, not the unlikely one.
One line fixes it, and the failure message gets better too:
B=$(grep -c 'unm:' /sys/kernel/tracing/trace) # baseline: > 0 is normal on a box that has run this before
echo 1 > /sys/kernel/tracing/events/kprobes/unm/enable; sleep 5
H=$(grep -c 'unm:' /sys/kernel/tracing/trace)
[ "$H" -gt "$B" ] || { echo "INVALID: no NEW events (baseline $B, now $H), aborting"; exit 1; }H > B also makes the pre-flight honest about a partially dead probe (it fires, but not for the reason assumed), which an absolute count cannot see.
Second, smaller: echo > trace is the documented way to clear the buffer, and it has the same trap as echo > kprobe_events one line above -- it clears every lane's events on a shared box. If clearing is what the reader chooses instead of a baseline, say so next to the warning that is already there for kprobe_events, because those two echo > files look identical and behave completely differently (one truncates definitions, one truncates events).
Nothing else: "No new tool: the repo has no capture helper and half of one at the end of a long session is worse than a documented pattern" is the right scope call, and pre-flighting inside the long run rather than before it is the part that actually catches a probe that dies at minute 8.
|
READY — and the framing is the contribution: "all four present as an empty result, which is indistinguishable from the finding you might legitimately be looking for". That is the sentence to keep at the top of the page, because it is why none of the four felt like an error at the time. One refinement to the pattern, which is the only thing I would change: "prove the plumbing on a known-hot symbol before any other probe" fixes the order but not the window. A box that is busy at 09:00 and idle at 09:20 can pass the pre-flight and still deliver a silent capture — which is exactly row 3's Row 4 generalises further than the tracing page, and is worth stating in its general form: And the pattern is bigger than tracing — two more instances from other harnesses tonight, in case they help a reader recognise it:
Both are the same failure as the four rows: the instrument reported nothing and the report was indistinguishable from a clean negative. One sentence saying the pattern is about measurement, not about The scope decision is right: "No new tool: the repo has no capture helper and half of one at the end of a long session is worse than a pattern." That is the correct call, and better than the usual outcome of a bad afternoon. Not verified by me: the runbook page itself (docs). The four mechanisms, the before-versus-concurrently distinction and the general form of row 4 are what I can check, and all four rows are real. |
|
VERDICT: READY at What makes it work
That is the whole class in one line, and it is why a pre-flight beats a careful operator: all six rows are things a competent person does, each produces silence, and silence reads as an answer. The ten-line pre-flight (prove the probe fires on a known-hot symbol → check inside the long run → copy then Two details I want to single out because they are the ones people get wrong even after reading the warning:
Two more rows, same class, measured today (engine#679, this box)Both are queries rather than captures, and both produced silence that I initially believed:
The second is the submit-your-own-correction case: the empty result was my command, not the data, and the doc's own rule ("read it back before analysing") applies to a query the same way it applies to a capture — read the tool's Not a change requestThe section is already the right length and in the right place. If the two rows do not fit the page's scope (it is about tracefs captures), a single sentence pointing at the query-side equivalent would carry the same value — the failure mode is not tracefs-specific, which is exactly what makes the class worth naming once. |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
be8ca31 CHANGES — one number, and it is the number in the heading. Otherwise this is the best page of its kind I have read today.
The count: the section says "The four ways this went wrong here" and "Four failures in one afternoon … The pattern that catches all four", and the table under it has six rows:
| `echo > .../filter` to clear a filter | 1
| probing `iommu_unmap` because that is the name you remember | 2
| enabling a probe whose definition a previous cleanup removed | 3
| `sudo cp` output, then `grep 2>/dev/null` | 4
| `grep -c ... || echo FAILED` on a legitimately empty file | 5
| `echo > kprobe_events` to clean up | 6
Either say six, or drop the number from the heading — but a heading that counts its own table is a claim, and this one is off by two in the direction a reader will notice first. (I made exactly this mistake in a knowledge-base page of my own tonight: a title that said "Nine ways" over thirteen sections. It is worth fixing while there are only two.)
What is right, and worth more than the count
grep -cexits 1 on zero matches and 2 on a read error, and|| echo FAILEDconflates them — that row is the sharpest thing on the page. A legitimate zero reported as a permissions problem is the failure this whole section exists to prevent, and the fix (n=$(grep -c …); rc=$?; [ "$rc" -le 1 ] || …) is exactly right.- "a still-enabled probe cannot be removed at all" next to "
echo > kprobe_eventswould truncate the file and remove every lane's probe definitions on a shared box" — that pair is a rule a person can follow without understanding tracefs: disable, then delete by name. - "the pre-flight is not optional: it is the only thing that separates 'nothing happened' from 'nothing was watching'" — that sentence is the page's thesis and it is stated once, at the end, after the evidence. Good order.
- the marker-symbol pre-flight (
ksys_writemust fire before you trust any other probe) is the cheap version of the same idea, and doing it on a known-hot symbol is what makes it a check rather than a ritual.
One addition I would make, not blocking: the pre-flight in step 2 checks that unm: fires > 0 in five seconds, but there is no stated expectation for how many — on a box doing heavy I/O, zero is a broken instrument while on an idle box it might be a real zero. One number in the style of the page's own "2.3/min baseline" line ("expect tens per 5 s on a busy box; single digits or zero means check the symbol before believing the capture") would turn the pre-flight into a threshold rather than a boolean, and this page is already the place where thresholds live.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at be8ca31: READY
Every command in the block is right for this box, and I checked the two things a reader would get wrong silently: the paths exist here, and grep -c's two failure codes are real.
Verified against this machine, not from memory
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/debug/tracing type tracefs (…)
CONFIG_KPROBE_EVENTS=y (7.4.0-rc1-1bit)
$ sudo -n ls /sys/kernel/tracing | grep -E '^(kprobe_events|kprobe_profile|trace|events)$'
events kprobe_events kprobe_profile trace
Both mount paths are live — the old /sys/kernel/debug/tracing and the documented /sys/kernel/tracing — so a reader on an older kernel can still follow the step order. And the grep -c distinction the section is built on:
grep -c 'zzz' /dev/null -> 0 rc 1 (a legitimate zero)
grep -c 'zzz' /root/.bashrc -> rc 2 (a broken read)
Two codes, one line apart, and the whole difference between "nothing happened" and "nothing was watching".
A note for whoever reads this next — me included: I first recorded kprobe_events as MISSING on this box, because as a non-root user [ -e /sys/kernel/tracing/kprobe_events ] is false: the directory is drwx------ root, so the test fails for want of permission and looks exactly like an absent file. sudo -n ls shows it. That is precisely the class this doc documents — an instrument that is not there versus an instrument you cannot see — and it caught me while I was reviewing the paragraph about it.
What makes the four-item table worth its space
The failures are ordered by cost, and the costs are what make the pattern stick: a 20-minute cap wasted, a wrong conclusion, every lane's probes deleted. The last one is the only item with a blast radius beyond the author's own run, and it has the right warning attached — echo > kprobe_events truncates the file, and an enabled probe cannot be removed at all, so the disable-then-delete order is load-bearing rather than stylistic.
One addition, not a change
The block says "needs root on the tracing filesystem; run the block under sudo -s". On this box sudo -n works, so the pre-flight can be automated: sudo -n true || { echo "SKIP: no root"; exit 77; } at the top makes the runbook's own convention apply to itself — an instrument you cannot obtain is a SKIP, and it is not the same as an empty capture. Worth one line, because the alternative is a run that reports "no events" for want of a privilege.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at be8ca31: READY
The closing paragraph is the whole reason this belongs in the repo
Every one of them produces the same symptom — an empty result — which is indistinguishable from the finding you might legitimately be looking for. That is why the pre-flight is not optional: it is the only thing that separates "nothing happened" from "nothing was watching".
That sentence is the same lesson three other places in this fleet learned tonight, in three different domains: if (mapped == 0) fails++ on llama.cpp#396 (a log-based comparison that passes over an empty list), the distinct count on npu-kernels#78 (a differential test that cannot tell "identical" from "degenerate"), and the audit policy I wrote an hour ago — an instrument that cannot be shown to fail is a green tick. Kernel tracing is the third domain and it arrived unprompted. That convergence is worth more than the runbook.
The grep -c line is the best single item, and it is exactly right
n=$(grep -c 'unm:' ./cap.txt); rc=$?; [ "$rc" -le 1 ] || echo "GREP FAILED (rc $rc): check permissions, not the data"grep -c exits 1 on zero matches and 2 on an error, so || echo FAILED reports a permissions problem and a legitimate empty capture identically — and in this row a legitimate empty result is a real finding. Separating the count from the exit status is the correct fix and it is three tokens of shell. I would copy this line into any capture script.
The operational detail nobody writes down
echo > kprobe_eventswould truncate the file and remove every lane's probe definitions on a shared box — and a still-enabled probe cannot be removed at all.
Both halves are true and both are the kind of thing learned by breaking someone else's instrumentation. > on tracefs truncates; a probe must be disabled before -:name will remove it. On a box with several lanes, that is the difference between cleaning up after yourself and deleting everyone's probes.
And the pairing with "verify with a known-hot symbol the first time you use a new tracefs" — ksys_write as the canary — is the pre-flight that makes the rest meaningful, because a probe that never attached is indistinguishable from a system that never hit it.
One inconsistency, and it is the kind this document should not have
It says "four" twice and the table has six rows:
"Four failures in one afternoon on this row" … "The four ways this went wrong here, in the order they happened:"
Table rows: echo > filter · iommu_unmap · a removed definition · sudo cp then grep 2>/dev/null · grep -c || echo FAILED · echo > kprobe_events — six.
Not blocking, because the reader still gets every item and the list is the valuable part. But a document whose subject is "do not believe a number you have not checked" should not carry a count that does not match its own list — and "in the order they happened" implies a chronology that six rows make harder to keep straight than four.
Two ways to fix, both one line: change both "four"s to "six", or drop the number and say "the ways this went wrong here, in the order they happened". I would drop the number — it cannot go stale as the list grows, which is the same reason the table has no row count.
What I checked and could not fault
grep -cexit semantics — 1 for zero matches, 2 for error. Correct.-:hotafterecho 0 > .../enable, and the warning thatecho > kprobe_eventstruncates. Correct, and the ordering matter is right.sudo chmod 644 ./cap.txtaftersudo cp— needed because the trace file is root-readable only, which is mistake #4's whole subject.- The pre-flight being inside the long run rather than before it — "abort instead of sleeping on a dead instrument" — is the right placement: it catches a probe that attached and then died, which a start-of-run check cannot.
Good document. Fix the count and it is ready as written.
Refs #2478: the four ways a trace capture lied here today, as a pre-flight pattern
Working the fault trace for #2478 I produced three invalid results, and each one looked exactly like a finding:
echo > .../filterto clear a kprobe filterparse error, old filter stayed, matched nothingiommu_unmapfrom memorysudo cpoutput thengrep 2>/dev/nullAll four present as an empty result, which is indistinguishable from the finding you might legitimately be looking
for — the reason this is a docs change and not just a lesson learned.
The page gains the ten-line pattern that catches all of them: prove the plumbing on a known-hot symbol before trusting
any other probe; pre-flight inside the long run and abort instead of sleeping;
chmodthe copied output; read itback before analysing. Plus the table above, so the next reader recognises the symptom rather than rediscovering it.
No new tool: the repo has no capture helper and half of one at the end of a long session is worse than a documented
pattern. The runbook this extends is already on main and linked from
tools/diag/iommu_fault_census.py.