From 4bdee70e7c17afb592b32c064bf55126d70acb6d Mon Sep 17 00:00:00 2001 From: bong-water-water-bong Date: Sun, 15 Nov 2026 05:15:44 -0400 Subject: [PATCH] Test 1 is one sysfs write, not a package install The runbook's first version reached main inside another lane's commit (4e59db05f) and still opens with `nvme set-feature`, which cannot run here: `command -v nvme` finds nothing, so a reader would see 'command not found' and read it as the test failing. The switch nvme-cli fronts is sysfs, and it is a better test than the tool: `cat /sys/class/nvme/nvme0/hmb` reads 1 on this host, the file is readable as a user and writable by root, so disabling the HMB -- and reading the value back to confirm it took -- is one command with no package. The nvme-cli spelling stays as the fallback for firmware that refuses the write. --- docs/nvme-iommu-fault-triage.md | 29 ++++++++++++++++++++--------- 1 file changed, 20 insertions(+), 9 deletions(-) diff --git a/docs/nvme-iommu-fault-triage.md b/docs/nvme-iommu-fault-triage.md index 43984a4ab..f3495e0e6 100644 --- a/docs/nvme-iommu-fault-triage.md +++ b/docs/nvme-iommu-fault-triage.md @@ -59,17 +59,26 @@ software IO TLB: mapped [mem 0x5927e000-0x5d27e000] (64MB) ## Test 1 — disable the HMB (no reboot, decides whether the buffer is the HMB at all) -**`nvme-cli` is not installed on this host** (checked: `command -v nvme` → nothing), so the first command below is -`apt install nvme-cli` on this box, not `nvme …`. Whoever takes this on should expect that step and not treat -"command not found" as the test failing: +**No `nvme-cli` needed, and none is installed here** (`command -v nvme` → nothing). The kernel exposes the same switch +in sysfs, so this is one write and no package. Measured on this host: ```bash -sudo apt install nvme-cli # not present as of 11-15 08:1xZ -nvme get-feature /dev/nvme0 -f 0x0d -H # Host Memory Buffer: current preference -nvme set-feature /dev/nvme0 -f 0x0d -v 0 # ask for no HMB -journalctl -k -f | grep -c IO_PAGE_FAULT # watch one burst interval (~2-5 min under load) +$ cat /sys/class/nvme/nvme0/hmb +1 # HMB is on; 0 means off +$ ls -l /sys/class/nvme/nvme0/hmb +-rw-r--r-- 1 root root 4096 … # readable as a user, writable only by root ``` +```bash +echo 0 | sudo tee /sys/class/nvme/nvme0/hmb # ask the controller for no HMB +cat /sys/class/nvme/nvme0/hmb # must read 0 before you trust the test +journalctl -k -f | grep -c IO_PAGE_FAULT # watch one burst interval (~2-5 min under load) +echo 1 | sudo tee /sys/class/nvme/nvme0/hmb # put it back +``` + +If a firmware refuses the sysfs write, the `nvme-cli` equivalent is `nvme set-feature /dev/nvme0 -f 0x0d -v 0` +(feature `0x0d`, Host Memory Buffer). + **Prediction:** the `c4:00.0` stream stops within one burst interval. **If it stops** the faulting region is the HMB, and the remaining question is only why its mapping goes stale. **If it continues** the HMB is a bystander and the faulting region is the SWIOTLB bounce area or another map — go to @@ -99,8 +108,10 @@ teardown/re-establish ordering bug on the NVMe side, and the next step is tracin ## Two traps this runbook exists to stop * **A count without a device split is not a measurement.** `distinct` exposes the mixing. -* **A command is not a fact until you check the box has it.** This page's first draft opened with `nvme get-feature`, - and `nvme-cli` is not installed on the very host it is written for -- found by applying this page's own rule to itself. +* **A command is not a fact until you check the box has it, and the interface is often better than the tool.** + This page's first draft opened with `nvme get-feature`; `nvme-cli` is not installed here, and the switch it fronts is + a root-writable sysfs file — so the test became one write instead of an apt step. Found by applying the page's rule + to the page. * **A flag is not a fact until you check the binary that parses it.** The first `strings` attempt here ran against the *compressed* `vmlinuz` and returned 0 for every pattern, including ones that exist — which would have "proved" the parameter absent. Decompress first (`extract-vmlinux`), then count.