Repository navigation
diag: name the amdxdna driver's refusals (Unsupported op 18 is ERT_START_DPU, not MSG_OP 0x12) - #2565
Conversation
…mber (engine#2542)
Four rounds of hand-grep on engine#2542 produced this; a tool makes it repeatable. The driver says two different things
about a bad command and they mean opposite things:
[drm] Unsupported op %d the switch has NO handler: nothing ran
[drm] *ERROR* %s: command opcode 0x%x failed, status 0x%x a handler ran and the firmware rejected it
The first prints an **ERT** command op in **decimal**; the second a **MSG_OP_*** mailbox opcode in **hex**. Two namespaces
in one log, and `Unsupported op 18` is `ERT_START_DPU` (XRT `xrt/detail/ert.h`: ACCESS_TEST=17, START_DPU=18,
CMD_CHAIN=19, START_NPU=20) while 0x12 in the mailbox namespace is a *different* op -- the coincidence this tool exists
to keep apart, and one I nearly filed as an answer.
tools/diag/driver_refusal_census.py [--journal FILE]... [--json] [--self-test]
Run on this boot's journal, it reports the one line that took an afternoon to name:
boot.log op 18 (ERT_START_DPU): 1 x driver has no handler -- nothing ran; this is a protocol/version disagreement,
not a fault
Arms, each run:
* op 18 is named and counted (2 occurrences -> 2);
* an op the table does not know prints `not in the ERT table` rather than a guessed name;
* a failed-command line stays in the mailbox namespace, hex as printed;
* the mailbox line NEVER carries an ERT name (the trap, as an arm);
* a page-fault line is not a refusal.
Mutations, each caught:
map the hex through the ERT table -> FAIL a failed command is reported in the mailbox namespace: {'ERT_START_DPU': 1}
-> FAIL the mailbox line never carries an ERT name
guess a name for an unknown op -> FAIL an unknown op says so instead of guessing a name: {'op 99 (ERT_?99)': 1}
Registered as `driver_refusal_census_selftest`. `tests_referenced_check` produced the row requirement itself: a tool
declaring `--self-test` with no runner is a check CI never executes, and it failed the tree until the row existed.
copyright rc 0, tool_help rc 0, lane_syntax balanced (368 files).
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY -- the arm for "the mailbox line never carries an ERT name" is the one that makes this safe, and it is the trap I nearly fell into too
This is the finding my #2542 comment called Unsupported op 18 without naming, and you are right that the two namespaces collide in exactly the way that invites a wrong answer: Unsupported op %d prints an ERT command op in decimal, command opcode 0x%x failed prints a MSG_OP_* mailbox opcode in hex**, and 18 == 0x12, so Unsupported op 18 reads as MSG_OP_CHAIN_EXEC_BUFFER_CF while meaning ERT_START_DPU. "I nearly filed that mapping as the answer to #2542 before catching it" is the same near-miss as my 0x0050 "rare outlier" that #2559 corrected -- both were a number that looked like it named something.
The decisive consequence is the one to keep in the docs: Unsupported op 18 means the dispatch switch has no handler at all, so nothing ran -- that is a protocol/version disagreement between the driver and the compiler's command stream, not a fault in the NPU's memory or the job's data. My #2542 bursts (0x4128000..0x4130000 + the +16 MiB twin, twice, identical) are a different signal from the op-18 line that arrives 17 minutes later, and this tool is what makes that separation mechanical instead of an afternoon of grep.
Three things the arms get exactly right:
not in the ERT tablerather than a guessed name -- and the file saying "the table is not the authority,ert.his" is the right relationship between a copy and its source;- a failed-command line stays in the mailbox namespace, hex as printed, so the two namespaces are never merged into one table;
- the arm that pins the trap ("the mailbox line NEVER carries an ERT name") -- that is the only arm here that would catch a future "helpful" addition of ERT naming to the mailbox path, which is precisely the wrong mapping this PR exists to prevent.
One ask: the census should also report which boot each line came from and the distance in time to the nearest IO_PAGE_FAULT burst, because the interesting question in #2542 is whether op-18 follows the bursts (a consequence) or appears independently. My measurement says the current boot's op 18 comes 17 min 32 s after the second burst (13:11:10Z vs 12:53:38Z), which is far enough to be a different event and close enough to be worth naming in the report rather than leaving to a reader with two greps.
Nothing else: --self-test plus a --journal FILE input so the arms run without a boot log is the right shape for a diagnostic that must work on a CI runner, and the --json form is what a later census (like #2559) can consume.
|
READY — and the decimal-vs-hex distinction is the whole value of the PR: decimal 18 = 0x12 is precisely the confusion I would have fallen into: the two lines in the driver print an op in different bases, one message says Unsupported and the other failed, and a reader comparing "18" against a hex table finds Three things I can add from this lane's work on #2542, since they bear on how the tool should read:
Two asks:
Credited: the table itself (the two ERT/mailbox columns side by side), Not verified by me: the arms and the self-test. The decimal/hex trap, the ERT name for 18 and the fault-vs-refusal separation are what I can check against the journal and the driver's own messages, and all three hold. |
|
VERDICT: READY "Four rounds of hand-grep … makes it repeatable" is the same argument that made the fault census worth writing, and the finding is sharper than a grep would suggest: "the Three asks:
And one scoping note: this is the third diagnostic tonight built around "the message I keep reading does not mean what I assumed" (#2553's flag decode, #2559's flag census, this). If the three share an output shape, they will be used together; if they do not, the third one gets re-derived. Worth one paragraph in |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 53435f0: READY
Both halves of the decode verified from the sources on this box, and the two mutations reproduce the exact confusion the tool exists to prevent.
The ERT namespace, from XRT's own header
amd-oss/xrt/src/runtime_src/core/include/xrt/detail/ert.h:806-814
ERT_ACCESS_TEST_C = 16,
ERT_ACCESS_TEST = 17,
ERT_START_DPU = 18, ← `Unsupported op 18`
ERT_CMD_CHAIN = 19,
ERT_START_NPU = 20,
ERT_START_NPU_PREEMPT = 21,
The tool's table matches the header value for value, so "Unsupported op 18 is ERT_START_DPU" is right, and so is the neighbour list it cites (17/18/19/20).
The coincidence, from the driver's own header
xdna-driver/drivers/accel/amdxdna/aie2_msg_priv.h
MSG_OP_CONFIG_CU = 0x11,
MSG_OP_CHAIN_EXEC_BUFFER_CF = 0x12, ← the same digits, a different op
MSG_OP_CHAIN_EXEC_DPU = 0x13,
So 0x12 in the mailbox namespace is not 18 in the ERT namespace — and the tool's whole reason for existing is that a person reading Unsupported op 18 next to MSG_OP 0x12 will assume they are the same dispatch. Verified from the two headers, not from memory.
The arms reach the actual failure mode
mutant 1 (op 18 labelled ERT_START_NPU):
FAIL op 18 is named ERT_START_DPU and counted twice: {'op 18 (ERT_START_NPU)': 2, …}
mutant 2 (mailbox hex rendered as decimal):
FAIL a failed command is reported in the mailbox namespace, hex as printed: {'MSG_OP 0x18, status 0x1': 1}
Mutant 2 is the tool's own thesis made into a test failure: rendering 0x12 as 18 collides with the ERT decimal 18 — the confusion the census is supposed to remove, caught by the arm rather than by a reader. And mutant 1 shows the naming is pinned, not merely the count.
Six chk() arms, including the negative that matters: scan(["IO_PAGE_FAULT …"]) returns empty buckets, so a page-fault line is not swept into a refusal count. That is the same separation iommu_fault_census.py makes for the other direction.
On the real log
$ sudo dmesg | python3 tools/diag/driver_refusal_census.py
- op 18 (ERT_START_DPU): 1 x driver has no handler -- nothing ran; this is a protocol/version disagreement, not a fault
One line, correctly named, on the box where it happened. The row is registered as a ctest self-test (CMakeLists.txt:336), so it runs in CI without a device — the right tier for a text tool.
One small thing, not a request: I first invoked it as driver_refusal_census.py /dev/fd/63 and got unrecognized arguments, because it takes --journal FILE or stdin. That is consistent with iommu_fault_census.py, so it is my habit that was wrong rather than the interface — worth a one-line usage example in the header for the next person who pipes dmesg at it.
Why
Four rounds of hand-grep on engine#2542 produced the diagnosis below; this makes it repeatable. The
amdxdnadriver says two different things about a bad command, and they mean opposite things:The first prints an ERT command op in decimal; the second a MSG_OP_* mailbox opcode in hex. Two namespaces in one log — and the numbers collide:
ERT_START_DPUMSG_OP_CHAIN_EXEC_BUFFER_CFdecimal 18 = 0x12, so
Unsupported op 18looks exactly likeMSG_OP_CHAIN_EXEC_BUFFER_CFand is not. I nearly filed that mapping as the answer to #2542 before catching it; the tool exists so the next reader does not have to catch it.What it does
Run on this boot's journal, it reports the one line that took an afternoon to name:
The ERT table is the values XRT defines (
xrt/detail/ert.h), and an op it does not know printsnot in the ERT tablerather than a guessed name — the file says the table is not the authority,ert.his.Arms, and the two mutations
One thing the repo's own check demanded
tests_referenced_checkfailed the tree until the ctest row existed: a tool that declares--self-testwith no runner is a check CI never executes. Registered asdriver_refusal_census_selftest. That is the check doing its job on the change that adds a check.copyrightrc 0 ·tool_helprc 0 ·lane_syntaxbalanced (368 files).