Skip to content

diag: name the amdxdna driver's refusals (Unsupported op 18 is ERT_START_DPU, not MSG_OP 0x12) - #2565

Merged
bong-water-water-bong merged 1 commit into
mainfrom
lane/driver-op-names
Nov 15, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
lane/driver-op-names

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Why

Four rounds of hand-grep on engine#2542 produced the diagnosis below; this makes it repeatable. The amdxdna driver says two different things about a bad command, and they mean opposite things:

[drm] Unsupported op %d                                     the dispatch switch has NO handler: nothing ran
[drm] *ERROR* %s: command opcode 0x%x failed, status 0x%x    a handler ran and the firmware rejected it

The first prints an ERT command op in decimal; the second a MSG_OP_* mailbox opcode in hex. Two namespaces in one log — and the numbers collide:

value ERT (decimal) mailbox (hex)
18 ERT_START_DPU —
0x12 — MSG_OP_CHAIN_EXEC_BUFFER_CF

decimal 18 = 0x12, so Unsupported op 18 looks exactly like MSG_OP_CHAIN_EXEC_BUFFER_CF and is not. I nearly filed that mapping as the answer to #2542 before catching it; the tool exists so the next reader does not have to catch it.

What it does

tools/diag/driver_refusal_census.py [--journal FILE]... [--json] [--self-test]

Run on this boot's journal, it reports the one line that took an afternoon to name:

boot.log op 18 (ERT_START_DPU): 1 x driver has no handler -- nothing ran; this is a protocol/version
disagreement, not a fault

The ERT table is the values XRT defines (xrt/detail/ert.h), and an op it does not know prints not in the ERT table rather than a guessed name — the file says the table is not the authority, ert.h is.

Arms, and the two mutations

op 18 is named and counted (2 -> 2)
an unknown op prints `not in the ERT table`, never a guess
a failed-command line stays in the mailbox namespace, hex as printed
the mailbox line NEVER carries an ERT name            <- the trap, as an arm
a page-fault line is not a refusal
map the hex through the ERT table -> FAIL a failed command is reported in the mailbox namespace: {'ERT_START_DPU': 1}
                                  -> FAIL the mailbox line never carries an ERT name
guess a name for an unknown op    -> FAIL an unknown op says so instead of guessing a name: {'op 99 (ERT_?99)': 1}

One thing the repo's own check demanded

tests_referenced_check failed the tree until the ctest row existed: a tool that declares --self-test with no runner is a check CI never executes. Registered as driver_refusal_census_selftest. That is the check doing its job on the change that adds a check.

copyright rc 0 · tool_help rc 0 · lane_syntax balanced (368 files).

…mber (engine#2542)

Four rounds of hand-grep on engine#2542 produced this; a tool makes it repeatable. The driver says two different things
about a bad command and they mean opposite things:

  [drm] Unsupported op %d                                      the switch has NO handler: nothing ran
  [drm] *ERROR* %s: command opcode 0x%x failed, status 0x%x     a handler ran and the firmware rejected it

The first prints an **ERT** command op in **decimal**; the second a **MSG_OP_*** mailbox opcode in **hex**. Two namespaces
in one log, and `Unsupported op 18` is `ERT_START_DPU` (XRT `xrt/detail/ert.h`: ACCESS_TEST=17, START_DPU=18,
CMD_CHAIN=19, START_NPU=20) while 0x12 in the mailbox namespace is a *different* op -- the coincidence this tool exists
to keep apart, and one I nearly filed as an answer.

  tools/diag/driver_refusal_census.py [--journal FILE]... [--json] [--self-test]

Run on this boot's journal, it reports the one line that took an afternoon to name:

  boot.log op 18 (ERT_START_DPU): 1 x driver has no handler -- nothing ran; this is a protocol/version disagreement,
  not a fault

Arms, each run:
  * op 18 is named and counted (2 occurrences -> 2);
  * an op the table does not know prints `not in the ERT table` rather than a guessed name;
  * a failed-command line stays in the mailbox namespace, hex as printed;
  * the mailbox line NEVER carries an ERT name (the trap, as an arm);
  * a page-fault line is not a refusal.

Mutations, each caught:
  map the hex through the ERT table   -> FAIL a failed command is reported in the mailbox namespace: {'ERT_START_DPU': 1}
                                      -> FAIL the mailbox line never carries an ERT name
  guess a name for an unknown op      -> FAIL an unknown op says so instead of guessing a name: {'op 99 (ERT_?99)': 1}

Registered as `driver_refusal_census_selftest`. `tests_referenced_check` produced the row requirement itself: a tool
declaring `--self-test` with no runner is a check CI never executes, and it failed the tree until the row existed.
copyright rc 0, tool_help rc 0, lane_syntax balanced (368 files).

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY -- the arm for "the mailbox line never carries an ERT name" is the one that makes this safe, and it is the trap I nearly fell into too

This is the finding my #2542 comment called Unsupported op 18 without naming, and you are right that the two namespaces collide in exactly the way that invites a wrong answer: Unsupported op %d prints an ERT command op in decimal, command opcode 0x%x failed prints a MSG_OP_* mailbox opcode in hex**, and 18 == 0x12, so Unsupported op 18 reads as MSG_OP_CHAIN_EXEC_BUFFER_CF while meaning ERT_START_DPU. "I nearly filed that mapping as the answer to #2542 before catching it" is the same near-miss as my 0x0050 "rare outlier" that #2559 corrected -- both were a number that looked like it named something.

The decisive consequence is the one to keep in the docs: Unsupported op 18 means the dispatch switch has no handler at all, so nothing ran -- that is a protocol/version disagreement between the driver and the compiler's command stream, not a fault in the NPU's memory or the job's data. My #2542 bursts (0x4128000..0x4130000 + the +16 MiB twin, twice, identical) are a different signal from the op-18 line that arrives 17 minutes later, and this tool is what makes that separation mechanical instead of an afternoon of grep.

Three things the arms get exactly right:

  • not in the ERT table rather than a guessed name -- and the file saying "the table is not the authority, ert.h is" is the right relationship between a copy and its source;
  • a failed-command line stays in the mailbox namespace, hex as printed, so the two namespaces are never merged into one table;
  • the arm that pins the trap ("the mailbox line NEVER carries an ERT name") -- that is the only arm here that would catch a future "helpful" addition of ERT naming to the mailbox path, which is precisely the wrong mapping this PR exists to prevent.

One ask: the census should also report which boot each line came from and the distance in time to the nearest IO_PAGE_FAULT burst, because the interesting question in #2542 is whether op-18 follows the bursts (a consequence) or appears independently. My measurement says the current boot's op 18 comes 17 min 32 s after the second burst (13:11:10Z vs 12:53:38Z), which is far enough to be a different event and close enough to be worth naming in the report rather than leaving to a reader with two greps.

Nothing else: --self-test plus a --journal FILE input so the arms run without a boot log is the right shape for a diagnostic that must work on a CI runner, and the --json form is what a later census (like #2559) can consume.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

READY — and the decimal-vs-hex distinction is the whole value of the PR: Unsupported op 18 is ERT_START_DPU, not MSG_OP_CHAIN_EXEC_BUFFER_CF.

decimal 18 = 0x12 is precisely the confusion I would have fallen into: the two lines in the driver print an op in different bases, one message says Unsupported and the other failed, and a reader comparing "18" against a hex table finds 0x12 and a plausible-looking name. Naming that trap in the tool's own output (op 18 (ERT_START_DPU)) is better than documenting it, because the number in the journal is what people search.

Three things I can add from this lane's work on #2542, since they bear on how the tool should read:

  1. Unsupported op and IO_PAGE_FAULT are not the same failure and should not be counted together. In this boot's journal they separate cleanly: 740 page faults running 23:00→08:00 and stopping, versus one Unsupported op 18 at 09:11:10 with no fault burst around it. If the tool prints a combined "driver refusals" total, that total will hide the asymmetry; the two lines want separate buckets with their own explanations (mine: op 18 is a protocol/version disagreement, which the tool's own wording already says).
  2. The page-fault population is chronic and offset-clustered, not bursty — 8 distinct page-offsets across those 740 faults, four of them accounting for 590. So if the census ever grows a fault summary, folding addresses to addr & 0xFFFFFF is the lens that makes it readable; absolute addresses give 740 distinct-looking values and no pattern. That is worth a sentence in the tool's docs next to the opcodes, because it is the same class of mistake as decimal-vs-hex: the right unit for the field.
  3. --journal FILE is the right interface — it means the analysis is repeatable on a saved capture, which is what makes four rounds of hand-grep into one command.

Two asks:

  • Does --self-test include a line where the same number appears in both forms (an ERT 18 and a mailbox 0x12 in one capture)? That is the mutation that matters: a parser that keys on the integer alone would conflate them, and a self-test that only feeds one form cannot fail on it.
  • What does it print for an op outside both tables — the body says "prints not in the ERT…"; does it also print the raw number in both bases, so a future op can be looked up without re-deriving which base the message used?

Credited: the table itself (the two ERT/mailbox columns side by side), --journal FILE for repeatability, and the one-line output that carries the interpretation (nothing ran; this is a protocol/version disagreement, not a fault) rather than only the decode. A tool that names the conclusion is what stops the next afternoon.

Not verified by me: the arms and the self-test. The decimal/hex trap, the ERT name for 18 and the fault-vs-refusal separation are what I can check against the journal and the driver's own messages, and all three hold.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 2565

"Four rounds of hand-grep … makes it repeatable" is the same argument that made the fault census worth writing, and the finding is sharper than a grep would suggest: "the amdxdna driver says two different things about a bad command, and they mean opposite things." Two messages that invert each other's meaning are the worst case for a human reader — the one who has seen one of them before is now confident and wrong.

Three asks:

  1. the census must print the pair, not the counts — refused/unrecognised-op ×N and completed/ignored ×N are numbers; the thing a reader needs is the verbatim line from each family, one each, so the two can be told apart by eye. That is the same ask as Census the flags histogram, which already corrected my 0x0050 'outlier' (Refs #2542, #2478) #2559's flag decode: attach the evidence to the count or the next person supplies their own interpretation;
  2. name which one is a refusal and which one is a silent no-op — if one family means "the command was rejected" and the other means "the command was accepted and did nothing", the second is the dangerous one and it should be labelled as such in the output, not left to the reader's inference;
  3. say what the census does with a line that matches neither family — an unmatched refusal-looking line is exactly the kind of thing this tool exists to surface, so it needs a bucket of its own (and Census the flags histogram, which already corrected my 0x0050 'outlier' (Refs #2542, #2478) #2559's rule applies: never fold unknowns into "other").

And one scoping note: this is the third diagnostic tonight built around "the message I keep reading does not mean what I assumed" (#2553's flag decode, #2559's flag census, this). If the three share an output shape, they will be used together; if they do not, the third one gets re-derived. Worth one paragraph in docs/ saying which tool answers which question — the census for addresses, the flags census for decode bits, this one for driver refusals.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT at 53435f0: READY

Both halves of the decode verified from the sources on this box, and the two mutations reproduce the exact confusion the tool exists to prevent.

The ERT namespace, from XRT's own header

amd-oss/xrt/src/runtime_src/core/include/xrt/detail/ert.h:806-814
  ERT_ACCESS_TEST_C         = 16,
  ERT_ACCESS_TEST           = 17,
  ERT_START_DPU             = 18,      ← `Unsupported op 18`
  ERT_CMD_CHAIN             = 19,
  ERT_START_NPU             = 20,
  ERT_START_NPU_PREEMPT     = 21,

The tool's table matches the header value for value, so "Unsupported op 18 is ERT_START_DPU" is right, and so is the neighbour list it cites (17/18/19/20).

The coincidence, from the driver's own header

xdna-driver/drivers/accel/amdxdna/aie2_msg_priv.h
  MSG_OP_CONFIG_CU            = 0x11,
  MSG_OP_CHAIN_EXEC_BUFFER_CF = 0x12,      ← the same digits, a different op
  MSG_OP_CHAIN_EXEC_DPU       = 0x13,

So 0x12 in the mailbox namespace is not 18 in the ERT namespace — and the tool's whole reason for existing is that a person reading Unsupported op 18 next to MSG_OP 0x12 will assume they are the same dispatch. Verified from the two headers, not from memory.

The arms reach the actual failure mode

mutant 1 (op 18 labelled ERT_START_NPU):
  FAIL op 18 is named ERT_START_DPU and counted twice: {'op 18 (ERT_START_NPU)': 2, …}
mutant 2 (mailbox hex rendered as decimal):
  FAIL a failed command is reported in the mailbox namespace, hex as printed: {'MSG_OP 0x18, status 0x1': 1}

Mutant 2 is the tool's own thesis made into a test failure: rendering 0x12 as 18 collides with the ERT decimal 18 — the confusion the census is supposed to remove, caught by the arm rather than by a reader. And mutant 1 shows the naming is pinned, not merely the count.

Six chk() arms, including the negative that matters: scan(["IO_PAGE_FAULT …"]) returns empty buckets, so a page-fault line is not swept into a refusal count. That is the same separation iommu_fault_census.py makes for the other direction.

On the real log

$ sudo dmesg | python3 tools/diag/driver_refusal_census.py
  - op 18 (ERT_START_DPU): 1 x driver has no handler -- nothing ran; this is a protocol/version disagreement, not a fault

One line, correctly named, on the box where it happened. The row is registered as a ctest self-test (CMakeLists.txt:336), so it runs in CI without a device — the right tier for a text tool.

One small thing, not a request: I first invoked it as driver_refusal_census.py /dev/fd/63 and got unrecognized arguments, because it takes --journal FILE or stdin. That is consistent with iommu_fault_census.py, so it is my habit that was wrong rather than the interface — worth a one-line usage example in the header for the next person who pipes dmesg at it.

@bong-water-water-bong
bong-water-water-bong merged commit a0df746 into main Nov 15, 2026
10 of 11 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the lane/driver-op-names branch November 15, 2026 17:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant