Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -325,6 +325,9 @@ add_python_test(tool_selftest_whisper_encoder_baseline ${CMAKE_SOURCE_DIR}/tools
# engine#1780: the DFlash draft package's gate before a loader -- tile-grid sizes from the config, tile-header sanity and
# the container's scalars against config.json, on a synthetic package with eight negative controls. Stdlib only.
add_python_test(tool_selftest_dflash_draft_info ${CMAKE_SOURCE_DIR}/tools/dflash_draft_info.py --selftest)
# engine#1750: what a TDR recovery did, counted from the kernel log (failed context restarts among them). The self-test
# parses lines in the driver's own formats for three module generations. Stdlib only, no device.
add_python_test(tool_selftest_amdxdna_tdr_report ${CMAKE_SOURCE_DIR}/tools/amdxdna_tdr_report.py --self-test)
# engine#2013: the qwen35 (gated linear attention) GGUF -> Q4NX converter. Its own selftest (tile round-trips, the
# value-head permutation), and a pack + verify of a small qwen35 GGUF the test writes itself, with the three inputs it
# has to refuse. numpy only: no gguf package, no model, no device.
Expand Down
16 changes: 16 additions & 0 deletions tests/amdxdna_restart_test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,22 @@ for p in "$series"/[0-9][0-9][0-9][0-9]-*.patch; do
done
echo "ok $n patches apply in order on ${pin:0:12}"

# The test-only fault injection (engine#1750) has to keep applying on top of the series, or the one run it exists
# for cannot be built. It is checked, not applied: the harness below is the carried code. And it must stay out of
# the numbered glob above, which is what a package build takes.
inject=$series/test-only/restart-fault-injection.patch
if ! git -C "$scratch/tree" apply --check "$inject" 2> "$scratch/apply.err"; then
echo "FAIL test-only/restart-fault-injection.patch does not apply on top of the $n carried patches:"
sed 's/^/ /' "$scratch/apply.err" | head -n 5
exit 1
fi
echo "ok test-only/restart-fault-injection.patch applies on top of them (checked, not applied)"
if ls "$series"/[0-9][0-9][0-9][0-9]-*.patch | xargs grep -l "TEST ONLY" > "$scratch/leak" 2> /dev/null; then
echo "FAIL a numbered patch carries TEST ONLY code: $(tr '\n' ' ' < "$scratch/leak")"
exit 1
fi
echo "ok no numbered patch carries TEST ONLY code"

# one function definition: from its header line (column 0, not a prototype) to the first closing brace in column 0
for f in aie2_hwctx_job_pause aie2_hwctx_stop aie2_hwctx_restart aie2_hwctx_job_unpause; do
awk -v f="$f" '!on && $0 ~ "^[a-z].*[ *]" f "\\(" && $0 !~ /;[ \t]*$/ { on = 1 } on { print } on && /^}/ { exit }' \
Expand Down
54 changes: 54 additions & 0 deletions third_party/patches/amdxdna/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -195,6 +195,60 @@ To load the pair (owner's call; strixhalo, 2026-11-06): the module built from th
`7.4.0-rc1-1bit` reports `336751493A77C50DE0B50F7`, and the install has to sign it, `depmod`, and **`update-initramfs`** —
the initramfs copy is what boots (2026-11-04: three reboots on the old module without that step).

## Reading what a recovery did: `tools/amdxdna_tdr_report.py`

The driver logs every step of a TDR recovery, and `tools/amdxdna_tdr_report.py` turns the kernel log into the counts
per module load: timeouts detected, power-cycle recoveries, throttled recoveries, skipped contexts, **failed restarts**
(context, error, the step that failed, and whether the module failed the parked jobs, i.e. carries `0013`), whether the
array latched, and refused context creates by error. It reads `journalctl -k -b` or saved logs, opens no device, and
`--fail-on-failed-restart` makes it a check.

The record on strixhalo with the module the fleet runs (`F87D0BDF248067862F83BA3`, pin + `0001`-`0009`), one boot,
2026-11-08 23:11 to 2026-11-10 17:40 AST, under the fleet's normal load:

```
$ python3 tools/amdxdna_tdr_report.py
module load 1: driver 0.18.0 (2636 amdxdna lines)
timeouts detected 26
power-cycle recoveries 18
throttled 8
skipped contexts 4
failed restarts 0
latched (reload needed) 0
context creates refused 767 (ret -19 x767)
```

So in 18 real recoveries no context restart failed: the path `0010` and `0013` change (engine#1750) was not reached
once. That says how often it happens here, not what firmware does when it does.

## The test-only patch, and the run that would measure `0010`/`0013`

`test-only/restart-fault-injection.patch` is **not part of the series**. It is outside the numbered files, so
`scripts/mk-kernel-packages.sh` (which takes patches by number) and the series loop in `tests/amdxdna_restart_test.sh`
never pick it up; that test checks both that it still applies on top of the series and that no numbered patch carries
`TEST ONLY` code. It adds two module parameters that make `aie2_hwctx_restart()` fail on request, before or after the
firmware context is created, because a real restart failure needs firmware to refuse a message and cannot be arranged
from user space.

Verified without the device: applies on `0001`-`0013`; the module compiles against `7.4.0-rc1-1bit`
(`amdxdna.ko` 14,045,256 B, `srcversion 57DF58D0185B9D6E0397B6F`, 0 errors, 0 warnings; `modinfo` lists both
parameters). **Never loaded.**

The run, which needs a module load and is therefore the owner's:

1. Build pin + `0001`-`0013` + the test-only patch (`make -C drivers/accel/amdxdna ARCH=x86`), load it on a box nobody
else is using the array on.
2. Start one client that keeps a context open and submits (any `npu-hold` serve).
3. `echo 1 > /sys/module/amdxdna/parameters/restart_inject_count`, then `echo 2 > .../restart_inject_step`
(2 = after the create, the case that left a half-built firmware context), and provoke one recovery (the kernel
lane's `tdr_test` hook, or a real timeout).
4. `python3 tools/amdxdna_tdr_report.py`. Expected with `0013`: one failed restart, with
`firmware context taken away (teardown ret 0), N parked job(s) failed with -ENODEV`, and the client's pending
requests returning at once instead of at their own timeout. `teardown ret` is the answer to the question engine#1750
put first: whether firmware accepts a destroy for a context it has only half of. The array then latches (a failed
restart in a recovery latches `hw_dead` by design), so reload the shipping module afterwards.
5. Repeat with `restart_inject_step` 1 (create failed: no firmware context to take away).

## What this directory deliberately does not carry

The **0010 debugfs test hook** (`debugfs/accel/<pci fn>/tdr_test`, root-only, commands `recover` / `throttle` /
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
From: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Subject: [PATCH TEST ONLY] accel/amdxdna: fail a context restart on request

TEST ONLY. Not part of the carried series, never built into a package or an
image: it sits in test-only/ so that nothing that takes the numbered patches
picks it up.

aie2_hwctx_restart() fails only when firmware refuses one of its four
messages, which cannot be arranged from user space, so what the driver does
with a context whose restart failed (engine#1750, patches 0010 and 0013)
could not be measured on the array. Two module parameters make the next
restarts fail on purpose:

restart_inject_step 0 = off
1 = fail before the firmware context is created
(as if aie2_create_context() had failed)
2 = fail after it was created
(as if aie2_map_host_buf() had failed)
restart_inject_count how many restarts to fail; each one takes one

Each injected failure logs "TEST ONLY: injecting a restart failure for
<ctx> at step <n>" at error level, followed by the driver's own lines for a
failed restart. Applies on top of 0001-0013.
---
diff --git a/drivers/accel/amdxdna/aie2_ctx.c b/drivers/accel/amdxdna/aie2_ctx.c
index 100a07d..55d99a7 100644
--- a/drivers/accel/amdxdna/aie2_ctx.c
+++ b/drivers/accel/amdxdna/aie2_ctx.c
@@ -34,6 +34,36 @@ uint tdr_timeout_ms = 2000;
module_param(tdr_timeout_ms, uint, 0400);
MODULE_PARM_DESC(tdr_timeout_ms, "TDR (Timeout Detection and Recovery) timeout in milliseconds (0 = disable)");

+/*
+ * TEST ONLY (engine#1750), never part of a shipped module: make the next
+ * context restarts fail on purpose, so the failed-restart path can be
+ * measured with one provoked TDR instead of waiting for firmware to
+ * refuse a restart by itself.
+ *
+ * restart_inject_step: 0 = off; 1 = fail before the firmware context is
+ * created (as if aie2_create_context() had failed); 2 = fail after it was
+ * created (as if aie2_map_host_buf() had failed).
+ * restart_inject_count: how many restarts to fail; each injected failure
+ * takes one, and at 0 the injection is off again.
+ */
+static uint restart_inject_step;
+module_param(restart_inject_step, uint, 0600);
+MODULE_PARM_DESC(restart_inject_step, "TEST ONLY: fail a context restart (1 = before create, 2 = after create)");
+static uint restart_inject_count;
+module_param(restart_inject_count, uint, 0600);
+MODULE_PARM_DESC(restart_inject_count, "TEST ONLY: number of context restarts restart_inject_step still fails");
+
+static bool aie2_restart_inject(struct amdxdna_dev *xdna, struct amdxdna_hwctx *hwctx, uint step)
+{
+ if (READ_ONCE(restart_inject_step) != step || !READ_ONCE(restart_inject_count))
+ return false;
+
+ WRITE_ONCE(restart_inject_count, READ_ONCE(restart_inject_count) - 1);
+ XDNA_ERR(xdna, "TEST ONLY: injecting a restart failure for %s at step %u (%u left)",
+ hwctx->name, step, READ_ONCE(restart_inject_count));
+ return true;
+}
+
struct aie2_ctx_health {
struct amdxdna_ctx_health header;
u32 txn_op_idx;
@@ -174,6 +204,12 @@ int aie2_hwctx_restart(struct amdxdna_dev *xdna, struct amdxdna_hwctx *hwctx)
goto out;
}

+ if (aie2_restart_inject(xdna, hwctx, 1)) {
+ ret = -EIO;
+ XDNA_ERR(xdna, "Create hwctx failed, ret %d", ret);
+ goto out;
+ }
+
ret = aie2_create_context(xdna->dev_handle, hwctx);
if (ret) {
XDNA_ERR(xdna, "Create hwctx failed, ret %d", ret);
@@ -183,6 +219,8 @@ int aie2_hwctx_restart(struct amdxdna_dev *xdna, struct amdxdna_hwctx *hwctx)
ret = aie2_map_host_buf(xdna->dev_handle, hwctx->fw_ctx_id,
amdxdna_obj_dma_addr(heap),
heap->mem.size);
+ if (!ret && aie2_restart_inject(xdna, hwctx, 2))
+ ret = -EIO;
if (ret) {
XDNA_ERR(xdna, "Map host buf failed, ret %d", ret);
goto out;
Loading
Loading