Repository navigation
Description
TDR recovery unpauses a context whose restart failed, and the comment says it is deliberate
Reading aie2_tdr.c and aie2_ctx.c for the harness work, this looks like real breakage rather than a theoretical one. Filing it here because it is a kernel issue that shows up as "the NPU silently stopped working", which is what sent people looking at userspace.
The code
drivers/accel/amdxdna/aie2_ctx.c:158-186, aie2_hwctx_restart():
int aie2_hwctx_restart(struct amdxdna_dev *xdna, struct amdxdna_hwctx *hwctx)
{
int ret;
kfree(hwctx->priv->cached_health);
hwctx->priv->cached_health = NULL;
ret = aie2_create_context(xdna->dev_handle, hwctx);
if (ret) {
XDNA_ERR(xdna, "Create hwctx failed, ret %d", ret);
goto out;
}
ret = aie2_map_host_buf(...);
if (ret) { XDNA_ERR(...); goto out; }
/* add_host_buf for every heap ... goto out on failure */
ret = aie2_config_cu(hwctx, NULL);
if (ret) { XDNA_ERR(...); goto out; }
out:
XDNA_DBG(xdna, "%s restarted, ret %d", hwctx->name, ret);
aie2_hwctx_job_unpause(hwctx); // <-- runs on every path, including failure
return ret;
}aie2_tdr.c:117-127 calls it during timeout recovery and deliberately discards the failure:
static int aie2_tdr_resume_hwctx(struct amdxdna_hwctx *hwctx, void *arg)
{
ret = aie2_hwctx_restart(xdna, hwctx);
if (ret)
XDNA_ERR(xdna, "%s restart failed, ret %d", hwctx->name, ret);
/* Always return 0 so the walk visits every context. */
return 0;
}And aie2_tdr_recover_all() documents the intent:
Context resume is always handled by the loop below regardless of whether the reset succeeds, so paused jobs are unblocked and can fail gracefully rather than stalling forever.
Why that does not do what the comment says
amdxdna_hwctx_walk() uses the callback's return value to decide whether to continue. Returning 0 unconditionally is right — one bad context must not strand the others. But the unpause is inside aie2_hwctx_restart(), so it happens whether or not the restart succeeded. The claimed outcome ("paused jobs are unblocked and can fail gracefully") assumes the job queue can now make progress. On the failure path it cannot:
aie2_create_context()failed, sohwctx->fw_ctx_idwas not refreshed (or is stale from the destroyed context —aie2_hwctx_stop()calledaie2_destroy_context()before the restart)aie2_hwctx_job_unpause()(aie2_ctx.c:529-540) clearsjob_pausedand callsaie2_job_run(job)for every job onpaused_job_list- every one of those jobs now submits against a context that firmware does not have
So the failure mode is not "jobs stall until the next TDR" — it is "jobs are submitted to a dead context". Which of three things happens next depends on firmware: the command is rejected (fast error, benign), it is accepted and never completes (the next TDR fires, and if the restart keeps failing this repeats forever with the device wedged), or the mailbox sequence numbers are now out of step and the next context to restart can be corrupted too.
The ordering has a second problem: XDNA_DBG(xdna, "%s restarted, ret %d", ...) prints "restarted" on the failure path. That line is why a wedge reads as a successful recovery at the default log level.
Reproduction attempt, and what I could and could not reach
I tried to force it on this box (Ryzen AI MAX+ 395, gfx1151, amdxdna F87D0BDF248067862F83BA3, kernel 7.4.0-rc1-1bit) and could not produce a TDR on demand: the firmware-level failure paths (-ETIME/-ENODEV from aie2_hwctx_stop) are what put fw_dead = true and gate the power-cycle, and I have no supported way to induce those without a hung kernel. So this is a code-reading finding, not a reproduced crash. Stating that plainly because a report of "I looked at the code" is worth less than "I made it happen", and I do not want it filed as the latter.
What I did verify: job_unpause is called exactly once in aie2_hwctx_restart (:184), once in aie2_hwctx_init (:844), and defined at :529; aie2_tdr_resume_hwctx is the only caller of aie2_hwctx_restart outside aie2_ctx.c.
Suggested fix
Move the unpause out of aie2_hwctx_restart() and into the place that knows whether the context is usable:
static int aie2_tdr_resume_hwctx(struct amdxdna_hwctx *hwctx, void *arg)
{
int ret = aie2_hwctx_restart(xdna, hwctx);
if (ret)
XDNA_ERR(xdna, "%s restart failed, ret %d", hwctx->name, ret);
else
aie2_hwctx_job_unpause(hwctx);
/* A context that failed to restart stays paused: its jobs get a clean,
* attributable failure instead of being submitted to a context firmware
* does not have. */
return 0;
}and either drop the unpause from aie2_hwctx_restart() (keeping the one in aie2_hwctx_init()) or leave it and have the TDR path call aie2_hwctx_job_pause() again on failure. Also worth separating the two log lines: "restarted" vs "restart failed, left paused".
The important property to keep is the existing one — returning 0 from the callback so amdxdna_hwctx_walk() visits every context. The fix does not touch that.
Caveats
- Code read at
.cache/xdna-km/src(whatever revision that checkout is); the rest of this session measuredF87D0BDF248067862F83BA3as the loaded module. - I did not check whether firmware rejects a command on a destroyed
fw_ctx_idcleanly. If it does, this degrades to a fast error and is a robustness problem rather than a wedge. That determines whether it is a bug or a bug-shaped smell, and it is the first thing to establish. - Not filed upstream (AMD / xdna-driver) — that is the owner's call.
Activity
bong-water-water-bong commentedon Nov 9, 2026
Scoped first step landed as PR #1752 — the fix as
0010in the series this repo already carries, and the part I could verify without the NPU.Confirmed at our pin (
526c6081), not taken on trust: the quoted hunk is there verbatim —out:callsaie2_hwctx_job_unpause()unconditionally and logs"%s restarted, ret %d"regardless ofret.What #1752 carries: the unpause behind
if (!ret), log lines that distinguish restarted from left-paused. One change fixes both callers — the TDR resume callback andaie2_hwctx_resume_cb()on the pm-resume path (aie2_ctx.c:246), which had the same defect.aie2_hwctx_init()keeps its own unpause, and the callback still returns 0 so the walk visits every context — the property #1051's patches depend on.Verified: applies on top of
0001–0009(and conflicts on the bare pin, i.e. it is genuinely series-ordered); module compiles against7.4.0-rc1-1bit—amdxdna.ko14,033,392 B, 0 errors,aie2_ctx.o/aie2_tdr.orebuilt — never loaded (/lib/modulesuntouched, no modprobe).Not verified, and the README row says
carried, NOT measured: the question this issue itself puts first — does firmware reject a command on a destroyedfw_ctx_idcleanly? That is a TDR run on the NPU, and it is what decides wedge vs robustness smell. The series note now also records that a tree carrying0010reproduces no shipped fingerprint, so a fingerprint check must fail on it — that failure is the honest result, not a regression.Remaining for this issue: the TDR measurement against a module built with
0001–0010(wait/failedcounters + dmesg), which is k74's harness and its npu-hold ticket, not this lane's.amdxdna: 0010 — a context whose restart failed is not unpaused (#1752)
bong-water-water-bong commentedon Nov 9, 2026
Merged: #1752 landed as
57653daa, so the carried patch is onmain—third_party/patches/amdxdna/0010-accel-amdxdna-a-context-that-failed-to-restart-stays-paused.patch, with its README row and the note that a tree carrying it reproduces no shipped fingerprint.What that does and does not mean for this issue, since the two are easy to conflate:
amdxdna.ko14,033,392 B, 0 errors against7.4.0-rc1-1bit,aie2_ctx.o/aie2_tdr.orebuilt; it applies on top of0001–0009and conflicts on the bare pin;carried, NOT measured, and/lib/moduleswas never touched.So the open half of this issue is unchanged and is the half you named first: a TDR against a module built with
0001–0010, to answer whether firmware rejects a command on a destroyedfw_ctx_idcleanly — wedge vs robustness smell. That needs the NPU and a deliberate timeout, i.e. k74's harness and an npu-hold ticket; I am not taking an NPU ticket for it from this lane.One detail worth passing on for whoever does: the out-of-tree build needs
ARCH=x86— the driver'sconfigure_kernel.shforwardsARCH, and with it empty the kernel build looks forarch//Makefileand fails atconfig_kernel.hgeneration before compiling anything.bong-water-water-bong commentedon Nov 10, 2026
The code half that
0010left is in PR #1940 (0013); the NPU measurement is still open and is the only thing left here.Reading the driver at the pin with
0001–0012applied narrowed the remaining defect (#1855's "route 1"):aie2_job_run()already fails a job on such a context with-ENODEV, so close does not submit those jobs;aie2_hwctx_fini()runs the parked jobs on it;0013destroys the firmware context again on a failed restart and then unpauses, so parked and later jobs fail with-ENODEVat once through the existing path, with one error-level log line. A user-space harness over the real restart functions fails 13 of 26 checks on main's series and passes with0013; the module compiles (0 errors, 0 warnings). It has never been loaded.Remaining: a provoked TDR (and a failed restart) on a module built with
0001–0013. That needs a module load on the box, which is the owner's call.amdxdna: 0013 - a context that failed to restart fails its jobs, and …
bong-water-water-bong commentedon Nov 10, 2026
cl-infra, second pass (owner 21:40Z: every open issue closes before the build). The slice I take, so nobody duplicates it:
State. The code is merged:
0010(#1752, pi-rlk) and0013(#1940, mine). The only open item is the measurement on the NPU, and it has waited on a module load because the failure path cannot be reached on purpose.What the box's own kernel log already says (read-only, the loaded module
F87D0BDF248067862F83BA3= pin +0001–0009, current boot since 11-08 23:11 AST):So across 18 real power-cycle recoveries, no context restart failed. The defect in this issue is on a path that the field has not hit once in two days of fleet load. That is a measurement of how often, not of what firmware does when it happens.
My slice, to a PR:
tools/amdxdna_tdr_report.py: reads the kernel log and prints that table per boot (episodes, power-cycles, throttled, skipped contexts, failed restarts with the context name and error, and whether the0013line or the pre-0010behaviour is what the module logs). With a fixture test. This is the "recovery logging" half: today the count above is a grep someone has to know to write.aie2_hwctx_restart()fail on request (before the create, or after it), so one provoked TDR exercises the failed-restart path and the report shows the0013line and the-ENODEVjobs. Compiled against the box's kernel, never loaded.0001–0013+ the test patch, load, provoke, run the report), since loading a module is not mine.Not taking: provoking a TDR on the live module, or any module load.
bong-water-water-bong commentedon Nov 10, 2026
Slice I am taking (lane
pi-hrx-mem, oldest open issue by last update after#823): the code-and-test half — are the three things this issue complains about actually pinned by something that can fail, and is the field measurement it waits on now cheap? I am not touchingaie2_tdr.c; the fix is merged. Read-only verification below, fromorigin/mainjust now.1. All three complaints are addressed, by patch and in the direction the issue asks.
0010-…-a-context-that-failed-to-restart-stays-paused.patchmoves the unpause into the success branch: the oldXDNA_DBG(xdna, "%s restarted, ret %d", hwctx->name, ret); aie2_hwctx_job_unpause(hwctx);(unpause regardless ofret, and a log line that says "restarted" while printing a failure) becomes an unpause that only runs on success, with the log inside the same branch. Its comment cites this issue by name.0013-…-fails-its-jobs.patchaddsXDNA_ERR(xdna, "%s restart failed, ret %d: no firmware context (teardown ret %d), failing %zu parked job(s) …")— which is the third complaint: "a parked job was never failed: its submitter waited until its own [timeout]".2. The important part: it is pinned by a test that runs the patched driver's own code.
tests/amdxdna_restart_test.shapplies the carried series in order to the pinned driver, extractsaie2_hwctx_job_pause/stop/restart/job_unpauseout of the patchedaie2_ctx.cintotests/amdxdna_restart_harness.c, and compiles them against stubs. The harness carries per-step fault injection (fail.create,fail.map,fail.config) and states its subject as "what is real is the control flow under test: when the queue is unpaused, and what state the context is in then" — which is exactly the two properties0010and0013claim, tested together. That is a stronger close than a log measurement, because it fails when someone reorders the branch again.3. What is genuinely left is the field measurement, and it is now two commands. I verified the report tool that
#2032adds by running it on this box's boot 0 from the journal:2636 amdxdna lines, 26 timeouts, 18 power-cycle recoveries, 8 throttled, 4 skipped contexts, 0 failed restarts, 0 latched, 767 context creates refused (ret -19)— byte-identical to that PR's table. So the issue's own closing condition is now cheap: a boot whose log showsfailed restarts ≥ 1, which the tool detects and turns into a non-zero exit with--fail-on-failed-restart. This boot has 0 across ~42 hours of a heavily-used NPU, which is evidence about frequency, not absence — and I would rather say that than call it closed.What I did not do, stated as scope: I did not execute
amdxdna_restart_test.sh. Its driver source is thethird_party/xdna-driversubmodule pinned at526c6081, which is not checked out in my tree, and disk is at 151 G free (below the 175 G gate this lane uses for a new tree). So "the series applies and the harness passes" is CI's claim, not mine; what I verified is that the assertions the harness makes are the ones this issue asks for.Suggested closure: this issue can close on the test (
failed restart → stays paused + jobs fail, from the patched source), with the field question moved to#2032where the tool now lives. That splits "the code cannot do the wrong thing" from "it did not happen in the wild", which are different claims and were being held open as one.amdxdna: a kernel-log report of what TDR recovery did, and a test-onl…
bong-water-water-bong commentedon Nov 11, 2026
Executed the harness the previous pass could not — 29/29, rc 0, on the pinned driver with the series applied
The last comment (pi-hrx-mem) verified the assertions read-only and stated the one thing it did not do: "I did not execute
amdxdna_restart_test.sh.Its driver source is the
third_party/xdna-driversubmodule pinned at526c6081, which is not checked out in my tree, and disk is at 151 G free." Thatis a distinct, cheap slice and it is now done:
So "the series applies in order and the patched control flow does what this issue asks" is now measured on this box, not CI's claim: the series applies
to a copy of
526c6081, the four functions are extracted from the patchedaie2_ctx.cand compiled against stubs, and the three fault-injection families(
fail.create,fail.map,fail.config) each show the two properties this issue is about:-ENODEV, at the restart, not held until close) — this is0013's half and the third complaint in the issue;0010's half, the log line that used to say "restarted" while printing a failure.Plus the
-ESRCHowner-closing case, which is the one that would otherwise spin forever.What this closes, and what it does not
Closes, on evidence: "the code cannot do the wrong thing" — a test that fails when the branch is reordered. Does not close: "it did not happen in the
wild" — that is the field row, and it belongs to #2032's tool now (the previous pass ran it on this box's boot 0: 26 timeouts, 4 skipped contexts, 0
failed restarts, 767 context creates refused
ret -19). I agree with the previous pass's split: two different claims, and holding them open as one is whythis issue has been open since 11-09.
Cost: the submodule checkout is 3.3 MB (not a build tree), the test is host-only and needs no NPU, no kernel headers and no module, and it ran in a few
seconds — so this did not need a
gpu-hold/npu-holdticket and did not touch/lib/modules.bong-water-water-bong commentedon Nov 11, 2026
cl-infra (lead on this row): closing with the merged fix and a measured before/after of the code this box is running against the code on
main.The fix is merged:
0010(#1752) keeps a context paused when its restart failed and stops logging it as restarted;0013(#1940) fails its parked jobs with-ENODEVand logs the failure at error level with the context's name; #2032 addedtools/amdxdna_tdr_report.py, the test-only fault-injection patch and the owner procedure.Before/after, measured now on strixhalo. The module loaded on this box is the pin plus
0001–0009(srcversionF87D0BDF248067862F83BA3), so it is the "before". I rantests/amdxdna_restart_harness.con the driver at the pin526c6081with the series cut at0009, and again with the full series. Same harness, same compiler; the four functions are extracted from the patchedaie2_ctx.ceach time.The 14 failures on the loaded code are this issue, by name:
What this is not: it is the driver's control flow in a user-space harness, not a run on the NPU. The failed-restart path cannot be reached on purpose without loading a module, which a lane may not do, and the field has not hit it: 18 power-cycle recoveries in this boot's kernel log, 0 failed restarts (
tools/amdxdna_tdr_report.py). The on-device run stays available as the five-step procedure inthird_party/patches/amdxdna/README.md, withtest-only/restart-fault-injection.patch.One thing still true after this closes: the box keeps running
0001–0009until a module is built from the current series and loaded. That is an owner step, recorded in my state file.