[Bug] Qwen3-TTS 0.6B TTS output is incorrect on Jetson unless Talker / CodePredictor / Code2Wav contracts are fixed

[suharvest]

Context

First of all, thank you for maintaining TensorRT-Edge-LLM and for the recent Qwen3 ASR/TTS/Omni work. We are building an on-device, low-latency voice application on Jetson, and this library has been very useful for that work.

Our target deployment is:

• Qwen3-ASR on Jetson
• Qwen3-TTS 0.6B on Jetson
• local real-time conversational applications

During deployment, Qwen3-ASR worked correctly, but Qwen3-TTS did not. The TTS pipeline could export/build/run and produce a .wav file, but the speech content was wrong or contained obvious artifacts. After debugging, we found several runtime / engine-contract issues. I am opening this issue to share a minimal repro and the fixes we used so the official implementation can be corrected.

For our own application we are also adding streaming support and some training/adaptation capabilities on top of this project. Because our use case requires low latency, we have a separate high-performance branch with streaming TTS adaptations. With the corrected runtime contracts, we have already run two voice applications on an 8GB Jetson Orin Nano, with Qwen3-ASR and Qwen3-TTS 0.6B loaded at the same time. The performance is very promising. If maintainers are interested, we can also share that high-performance / streaming branch separately. This is why we would like to contribute the correctness findings back upstream.

This issue itself is intentionally non-streaming and minimal, so it is easier to reproduce and review.

Related issues checked

• support qwen3-tts-0.6B #86
  • Similar model: Qwen/Qwen3-TTS-12Hz-0.6B-Base.
  • That issue mainly discusses whether the 0.6B model has an MTP module.
  • The maintainer clarified that Qwen3-TTS has a CodePredictor, which is unrelated to Qwen3.5 speculative-decoding MTP.
  • This report is different: the TTS pipeline can produce WAV files, but the generated speech content is incorrect unless the Talker / CodePredictor / Code2Wav contracts are handled correctly.
• streaming output for qwen3-tts #61
  • About streaming output for Qwen3-TTS. This report is a non-streaming correctness repro.
• [Question] Qwen3-Omni Full Support on Thor/Orin (v0.6.0) #70
  • General Qwen3-Omni support question. This report is a concrete Qwen3-TTS 0.6B runtime correctness issue.

Environment

• Repository: NVIDIA/TensorRT-Edge-LLM
• Baseline: origin/main / v0.7.0-era code
• Device used for repro: Jetson Orin NX / SM87
• Additional deployment validation: Jetson Orin Nano 8GB
• TensorRT: 10.3
• CUDA: 12.6
• Model: Qwen/Qwen3-TTS-12Hz-0.6B-Base
• Test mode: non-streaming examples/omni/qwen3_tts_inference

What fails

The failing case is not simply a crash. The current/generic path can complete successfully and write a valid WAV container, but the speech content is wrong.

Examples from our validation:

• Non-high-performance runtime + native CodePredictor + stateless Code2Wav:
  • Input: 你好，今天天气真不错。
  • ASR result on one device: 有你好
  • ASR result on another device: 你好，你好
• Other minimal stateless/projection experiments:
  • ASR often recognized only 嗯, 你, or other partial text.
• In one stateless Code2Wav variant, the vocoder also hit CUDA illegal memory access.

So checking only process exit status or whether a WAV file exists is not enough for Qwen3-TTS validation. The output needs semantic validation, for example by ASR or an equivalent correctness test.

ASR-verified successful output

With the corrected minimal path, we generated:

Input:

您好，今天天气真不错。

Output:

• Generated frames: 49
• Output samples: 94080
• Duration: 3.92s at 24 kHz
• ASR device 1: 您好，今天天气真不错
• ASR device 2: 您好，今天天气真不错

We can attach the generated WAV:

qwen3-tts-minimal-stateful-slim.wav

We also generated an English sample that can be attached:

qwen3-tts-minimal-stateful-en.wav

Findings / suspected root causes

1. Talker prefill contract

The Qwen3-TTS explicit Talker engine may expose an inputs_embeds profile with max sequence length 1, while its KV cache supports a longer sequence. The runtime must distinguish:

• max input sequence length for the current enqueue
• max KV sequence length / cache capacity

If the whole prompt prefill is sent at once to a single-token input profile, it fails. If the runtime falls back to a generic prefill layout, the generated codec stream can be semantically wrong. Iterative prefill fixed this for our explicit-KV Talker path.

2. CodePredictor contract

Qwen3-TTS uses a CodePredictor to generate residual codec codebooks. This should not be treated as Qwen3.5 speculative-decoding MTP, but the runtime still needs a native Qwen3-TTS CodePredictor path that matches its engine contract and residual-code generation semantics.

The working path uses the native Qwen3-TTS CodePredictor engine and the expected residual-code generation contract.

3. Code2Wav state contract

The vocoder path that produced correct audio is stateful. The engine exposes codes / waveform plus recurrent state bindings named like:

<name>_in
<name>_out

Ignoring those state tensors, or using a stateless Code2Wav runner for this engine contract, can still produce a WAV but with incorrect content, or can fail at runtime.

The stateful Code2Wav engine profile we validated has:

codes min/opt/max = [1, 16, 1] / [1, 16, 4] / [1, 16, 16]

So runtime chunks must not exceed 16 codec frames for that engine.

Minimal implementation used to verify

We reduced the successful path to a non-streaming example:

• No worker.
• No streaming protocol.
• No async queue.
• No service framework.
• A small StatefulCode2WavRunner that mirrors the existing Code2WavRunner::generateWaveform() API.
• qwen3_tts_inference selects the stateful runner when code2wav_stateful.engine exists under --code2wavEngineDir; otherwise it keeps the existing stateless fallback.

Final stateful runner integration diff size:

cpp/multimodal/statefulCode2WavRunner.cpp | 518 ++++++++++++++++++++++
cpp/multimodal/statefulCode2WavRunner.h   | 122 +++++
examples/omni/qwen3_tts_inference.cpp     |  32 +-

The full local validation stack also includes the Talker / CodePredictor / export / Jetson build fixes needed to run from origin/main.

Branches / patches

We can provide both a minimal correctness repro branch and a separate high-performance branch.

Minimal correctness repro

qwen3tts-minimal-stateful-repro

Purpose:

• Minimal non-streaming correctness repro.
• Closest to the official example flow.
• Intended as the main reference for this issue.

Patch generated from origin/main..qwen3tts-minimal-stateful-repro:

git format-patch origin/main..qwen3tts-minimal-stateful-repro --stdout > qwen3-tts-minimal-repro.patch
git am qwen3-tts-minimal-repro.patch

Patch stack used in our local repro:

b9c57c8 Freeze Qwen3-TTS direct BF16 reference runtime
b09f644 Fix CuTe DSL CUDA 12.x compatibility for Talker MLP kernels
d4ccf66 Fix CuTe DSL shim propagation to static library consumers
2077555 Fix tokenizer loading for Qwen3-TTS models
1eebd83 Make quantization import optional in llm_export.py
111a684 Make modelopt import lazy in model_utils.py
8d21f47 Add tensor dtype accessor for Qwen3-TTS validation
32446c1 fix: Add minimal stateful Qwen3-TTS Code2Wav repro

The last commit is only the stateful Code2Wav runner + minimal non-streaming example integration. The earlier commits are the Talker / CodePredictor / export / build fixes needed to make the full repro run from origin/main.

If using my fork, the branch URL will be:

https://github.com/suharvest/TensorRT-Edge-LLM/tree/qwen3tts-minimal-stateful-repro

High-performance / streaming reference

highperf/runtime-service

Purpose:

• Low-latency runtime-service path for our product/application needs.
• Includes streaming-oriented Qwen3 ASR/TTS work and service integration.
• Useful as an additional reference if maintainers are interested in streaming behavior or performance tradeoffs.
• Not intended as the minimal fix for this issue.

If using my fork, the branch URL will be:

https://github.com/suharvest/TensorRT-Edge-LLM/tree/highperf/runtime-service

There is also a W8A16/high-performance experimental branch:

qwen3-tts-highperf-runtime-w8a16

That one is more experimental and broader in scope. I would recommend reviewing qwen3tts-minimal-stateful-repro first for correctness, and only looking at the high-performance branches if the streaming/runtime-service direction is useful.

Reproduction outline

1. Apply patch

git clone https://github.com/NVIDIA/TensorRT-Edge-LLM.git
cd TensorRT-Edge-LLM
git checkout origin/main
git am /path/to/qwen3-tts-minimal-repro.patch
git submodule update --init --recursive

2. Export Qwen3-TTS 0.6B ONNX

Export on an x86 host. Qwen3-TTS currently requires qwen-tts, so we used a dedicated Python environment.

cd ~/TensorRT-Edge-LLM
python3 -m venv venv-qwen3-tts
source venv-qwen3-tts/bin/activate
pip install .
pip install qwen-tts

export TTS_MODEL=Qwen3-TTS-12Hz-0.6B-Base
export MODEL_DIR=Qwen/$TTS_MODEL
export ONNX_OUTPUT_DIR=$HOME/tensorrt-edgellm-workspace/$TTS_MODEL/onnx
export TTS_CHAT_TEMPLATE=./tensorrt_edgellm/chat_templates/templates/qwen3tts.json

tensorrt-edgellm-export-llm \
  --model_dir $MODEL_DIR \
  --output_dir $ONNX_OUTPUT_DIR \
  --chat_template $TTS_CHAT_TEMPLATE \
  --export_models talker,code_predictor

tensorrt-edgellm-export-audio \
  --model_dir $MODEL_DIR \
  --output_dir $ONNX_OUTPUT_DIR \
  --export_models tokenizer_decoder

Expected export layout:

$ONNX_OUTPUT_DIR/
├── llm/
│   ├── talker/
│   └── code_predictor/
└── audio/
    └── tokenizer_decoder/

The CodePredictor directory must include:

codec_embeddings.safetensors
lm_heads.safetensors
small_to_mtp_projection.safetensors

3. Build TensorRT-Edge-LLM on Orin NX / TRT 10.3 / SM87

cd ~/TensorRT-Edge-LLM
mkdir -p build_trt103_sm87_aarch64
cd build_trt103_sm87_aarch64

PATH=/usr/local/cuda/bin:$PATH cmake .. \
  -DTRT_PACKAGE_DIR=/usr \
  -DAARCH64_BUILD=ON \
  -DCMAKE_CUDA_COMPILER=/usr/local/cuda/bin/nvcc \
  -DCUDA_DIR=/usr/local/cuda \
  -DCUDA_CTK_VERSION=12.6 \
  -DCMAKE_CUDA_ARCHITECTURES=87 \
  -DENABLE_CUTE_DSL=gemm \
  -DCMAKE_BUILD_TYPE=Release

PATH=/usr/local/cuda/bin:$PATH cmake --build . \
  --target edgellmCore qwen3_tts_inference NvInfer_edgellm_plugin -j2

4. Build engines / prepare artifacts

The runtime repro needs this artifact layout:

$ENG/
├── llm/
│   ├── llm.engine
│   ├── config.json
│   ├── text_embedding.safetensors
│   ├── text_projection.safetensors
│   ├── embedding.safetensors
│   └── tokenizer / chat-template files
├── talker_explicit_kv/
│   └── talker_decode_bf16_dual.engine
├── code_predictor/
│   └── cp_dir/
│       ├── qwen3_tts_cp.engine
│       ├── cp_embed_fp32.bin
│       ├── codec_embeddings.safetensors
│       ├── lm_heads.safetensors
│       ├── small_to_mtp_projection.safetensors
│       └── config.json
└── code2wav_stateful/
    ├── code2wav_stateful.engine
    └── config.json

Example preparation commands:

cd ~/TensorRT-Edge-LLM

export TTS_MODEL=Qwen3-TTS-12Hz-0.6B-Base
export WORKSPACE_DIR=$HOME/tensorrt-edgellm-workspace
export ONNX=$WORKSPACE_DIR/$TTS_MODEL/onnx
export ENG=$WORKSPACE_DIR/$TTS_MODEL/engines

export EDGELLM_PLUGIN_PATH=$PWD/build_trt103_sm87_aarch64/libNvInfer_edgellm_plugin.so
export LD_LIBRARY_PATH=$PWD/build_trt103_sm87_aarch64:/usr/lib/aarch64-linux-gnu:/usr/local/cuda/lib64:$LD_LIBRARY_PATH

./build_trt103_sm87_aarch64/examples/llm/llm_build \
  --onnxDir $ONNX/llm/talker \
  --engineDir $ENG/llm \
  --maxInputLen 32 \
  --maxKVCacheCapacity 128 \
  --maxBatchSize 1

mkdir -p $ENG/code_predictor/cp_dir
cp $ONNX/llm/code_predictor/codec_embeddings.safetensors $ENG/code_predictor/cp_dir/
cp $ONNX/llm/code_predictor/lm_heads.safetensors $ENG/code_predictor/cp_dir/
cp $ONNX/llm/code_predictor/small_to_mtp_projection.safetensors $ENG/code_predictor/cp_dir/
cp $ONNX/llm/code_predictor/config.json $ENG/code_predictor/cp_dir/

mkdir -p $ENG/code2wav_stateful
cp $ONNX/audio/tokenizer_decoder/config.json $ENG/code2wav_stateful/

Important note: building only the current stateless code2wav.engine path is enough to reproduce the bug, but not enough to reproduce the ASR-correct output. The ASR-correct path requires the stateful Code2Wav engine contract above.

The most useful thing to check is whether the exported/built Qwen3-TTS engines match these runtime contracts. I can attach the exact ASR-verified WAVs and the minimal patch used to route qwen3_tts_inference through the stateful vocoder path.

5. Run non-streaming TTS

cd ~/TensorRT-Edge-LLM

export EDGELLM_PLUGIN_PATH=$PWD/build_trt103_sm87_aarch64/libNvInfer_edgellm_plugin.so
export LD_LIBRARY_PATH=$PWD/build_trt103_sm87_aarch64:/usr/lib/aarch64-linux-gnu:/usr/local/cuda/lib64:$LD_LIBRARY_PATH

export TTS_MODEL=Qwen3-TTS-12Hz-0.6B-Base
export WORKSPACE_DIR=$HOME/tensorrt-edgellm-workspace
export ENG=$WORKSPACE_DIR/$TTS_MODEL/engines
export OUT=/tmp/qwen3_tts_minimal_stateful
mkdir -p $OUT

cat >/tmp/qwen3_tts_input.json <<'JSON'
{
  "requests": [
    {
      "messages": [
        {
          "role": "user",
          "content": "您好，今天天气真不错。"
        }
      ]
    }
  ]
}
JSON

./build_trt103_sm87_aarch64/examples/omni/qwen3_tts_inference \
  --talkerEngineDir $ENG/llm \
  --codePredictorEngineDir $ENG/code_predictor/cp_dir \
  --code2wavEngineDir $ENG/code2wav_stateful \
  --tokenizerDir $ENG/llm \
  --inputFile /tmp/qwen3_tts_input.json \
  --outputFile $OUT/output.json \
  --outputAudioDir $OUT \
  --dumpOutput

Expected result:

$OUT/audio_req0.wav

The important correctness check is semantic: the generated WAV should be
recognized as the input text by ASR, rather than only verifying that a WAV file
was written successfully.

Validation artifact links

The WAV files and JSON outputs are also available from this artifact branch:

• Branch: https://github.com/suharvest/TensorRT-Edge-LLM/tree/qwen3tts-issue-artifacts-20260516/docs/deploy-container
• Chinese WAV: https://raw.githubusercontent.com/suharvest/TensorRT-Edge-LLM/qwen3tts-issue-artifacts-20260516/docs/deploy-container/qwen3-tts-minimal-stateful-slim.wav
• Chinese output JSON: https://raw.githubusercontent.com/suharvest/TensorRT-Edge-LLM/qwen3tts-issue-artifacts-20260516/docs/deploy-container/qwen3-tts-minimal-stateful-slim-output.json
• English WAV: https://raw.githubusercontent.com/suharvest/TensorRT-Edge-LLM/qwen3tts-issue-artifacts-20260516/docs/deploy-container/qwen3-tts-minimal-stateful-en.wav
• English output JSON: https://raw.githubusercontent.com/suharvest/TensorRT-Edge-LLM/qwen3tts-issue-artifacts-20260516/docs/deploy-container/qwen3-tts-minimal-stateful-en-output.json

[nvluxiaoz]

Thanks and this is a great finding! Could you try the latest release/0.7.1 version? This version adds correct support to Qwen3-TTS 0.6B with runtime streaming support. The end to end example might not be ready but could be achieved with some quick changes.

[sc-hua]

Hi, @nvluxiaoz
will these fixes be merged in next release soon?

[suharvest]

Follow-up after testing v0.7.1 on the same Orin NX setup

@nvluxiaoz Thanks for the pointer to release/0.7.1. We rebuilt and ran the same reproducer (Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice, sm_87, JetPack 6.x, CUDA 12.6, TRT 10.3) and validated end-to-end with an objective ASR loop (radxa sherpa-onnx SenseVoice model.int8.onnx, language='zh', use_itn=True) against the input text "今天天气真不错". The ASR transcription is the ground-truth check below.

Net result on v0.7.1 with the official Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice checkpoint (the only checkpoint family docs/source/user_guide/getting_started/supported-models.md says is supported for Qwen3-TTS):

State	ASR result	Note
v0.7.1 + default cmake	unintelligible (嗯嗯。)	early EOS after ~7 codec frames
v0.7.1 + -DENABLE_CUTE_DSL=gemm -DCUTE_DSL_ARTIFACT_TAG=sm_87 -DEMBEDDED_TARGET=jetson-orin	still wrong, different garbage	kernel runs but produces wrong row-1 output
v0.7.1 + our 3 patches + workaround	今天天气真不错。 ✅ char-exact	FP16 baseline
v0.7.1 + our 3 patches + workaround + W8A16 talker	今天天气真不错哦。 ✅ semantic match	peak unified memory −48.7%

We localized three distinct issues. Two we have clean patches for and just opened as separate PRs; one is in the prebuilt CuTe DSL artifact (no source available) and needs your team to look at it.

• PR A (Issue A below): fix https://github.com/NVIDIA/TensorRT-Edge-LLM/issues/87: hard-error instead of silent return when CuTe DSL GEMM is not compiled #97 — fix #87: hard-error instead of silent return when CuTe DSL GEMM is not compiled
• PR B (Issue B below): feat https://github.com/NVIDIA/TensorRT-Edge-LLM/issues/87: add CustomVoice language conditioning support for Qwen3-TTS #98 — feat #87: add CustomVoice language conditioning support for Qwen3-TTS

Issue A — talkerMLPKernels::invokeTalkerMLP silently writes nothing when CuTe DSL is OFF

cpp/kernels/talkerMLPKernels/talkerMLPKernels.cu (around lines 324–388) is wrapped in #ifdef CUTE_DSL_GEMM_ENABLED. When the macro is not set — which is the default cmake configuration — the function logs LOG_ERROR("CuTe DSL GEMM not compiled. Rebuild with -DENABLE_CUTE_DSL=gemm (or ALL).") and returns with the output tensor uninitialized. The downstream sampler then consumes whatever the buffer happened to contain, which produces a structured-but-wrong codec sequence rather than a hard failure. It took us a while to track this down because the runtime "succeeds" and produces a valid WAV container.

Proposed fix (small): change the else branch to TLLM_THROW(...) (or equivalent) so the build flag mismatch is impossible to miss. Optionally also tighten the build doc / quickstart so the required -DENABLE_CUTE_DSL=gemm -DCUTE_DSL_ARTIFACT_TAG=sm_87 -DEMBEDDED_TARGET=jetson-orin flags are mentioned in the Qwen3-TTS path.

Issue B — assistantPreambleKernel has no CustomVoice (language-conditioned) prefix layout

The supported-models doc says Qwen3-TTS support is limited to the CustomVoice checkpoints. The Python reference (modeling_qwen3_tts.py around lines 2120–2186) handles those checkpoints by emitting a 9-row prefix with a language_id codec token injected at the dedicated language row (between codec_think_bos_id and codec_think_eos_id):

if language_id is None:                          # base 12Hz / no-lang
    codec_prefill_list = [[codec_nothink_id, codec_think_bos_id, codec_think_eos_id]]
else:                                             # CustomVoice + language
    codec_prefill_list = [[codec_think_id, codec_think_bos_id, language_id, codec_think_eos_id]]

The v0.7.1 C++ kernel only implements the no-language (8-row) layout. So even after Issue A is fixed (CuTe DSL is on and the MLP actually writes its output), the Talker prefill embeddings are still wrong for the only supported family. We grepped v0.7.1 source for language_id / customvoice / codec_language and got 0 hits anywhere in cpp/, examples/omni/, or experimental/llm_loader/models/qwen3_tts/.

Proposed fix (medium, contained):

• cpp/kernels/talkerMLPKernels/talkerMLPKernels.{cu,h}: add the 9-row branch; inject codec_language_id at row 6.
• cpp/runtime/qwen3OmniTTSRuntime.{cpp,h}: parse "language" field from input, look up the integer codec id from a new codec_language_id map in the engine config.json, thread (speakerId, langId, prefixRows) through projectToTalkerInput.
• examples/omni/qwen3_tts_inference.cpp: forward the language field from input JSON.
• 12-language codec_language_id map at the talker config level (chinese=2055, english=2050, german=2053, italian=2070, portuguese=2071, spanish=2054, japanese=2058, korean=2064, french=2061, russian=2069, beijing_dialect=2074, sichuan_dialect=2062 — taken from the Qwen3-TTS CustomVoice public model config).

We have this implemented and validated and sent it as PR #98 — happy to break it up smaller if you prefer (e.g. start with just the kernel layout + runtime threading and defer the config schema in a follow-up).

Issue C — CuTe DSL gemm_ampere_medium_bias_silu_fp16 row-1 corruption (artifact bug, no source)

After A + B are applied, end-to-end is still broken because the prebuilt CuTe DSL kernel gemm_ampere_medium_bias_silu_fp16 (shipped as kernelSrcs/cuteDSLPrebuilt/cutedsl_aarch64_sm_87_cuda12.tar.gz, md5 e076c4d83e002db9cf1558abcb8c34d7, cutlass_dsl_version 4.4.1, build_date 2026-05-06T21:23:32Z, gpu_arch=sm_87) produces wrong output at row index 1 of every sub-block invocation during the text-projection MLP FC1.

Sub-block	Call shape	Row 0	Row 1	Tail rows
Prefix (M=9)	M=9, N=2048, K=2048	matches ref (max_diff ≤ 0.002, fp16 noise)	wrong (max_diff 0.65, cos 0.39, sign flips)	matches ref
Suffix (M=4)	M=4, N=2048, K=2048	matches ref	wrong (max_diff 0.16, cos 0.94)	matches ref

Properties of the failure:

• Deterministic / bit-exact: 10/10 reruns produce identical (wrong) bytes (md5 d1bfa2a1a234971cc92061445e306544 × 10). Rules out races / uninitialized memory.
• Row-position-dependent: row 0 and the tail rows are correct; row 1 of each sub-block call is wrong. Two independent sub-block calls (M=9 and M=4) within the same request both exhibit the pattern.
• Variant bisect is not available: in cpp/kernels/talkerMLPKernels/cuteDslGemmRunner.cu:768-784, Ampere runBiasSiLU has exactly one prebuilt variant (gemm_ampere_medium_bias_silu_fp16) — no _small_* or _large_* Ampere alternative ships in the artifact, so we cannot work around by influencing the variant dispatcher.

Strong hypothesis: partial-M-tile predication bug in the Ampere medium fused-bias-SiLU kernel for small M, since both failing rows are "position 1 within their sub-block" and the rest of the sub-block is correct.

We have a full investigation writeup with environment, minimal reproducer, what we ruled out, suspected root causes ranked, and the asks back to your team here (CC'd in case it's easier than re-pasting in this comment):

• docs/specs/cute-dsl-medium-gemm-bug-investigation.md in the fork
• public URL: https://github.com/suharvest/openvoicestream/blob/main/docs/specs/cute-dsl-medium-gemm-bug-investigation.md

Asks for your team on Issue C:

1. Verify gemm_ampere_medium_bias_silu_fp16 on sm_87 with M ∈ {3, 4, 9}, N=2048, K=2048 and bias broadcast enabled against a cuBLAS HGemm reference.
2. Either ship a fixed prebuilt artifact, expose the kernel source under kernelSrcs/cuteDSLPrebuilt/ so it can be debugged downstream, or point us at a known-good Ampere fused-bias-SiLU variant we can switch to.
3. Confirm whether partial-M-tile predication with bias broadcast is in the CuTe DSL regression suite for sm_87 — if not, this is the smallest test that would have caught the bug.

Our current production workaround for Issue C is the same env-var-based override mentioned in our original report (QWEN3_TTS_PRELOAD_TALKER_EMBEDS=<fp32 bin>) which is functional but obviously not what we'd want long-term.

Validation evidence we can share

If useful, we can also share:

• The 4 TRT engines (FP16 talker + W8A16 talker + CodePredictor + Code2Wav) and W8A16 ONNX, hosted at harvestsu/seeed-local-voice-artifacts/orin-nx/qwen3-tts-12hz-0.6b-customvoice/ on Hugging Face.
• A reproducer script that runs the patched binary + transfers the WAV to a second device + runs sherpa-onnx SenseVoice ASR and emits a PASS/FAIL JSON (scripts/verify_customvoice_tts_radxa_asr.sh in our fork — both FP16 and W8A16 paths reproduce byte-identical WAV md5s across reruns).

Re sc-hua's question above — PRs #97 and #98 are open; happy to iterate on shape, split, or test plan if you'd like a different cut. Issue C we can leave as an issue/discussion until your team has a chance to look at the artifact.

[suharvest]

Correction on Issue C — it's not a CuTe DSL artifact bug, it was on our side

After deeper investigation we can retract Issue C. The gemm_ampere_medium_bias_silu_fp16 row-1 corruption is not a kernel/artifact defect — the GEMM is faithful. The real root cause was a prompt-construction bug in our talker input.

How we ruled out the GEMM:

• cuBLAS fallback (built -DENABLE_CUTE_DSL=OFF, CuTe artifact not linked) produces a byte-identical wrong row-1 (cos 0.395) — two independent GEMM backends can't share the same per-row bug.
• Offline numpy recompute of SiLU(X·W1ᵀ+b1)·W2ᵀ+b2 from the dumped X and weights matches the C++ output bit-for-bit, while both deviate from the reference → the GEMM faithfully projects its input; the input was wrong.
• Embedding forensics: input_ids[1] embedded token 872 ("user"), but the reference expects 77091 ("assistant"). <|im_start|> (151644) and \n (198) coincide between the user and assistant prefixes, so only the role-token row diverged — which looked exactly like a "row 1" kernel bug.

Root cause: the talker prefill was fed the user role prefix instead of the assistant prefix. The text to synthesize is the assistant's spoken content, so the input must be assistant-framed (<|im_start|>assistant\n{text}…) → input_ids[:3] = [151644, 77091, 198]. The QWEN3_TTS_PRELOAD_TALKER_EMBEDS diagnostic masked this by injecting Python-computed (correctly-framed) embeds.

Fix: coerce the request to an assistant-role turn. Verified end-to-end (no preload, multiple texts, ASR round-trip exact). Issues A and B (PRs #97/#98) still stand.