Skip to content

onnx: a public ONNX-native int8 export cannot even create a session (no ConvInteger kernel in the pinned ORT) #1436

Closed

Description

@bong-water-water-bong

Found while measuring the TGPNPU vision rows against our ORT plugin EP (#647). It is not an EP problem: the session never
starts.

Reproduced

Xenova/yolov8n-pose (AGPL-3.0), the onnx/model_int8.onnx export, 3,755,740 B — a very common shape of public
"quantized ONNX" (Hugging Face Optimum / Xenova exports). Through our runner:

$ build/ort_run --model model_int8.onnx --input images=in.bin:float32:1x3x640x640
onebit-ort-run: FAIL create session: Could not find an implementation for
    ConvInteger(10) node with name '/model.0/conv/Conv_quant'

The graph is 739 nodes, all in the default ai.onnx domain, opset 17: ConvInteger 73, DynamicQuantizeLinear 65,
Mul/Add/Cast/Sigmoid/Reshape, and no MatMul. So it is not a graph our EP could serve either way — but it does not
get that far, because ORT 1.23.2 (our pin, docs/onnx.md "The pin") has no CPU kernel for ConvInteger.

Why it matters beyond one row

"Quantized ONNX" is at least two formats, and our docs and boards do not currently distinguish them:

flavour example op vocabulary our status
fp32 weights + in-graph Q/DQ AMD's *_amdgpu / aws exports, ~/models/q8/qwen3-0.6b-a8w8-proj-pc.onnx QuantizeLinear/DequantizeLinear around float ops runs; our EP claims these where K/N are static
ONNX-native integer ops Xenova/yolov8n-pose model_int8.onnx, many Optimum exports ConvInteger, MatMulInteger, DynamicQuantizeLinear does not load at all

So a row that "fails" can be the runtime's op coverage rather than the model or our EP, and a reader cannot tell from the current
board. That is the class of misattribution this project keeps trying to avoid.

Options (not decided here)

  1. Say it out loud and route around it: the OOB path for such repos uses the fp32 export when the repo ships one (it does
    here: onnx/model.onnx, 13,484,153 B, 311 nodes, 73 Conv, runs at 62 ms CPU), and the board records the int8 variant as
    "runtime has no ConvInteger" instead of "model fails".
  2. Build the pinned ORT with the quantized-op kernels — needs a check whether 1.23.2 ships ConvInteger for int8
    activations at all, or only uint8 (the ONNX spec allows both; the error does not say).
  3. Support it in our EP — out of scope for a plugin EP: ConvInteger is a conv, and our seam is MatMul.

Acceptance for this issue

A row that ships an ONNX-native int8 export either runs, or says in one line that the pinned runtime has no kernel for it and
which export of the repo does run. No silent "does not load".

Found in: ~/amd-oss/ONNX-VISION-OOB.md (§4a's trap note) and the GPNPU-PARITY.md YOLOv8-pose row, both of which now record
both exports separately.

Activity

  1. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Claiming this: lane cl-1077. It was unowned (no comments, no board entry; the filer's state file records it as filed, not taken). Plan:

    1. Check whether ORT 1.23.2's CPU ConvInteger exists only for uint8 x uint8, and what this export actually feeds it.
    2. Make ort_run turn "Could not find an implementation for (N)" into one line that names the missing kernel and points to the repo's float/QDQ export.
    3. Add the two-flavour table to docs/onnx.md.
      CPU only.
  2. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Second data point, different model, same exporter family — so this is not one repo's quirk.

    onnx-community/yolo26n-ONNX, onnx/model_int8.onnx, 2,848,000-ish B (the same optimum/onnx-community export pipeline that produced the Xenova file):

    graph: 999 nodes, ConvInteger 102, DynamicQuantizeLinear 92, MatMulInteger 0, Q/DQ 0  (opset 18, all ai.onnx)
    $ build/ort_run --model model_int8.onnx --input pixel_values=in.bin:float32:1x3x640x640
    onebit-ort-run: FAIL create session: Could not find an implementation for ConvInteger(10) node with name '/model.0/conv/Conv_quant'
    

    Identical failure, identical message, at session creation — and note the counts: every conv in the model is a ConvInteger, so for these exports there is nothing to fall back to inside the graph.

    Two things this sharpens for the fix chosen in this issue:

    • the fp32 export of the same repo runs fine (onnx/model.onnx, 9,890,454 B, 397 nodes, 102 Conv, 4 MatMul): 1,270 ms on the CPU EP, and 1,797 ms with our plugin EP registered — which claimed 0 of 400 offered nodes because those four matmuls are dynamically shaped. So option 1 in the issue (route the row to the repo's fp32 export, say so on the board) is measured to work on a second row now.
    • the failure is a runtime op-coverage question, not a model or an EP question: ConvInteger is ai.onnx opset 10, and our pin is 1.23.2. Whether the pin implements it only for uint8 activations (the ONNX spec allows int8 and uint8) is still unchecked — that is the one thing worth knowing before deciding between "document and route around it" and "build the pin differently".

    No action requested; recorded so the next lane that meets a model_int8.onnx from this exporter does not spend an hour on it.

  3. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Answering this issue's own open question (option 2): the pinned ORT 1.23.2 does implement ConvInteger — but only for uint8 × uint8. Measured with three minimal single-node graphs, each run through our own runner with the vendored pinned ORT:

    probe (ConvInteger, opset 10, IR 10) session
    x uint8, W uint8 creates and runs (run: 97.93 ms, int32 output)
    x uint8, W int8 fails: Could not find an implementation for ConvInteger(10)
    x int8, W int8 fails, same message

    So it is not "the pin has no ConvInteger" — it is type-combination coverage, and the combination the public exports use is the unsupported one. Checked in the failing file itself (Xenova/yolov8n-pose model_int8.onnx):

    ConvInteger inputs (x, W, xz, wz) = ('dyn' → uint8 from DynamicQuantizeLinear, 'INT8', 'dyn', 'INT8')   ×73 nodes
    

    i.e. unsigned activations × signed weights, which is precisely probe row 2. The same holds for the onnx-community/yolo26n-ONNX file (102 ConvInteger, same exporter family).

    What that changes for the fix

    • Option 1 stands and is the cheap path: route the OOB row to the repo's fp32 export (measured to run on both repos: 62 ms and 1,270 ms CPU) and say on the board that the int8 variant is a runtime op-coverage limit, naming this issue.
    • A workaround now exists and is bounded: re-quantize the same weights to uint8 × uint8 — ONNX's QOperator with QUInt8 weights — which the pin executes. It costs a quantisation step per repo, so it is a "when a row really needs int8" answer rather than the default.
    • Option 2 (build the pin differently) is unnecessary on this evidence; the kernel is there, the registration is narrow.

    Nothing here changes the EP's own scope: ConvInteger is a conv, our seam is MatMul, so even a working ConvInteger would run on the CPU EP — recorded so nobody reads this as an EP gap.

    Evidence reproducible from ~/lb/onebp-ort2/ort_run plus three 300-byte models; the probes are ~50 lines of onnx.helper and I can commit them as a ctest if you want this pinned rather than commented.

  4. bong-water-water-bong commented on Nov 8, 2026

    @bong-water-water-bong
    CollaboratorAuthor

    Closed as completed — recording what that does and does not cover, so the closure is not read as "no residual".

    Done and pinned in the tree: the runtime behaviour, as a ctest. #1450 (merged 1b480031) runs three single-node graphs and asserts that uint8 × uint8 ConvInteger works — including its arithmetic — while uint8 × int8 and int8 × int8 are reported rather than asserted, so a runtime that gains them says so instead of turning the test red. That is the fact every row in this area depends on, and it no longer lives in an issue comment.

    Not done, and not a code defect: the policy this issue was really about — what an OOB row does when a repo ships only an ONNX-native int8 export and no fp32 export. Today's answer is a convention, not a mechanism: route to the fp32 export when the repo has one (measured on two repos: Xenova/yolov8n-pose, onnx-community/yolo26n-ONNX), otherwise record "runtime has no kernel for this op/type combination" on the row. Whether that becomes a registry field, a refusal at fetch time, or nothing at all is a product call on the ONNX rows.

    That residual belongs to TAMDCOL's ONNX kinds, which are deferred by OWNER FOCUS ("AMDGPU ONNX (DirectML)" and the non-NPU items), so closing this now is the right sequencing — it is parked with its evidence attached rather than lost. If the ONNX rows come back off deferral, the policy question should be filed fresh against whoever owns those rows, with this issue as the reference: ~/amd-oss/ONNX-VISION-OOB.md §6b has the per-row state (8 of my 12 rows measured, 4 blocked on an artifact) and §4a has the two-flavour trap with the measured table.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions