fix: recover declared vision model inputs

[ssss141414]

Summary

Adds generalized WinML support hardening for nielsr/vitpose-base-simple keypoint-detection while preserving existing CPU fp32/fp16 recipe coverage through L2. The shipped outcome is L2 PASS for required tuples, plus mandatory final-head functional smoke eval PASS on real one-row COCO data in both trust modes (functional operability evidence only). Final refresh advances the PR head to 63bfd4a and records the d5fa845->63bfd4ad evaluator follow-up fix (non-null model_id guard/control-flow narrowing plus missing-model-id regression test), while preserving baseline main 9f74f91 provenance and Optimum WINML-ONLY status.

Model metadata

What the model does

VitPoseForPoseEstimation consumes normalized person-crop image tensors and outputs per-keypoint heatmaps for human pose keypoint localization.

• Evidence/confidence (verified): inspect task resolution to keypoint-detection with VitPose ONNX export config; Transformers VitPoseForPoseEstimation forward returns heatmaps.

Primary user stories

• A user supplies a cropped person image tensor to obtain keypoint heatmaps for downstream pose coordinate decoding.
  • Evidence/confidence (verified): VitPose forward(pixel_values) to heatmaps; model I/O contract uses 256x192 image input.

Supported tasks

• keypoint-detection across checkpoint, Transformers, Optimum ONNX, and WinML.
  • Evidence/confidence (verified): Optimum probe vendor=[] and after_winml=[keypoint-detection], added_by_winml=[keypoint-detection].

Model architecture

VitPoseForPoseEstimation
|- Backbone (load_backbone(config))
|  |- Final feature map extraction (outputs.feature_maps[-1])
|- Sequence-to-spatial reshape
|  |- permute(0,2,1)
|  |- reshape(batch, channels, patch_h, patch_w)
|- Head
   |- VitPoseSimpleDecoder (use_simple_decoder=true)
      |- ReLU
      |- Bilinear upsample (scale_factor)
      |- Conv2d(hidden_size -> num_labels heatmaps)

• Source/confidence (verified): pinned checkpoint metadata plus Transformers VitPoseForPoseEstimation and VitPoseSimpleDecoder implementation.

Validation and support evidence

1. Baseline

• Pinned main commit: 9f74f91 (WinML 0.3.0).
• Baseline floor: build FAIL with list index out of range.
• Baseline optimum probe: WINML-ONLY (vendor provided none; WinML adds keypoint-detection).
• Baseline refresh decision: PARTIAL-RERUN from previous evidence main e7b132d to current main 9f74f91.
• Old-to-new provenance closure:
  • Reused stages from e7b132d evidence: config, build, eval.
  • Rerun stage on patch-equivalent candidate e6d5aa4: perf.
  • Refreshed baseline perf evidence (rerun):
    • fp32 mean 85.493 ms, p50 88.467 ms, throughput 11.70/s, RSS delta +79.73 MB.
    • fp16 mean 119.342 ms, p50 110.072 ms, throughput 8.38/s, RSS delta +72.43 MB.
• Revision-3 candidate invalidation decision: PARTIAL-RERUN from e6d5aa4 to d5fa845.
  • Reused from e6d5aa4: L0 build, L1 perf, L2 parity, analyze.
  • Rerun on d5fa845: mandatory functional smoke eval, focused evaluator tests, touched-file ruff gates, affected eval/export partition.
• Final-head invalidation decision: PARTIAL-RERUN from d5fa845 to 63bfd4a.
  • Reused at final head after no-impact proof: L0 build, L1 perf, L2 parity, analyze.
  • Provenance chain: reused tuples come from d5fa845 immediate evidence, which itself carries forward the earlier e6d5aa4 validated build/perf/parity/analyze provenance.
  • Rerun on 63bfd4a: focused evaluator tests, affected eval/export partition, touched-file ruff gates, and mandatory real one-row COCO functional smoke (without and with trust_remote_code).

2. Goal

• Committed Effort: L2.
• Goal ceiling: L2.
• Intended Outcome: L2.
• Goal success definition: L0-L2 CPU fp32/fp16 remain PASS, and mandatory supplemental functional-smoke eval must process samples and emit semantically valid keypoint metric on final head.

3. Outcome

• Final outcome: shipped tier L2, highest goal verdict L2 PASS.
• Coverage: full required tuples (CPUExecutionProvider/cpu/fp32 and CPUExecutionProvider/cpu/fp16), no deferred tuples.
• Mandatory functional smoke status: PASS on final head 63bfd4a.
• Candidate and metadata state for this PR: head 63bfd4a, Draft=true, label model-scale-by-skill present.
• Quality gates on final-head rerun scope:
  • ruff check touched files: PASS.
  • ruff format --check touched files: PASS (3 already formatted).
  • pytest tests/unit/eval/test_keypoint_detection_evaluator.py: 14 passed.
  • pytest tests/unit/eval + tests/unit/export/test_io.py: 729 passed, 1 warning.
• GitHub checks on current head: all 9 completed and 9 successful (0 failing, 0 pending), independently verified.

4. Per-EP/device/precision results and Functional smoke Eval

Goal ladder

Tier	CPUExecutionProvider / cpu fp32	CPUExecutionProvider / cpu fp16	Verdict
L0 build	PASS	PASS	PASS
L1 perf	PASS	PASS	PASS
L2 parity vs pinned PyTorch	PASS	PASS	PASS

Perf

Tier	EP / Device	Precision	Verdict	Mean	p50	Throughput	RAM delta
L1	CPUExecutionProvider / cpu	fp32	PASS	89.305 ms	88.314 ms	11.2 samples/s	+71.21 MB
L1	CPUExecutionProvider / cpu	fp16	PASS	127.034 ms	123.224 ms	7.87 samples/s	+72.18 MB

Output parity

Tier	EP / Device	Precision	Verdict	Cosine	Max abs	Mean abs
L2	CPUExecutionProvider / cpu	fp32	PASS	0.9999999999952179	4.3958425521850586e-07	1.1096784824360384e-08
L2	CPUExecutionProvider / cpu	fp16	PASS	0.9999936832120094	0.00035926327109336853	1.3609533198177814e-05

Functional smoke Eval (mandatory supplemental evidence; operability only)

• Scope: FP32 CPU only, keypoint-detection, real one-row authoritative COCO subset, deterministic first-N, sample_limit=1.
• Contract checks: schema/label/prediction semantics verified before scoring.
• Final-head run A (without trust_remote_code): exit=0, processed_samples=1, map=0.8.
• Final-head run B (with trust_remote_code): exit=0, processed_samples=1, map=0.8.
• Classification: functional smoke only; not benchmark-quality or representative accuracy.
• Evaluator capability change that removed prior blocker: strict AutoImageProcessor remains first path; on known Unrecognized image processor failure and vitpose-family metadata gate, fallback retries with VitPoseImageProcessor while preserving trust_remote_code propagation.

5. Delta

• Recipe paths (shipped):
  • examples/recipes/nielsr_vitpose-base-simple/cpu/cpu/keypoint-detection_fp32_config.json
  • examples/recipes/nielsr_vitpose-base-simple/cpu/cpu/keypoint-detection_fp16_config.json
• Recipe delta vs baseline generated config remains inherited from prior accepted recipe work; revision 4 introduced no new recipe semantic changes.
• Recipe-free acceptance: PASS (revision 4 touched evaluator/test path only; no new recipe semantics introduced).
• Production recipe README: untouched.

Bug fix explanation

• Symptom and trigger: keypoint eval failed before row processing with ValueError: Unrecognized image processor in nielsr/vitpose-base-simple, reproducible with and without trust_remote_code.
• Root cause: metadata alias drift between preprocessor image_processor_type and installed AutoImageProcessor mapping/class symbol for vitpose caused strict resolver failure.
• Changed symbols and mechanism:
  • WinMLKeypointDetectionEvaluator.prepare_pipeline
  • WinMLKeypointDetectionEvaluator._load_image_processor
  • WinMLKeypointDetectionEvaluator._should_fallback_to_vitpose_image_processor
  • WinMLKeypointDetectionEvaluator._is_vitpose_family
  • Behavior: keep strict AutoImageProcessor first; on exact known resolver signature plus vitpose-family metadata gate, retry with VitPoseImageProcessor.
• Generalization rule: class-of-models metadata-driven fallback (model_type/architectures family), no model-id hardcoding.
• Compatibility and blast radius: non-vitpose successful path remains unchanged; broad exception swallowing not introduced; trust_remote_code forwarding preserved in both primary/fallback branches.
• Regression evidence: focused evaluator tests PASS (14), affected eval/export partition PASS (729 passed, 1 warning), touched-file ruff gates PASS.

Revision-4 follow-up fix (d5fa845 -> 63bfd4a)

• Symptom and trigger: evaluator processor loading accepted Optional model_id flow and allowed late failure paths when model_id was absent.
• Root cause: control-flow did not narrow config.model_id before from_pretrained calls, so Optional[str] could flow into processor loading.
• Changed symbols and mechanism:
  • WinMLKeypointDetectionEvaluator._load_image_processor
  • Added local binding for model_id, explicit ValueError guard when missing, and reused narrowed non-null local for both AutoImageProcessor and VitPoseImageProcessor from_pretrained calls.
• Generalization rule: invariant enforcement is data-driven and model-family-neutral (guard on required evaluator config), not checkpoint hardcoding.
• Compatibility and blast radius: preserves existing successful model_id-present behavior and only introduces earlier explicit failure for invalid missing-model_id inputs.
• Regression evidence: new test_missing_model_id_raises_before_processor_load added and passing; focused evaluator suite 14/14 PASS; affected partition 729 PASS; final functional smoke on 63bfd4a remains PASS in both trust modes.

6. Analyze summary - component level and op level

Static rule analysis status: ANALYZE-PARTIAL-SUCCESS. This section is compatibility analysis output, not runtime execution proof.

Component-level summary

Artifact	Architecture coverage	Mapping	Actionable EP findings
fp32	VitPose backbone/encoder/reshape/head regions represented via heuristic components	124 mapped heuristic components, 0 unmapped nodes	Grouped static-unknown outcomes for NvTensorRTRTX/OpenVINO/QNN; no CPU runtime-support claim from static rules
fp16	Same architecture regions, plus fp16 boundary behavior in graph form	126 mapped heuristic components, 0 unmapped nodes	Same grouped static-unknown/no-rule pattern

Op-level summary

Artifact	Graph	Dominant ops	EP roll-up
fp32	359 ops / 12 types	Reshape 122; Gemm 72; Transpose 50; Add 26; LayerNormalization 25	Static-unknown: NvTensorRTRTX/OpenVINO/QNN. No-rule-data: CUDA/DML/MIGraphX/TensorRT
fp16	361 ops / 13 types	Reshape 122; Gemm 72; Transpose 50; Add 26; LayerNormalization 25	Same EP roll-up pattern as fp32

7. Reproduce commands

$OUT='temp/model-support-repro'
winml build -c examples/recipes/nielsr_vitpose-base-simple/cpu/cpu/keypoint-detection_fp32_config.json -m nielsr/vitpose-base-simple -o $OUT/fp32 --precision fp32 --rebuild
winml build -c examples/recipes/nielsr_vitpose-base-simple/cpu/cpu/keypoint-detection_fp16_config.json -m nielsr/vitpose-base-simple -o $OUT/fp16 --precision fp16 --rebuild
winml perf -m $OUT/fp32/model.onnx --device cpu --skip-build --memory
winml perf -m $OUT/fp16/model.onnx --device cpu --skip-build --memory
winml eval -m $OUT/fp32/model.onnx --model-id nielsr/vitpose-base-simple --task keypoint-detection --device cpu --ep cpu

[ssss141414]

REQUEST_CHANGES

Reviewed GitHub PR #1299 at exact head 88be42b4f4f68e2e299ffc58509ed190c154511e from a fresh detached checkout. The PR remains OPEN/DRAFT with model-scale-by-skill. Original PR #1195 remains unchanged at 26b538f96a05881b62ca334517b2eb6d18e3b7fc, draft, updated 2026-07-24T02:33:40Z.

Required change

1. Current-main baseline provenance is stale. The frozen charter and public PR body cite baseline commit 79d7baad10f28cd79e65072f78494e8be758f3df, while independently fetched origin/main and GitHub's PR base tip are ac38b61e4501ffeb9573a1d37d1bf811e4a2c1d7. git rev-list --count 79d7baad10f28cd79e65072f78494e8be758f3df..origin/main is 1. This violates reviewer.md Outcome-L0 Baseline HEAD == current origin/main (_meta-052), which requires rebase and baseline rerun even when the intervening commit is non-overlapping.

   • planner: refresh/reissue the charter on ac38b61e4501ffeb9573a1d37d1bf811e4a2c1d7; rerun the Optimum probe, winml --version, exact git rev-parse HEAD, recipe-free baseline build, and same-model/task winml config comparison. Reassess Effort/Goal only from those refreshed results.
   • producer: rebase the candidate onto current origin/main; apply only changes required by the refreshed charter and keep examples/recipes/README.md untouched.
   • tester: use a new immutable evidence root and validate the new exact head, including fp32/fp16 L0-L2, focused offline behavior, models+commands partitions, lint/mypy, artifacts, named inputs, and CPU/DML analyze retries.
   • learner: re-audit and reseal findings against the refreshed baseline/new candidate; explicitly preserve or revise vitpose-017..019 rather than carrying the old candidate identity implicitly.
   • explainer: update the public Baseline section and all derived hashes/head references from refreshed source-owned evidence; keep the PR draft/labeled and re-enumerate checks/comments/reviews/threads after the push.

Verified in this pass

• Live identity: head exact; base tip ac38b61e...; draft and label exact; original PR recipe(vitpose-base-simple): add CPU keypoint detection configs #1195 unchanged.
• Diff: 7 paths, two recipes plus generalized export/perf/session code and tests; no README, skill, or agent change; no source model/checkpoint hardcoding.
• Engineering review: metadata-driven single-input vision recovery is guarded by vendor vision declarations and unambiguous shape/processor metadata; incompatible/non-vision paths retain fallback behavior.
• Independent focused check: repaired empty-cache/offline VitPose test passed, 1 passed in 0.81s.
• Exact-head GitHub gate: 9/9 attached checks COMPLETED/SUCCESS, including lint, models, commands, remaining, optim, analyze, CodeQL, and CLA.
• Fresh conversation inventory: 0 issue comments, 0 reviews, 0 inline comments, 0 review threads, 0 unresolved threads.
• Evidence: fp32/fp16 structures and sizes, realized fp16, perf/memory, L2 values, portable named-input hash, complete CPU/DML analyze JSON and exact provider-bridge errors all match the PR report.
• Local full-partition harness was not accepted because the VS Code terminal stopped launching long or trivial subprocesses after one queued pytest run; exact-head GitHub checks and sealed command outputs were directly verified instead. This is not the routed defect and does not weaken the baseline failure above.

No PR branch, PR body, evidence root, label, draft state, or original PR #1195 content was modified by this reviewer.

[ssss141414]

APPROVE (content-only reviewer verdict; not a GitHub Review state)

Reviewed PR: #1299
Reviewed head SHA: 63bfd4a
Base SHA: 9f74f91

1. Fresh-checkout and diff-scope verification

• Independent detached worktree created at exact head SHA 63bfd4a.
• merge-base(HEAD, origin/main) = 9f74f91.
• Diff scope (origin/main...HEAD) is 9 files:
  • examples/recipes/nielsr_vitpose-base-simple/cpu/cpu/keypoint-detection_fp16_config.json
  • examples/recipes/nielsr_vitpose-base-simple/cpu/cpu/keypoint-detection_fp32_config.json
  • src/winml/modelkit/commands/perf.py
  • src/winml/modelkit/eval/keypoint_detection_evaluator.py
  • src/winml/modelkit/export/io.py
  • src/winml/modelkit/session/session.py
  • tests/unit/commands/test_perf_cli.py
  • tests/unit/eval/test_keypoint_detection_evaluator.py
  • tests/unit/export/test_io.py
• Production README index untouched: examples/recipes/README.md not in diff.

2. Code and invariants review

• No model-id hardcoding introduced in touched source.
• Evaluator fallback remains narrow and generalized:
  • AutoImageProcessor first path retained.
  • Fallback only on known "Unrecognized image processor" ValueError and vitpose-family metadata gate.
  • trust_remote_code forwarded in both primary and fallback branches.
• model_id guard correctness verified: explicit early ValueError when model_id is missing before any processor loader call.
• Regression test presence verified, including missing-model-id and non-vitpose no-regression cases.

3. Planner/producer/tester/learner/explainer consistency and provenance

• Charter revision 3 confirms:
  • baseline main_commit = 9f74f91,
  • baseline refresh decision = PARTIAL-RERUN,
  • optimum probe verdict = WINML-ONLY for vitpose keypoint-detection.
• Producer deliverable revision 4 and tester verdict revision 4 are aligned on final head 63bfd4a and PARTIAL-RERUN invalidation logic.
• Learner revision 3 remains a d5fa845-stage artifact as declared by explainer inputs; rev4 body explicitly documents final-head delta and rerun evidence.
• PR body at live GitHub head 63bfd4a is updated, coherent, and labels functional smoke correctly as operability-only.

4. Independent execution evidence

• Mandatory final-SHA FP32 CPU functional smoke rerun executed independently (both modes):
  • without --trust-remote-code: processed num_images=1, map=0.8, exit 0
  • with --trust-remote-code: processed num_images=1, map=0.8, exit 0
• Artifact precision realization independently verified from binary artifacts:
  • fp32 model.onnx sha256 = 0040b80cfb62adb2a78892cc4a8fdea39d68e92d95bc989d031971e5eae1ed72
  • fp16 model.onnx sha256 = e3cf530ab2ed8aa738f72026dbb6bb776085a39fe31c4e5c19ad23e470ae3f43
  • model.onnx.data sizes: fp32=343649280, fp16=171824640
  • fp16 FLOAT16 initializers: 201
• Independent affected partition rerun (eval/export slice) PASS:
  • 729 passed, 1 warning.

5. Local gate notes and pre-existing baselines

• Local ruff check (src/ tests/) shows 5 existing RUF036 diagnostics; same 5 reproduced on clean base worktree at 9f74f91 (pre-existing, not introduced by this PR).
• Local full models partition shows one existing failure:
  • tests/unit/models/wav2vec2/test_onnx_config.py::TestWav2Vec2EmotionModelClassMapping::test_model_initialization_sets_transformers_bookkeeping
  • same failure reproduced on clean base 9f74f91 (pre-existing, not introduced by this PR).

6. GitHub final-sha gates and threads

• Live PR metadata at review time:
  • headRefOid = 63bfd4a
  • Draft = true
  • label model-scale-by-skill = present
• GraphQL reviewThreads enumeration on current head:
  • total threads = 0
  • unresolved threads = 0
• gh pr checks on exact head:
  • 9 successful, 0 failing, 0 pending, 0 cancelled, 0 skipped.

7. Coverage and verdict

• Required tuples: CPUExecutionProvider/cpu/fp32 and CPUExecutionProvider/cpu/fp16.
• Evidence status: full coverage; no deferred tuples.
• Mandatory final-SHA FP32 CPU smoke: PASS (processed=1, map=0.8), clearly labeled functional-only.

Final reviewer verdict: APPROVE for head 63bfd4a.