feat(vit-gpt2): add CPU image-captioning recipes

[ssss141414]

Summary

Rebuilds historical #934 for nlpconnect/vit-gpt2-image-captioning as a current-main, full-ladder contribution.

• Adds the composite encoder+decoder recipe pair for CPU FP32 and genuine FP16.
• Fixes a class-wide winml inspect --format json crash when nested Hugging Face configs occur in IOConfigInfo.extra.
• Adds an offline regression test for nested PretrainedConfig serialization.
• Does not modify examples/recipes/README.md.

Base and toolchain: origin/main@e8a1c21e8e4b38a2b6831eade91027a3ad4e4916, winml 0.2.0, Python 3.11.14, ORT 1.24.4, transformers 4.57.6.

1. Recipe paths and auto-config diff

• examples/recipes/nlpconnect_vit-gpt2-image-captioning/cpu/cpu/image-to-text_fp32_config_encoder.json
• examples/recipes/nlpconnect_vit-gpt2-image-captioning/cpu/cpu/image-to-text_fp32_config_decoder.json
• examples/recipes/nlpconnect_vit-gpt2-image-captioning/cpu/cpu/image-to-text_fp16_config_encoder.json
• examples/recipes/nlpconnect_vit-gpt2-image-captioning/cpu/cpu/image-to-text_fp16_config_decoder.json

winml config emitted the encoder/decoder export, optimizer, loader, and null quant/compile sections unchanged. The only recipe delta is an explicit, checkpoint-specific L3 dataset declaration:

• dataset: priyank-m/SROIE_2019_text_recognition
• split: test
• columns: image -> input, text -> label

Precision remains an explicit build argument (--precision fp32 / --precision fp16); emitted initializer dtypes were verified rather than inferred from filenames.

2. Precision and EP coverage

Host provider snapshot: ["AzureExecutionProvider", "CPUExecutionProvider"].

EP	Device	Precision	L0/L1/L3	Recipe
cpu	cpu	fp32	PASS / PASS / PASS	encoder+decoder
cpu	cpu	fp16	PASS / PASS / PASS	encoder+decoder
dml	gpu	fp32, fp16	HOST-BLOCKED	none
migraphx	gpu	fp32, fp16	HOST-BLOCKED	none
nvtensorrtrtx	gpu	fp32, fp16	HOST-BLOCKED	none
openvino	cpu	fp32, fp16	HOST-BLOCKED	none
openvino	gpu	fp32, fp16	HOST-BLOCKED	none
openvino	npu	fp32, fp16, w8a8, w8a16	HOST-BLOCKED	none
qnn	gpu	fp32, fp16	HOST-BLOCKED	none
qnn	npu	fp32, fp16, w8a8, w8a16	HOST-BLOCKED	none
vitisai	npu	fp32, fp16, w8a8, w8a16	HOST-BLOCKED	none

All 26 required tuples remain explicit. No accelerator recipe or runtime claim is fabricated.

3. Current-main baseline

Without any recipe, current main auto-expanded the model into both composite halves and built them on CPU FP32:

winml build -m nlpconnect/vit-gpt2-image-captioning -o temp/rerun934/baseline_current \
  --ep cpu --device cpu --precision fp32 --no-analyze --no-optimize \
  --no-quant --no-compile --rebuild

Result: PASS; encoder and decoder artifacts emitted. L0 is inherited from main, so validation climbed through L3.

Current-main winml inspect --format json did not pass: it crashed with TypeError: Object of type ViTConfig is not JSON serializable. The fix uses json.dumps(default=...), converts enums and paths, and invokes structured to_dict() objects without naming this model or architecture. Post-fix inspect emits valid JSON with composite.pipeline_tasks=["image-to-text"] and encoder/decoder components.

4. Artifact inventory

Artifact	Nodes	Initializer dtypes	External data
FP32 encoder	366	199 FLOAT, 6 INT64	343,194,624 B
FP32 decoder	803	254 FLOAT, 46 INT64, 1 BOOL	765,632,512 B
FP16 encoder	368	199 FLOAT16, 6 INT64	171,597,312 B
FP16 decoder	853	254 FLOAT16, 46 INT64, 1 BOOL	382,824,448 B

All four models pass onnx.checker.check_model; .onnx and .onnx.data are co-located. Encoder output encoder_hidden_states [1,197,768] matches the decoder cross-attention input.

5. Goal ladder

Tier	FP32	FP16	Evidence
L0 build/structure	PASS, both halves	PASS, both halves	checker, I/O, external data, dtype counts
L1 CPU perf	PASS, both halves	PASS, both halves	20 iterations + 3 warmups
L2 PyTorch parity	PASS, both halves and all decoder outputs	PASS, both halves and all decoder outputs	fixed-seed identical inputs
L3 task metric	PASS	PASS	20 real SROIE test images

No higher tier was skipped and the historical decoder-L2 deferral is removed.

6. Performance

Component	FP32 P50	FP16 P50	FP32 throughput	FP16 throughput
encoder	68.97 ms	81.18 ms	14.32/s	12.23/s
decoder	42.02 ms	47.49 ms	23.64/s	21.06/s

CPU FP16 is a storage/compatibility coverage claim, not a speedup claim.

7. Numerical parity

Component	Precision	Minimum cosine	Maximum absolute delta
encoder	fp32	0.999999999998	5.22e-6
decoder logits + 24 present-KV outputs	fp32	0.9999999999996	4.96e-5
encoder	fp16	0.999998591619	0.00720
decoder logits + 24 present-KV outputs	fp16	0.999999722505	0.02686

8. L3 evaluation

Both precisions used the same shuffled 20-sample SROIE test subset (seed 42):

Precision	CER	CIDEr	Samples
fp32	4.1606	0.0	20
fp16	4.1606	0.0	20

The metrics are reported as observed; the checkpoint is a natural-image captioner evaluated on receipt OCR, so this run proves end-to-end task execution and precision consistency rather than dataset suitability.

9. Static all-EP analysis

FP32 encoder: 366 operators / 11 unique. NvTensorRTRTX, QNN GPU/NPU, and OpenVINO CPU/GPU/NPU classify all 366 supported. CPU, DML, MIGraphX, CUDA, and VitisAI have no matching rule data and remain unknown.

FP32 decoder: 803 operators / 22 unique. NvTensorRTRTX and OpenVINO classify 778 supported / 25 unknown; QNN NPU classifies 777 supported / 4 partial / 25 unknown; QNN GPU classifies 764 supported / 16 partial / 1 unsupported / 25 unknown. Rule analysis is static and is not presented as runtime validation.

10. Optimum and fix classification

Optimum covers the user-facing vision-encoder-decoder image-to-text path. WinML already provides split encoder and static-KV decoder overrides. The recipes are per-checkpoint deployment declarations, while the inspect failure is a class-wide shared-infrastructure bug: any inspect result containing a nested object with to_dict() could fail JSON output. Therefore the contribution correctly includes a generalized code fix and a no-model-hardcoding regression test.

Claimed tiers: Effort L2 / Goal L3 / Outcome L2.

11. Tests

pytest tests/unit/inspect/test_composite_rendering.py tests/unit/commands/test_inspect_cli.py -q
33 passed in 25.34s

ruff check src/winml/modelkit/inspect/formatter.py tests/unit/inspect/test_composite_rendering.py
All checks passed!

All four recipe files also pass python -m json.tool; git diff --check passes.

12. Methodology and historical replacement

This replaces historical #934 with current-main evidence. The old PR edited the production README, covered only FP32, deferred decoder L2, and stopped L3 at the missing-default-dataset error. This rebuild leaves the README untouched, validates genuine FP16, supplies an explicit real dataset, closes both decoder parity and L3, and fixes the generalized inspect failure discovered by current-main diagnostics.

Independent pre-PR review verdict: APPROVE, coverage partial only because the host exposes CPU runtime; all blocked accelerator precision tuples remain enumerated above.

[ssss141414]

Independent pre-PR review: APPROVE. Scope is coherent: the nested-config JSON fix is generic and regression-tested; all four CPU recipes parse and retain the auto-config export/optimizer/loader contract; genuine FP16 is evidenced by initializer dtypes; encoder and decoder pass L0-L3 in both CPU precisions; and examples/recipes/README.md is untouched. Coverage annotation: partial because this host exposes CPU runtime only. All 26 required EP/device/precision tuples are nevertheless enumerated, and static analysis is not represented as accelerator runtime validation.

[xieofxie]

Quantized variants are still not pinned to their authored precision here. Because this change also passes --device npu, winml build treats the device as an explicit override; with no --precision, its auto policy resolves to w8a16 and rewrites an authored W8A8 config (for example usyd-community_vitpose-base-simple) to uint16 activations. The result remains labeled W8A8 even though a W8A16 artifact was built. Please pass variant.precision for quantized variants too, or avoid applying a device policy override to authored quantization.

[xieofxie]

Selecting a target directory here discards every flat variant, even when the target directory only overrides one precision. Existing mixed-layout models such as facebook_dinov2-base have flat FP16/W8A16 recipes plus a target-specific qnn/npu W8A16 recipe; a QNN/NPU run now returns only W8A16 and silently loses the valid FP16 job. Please apply target precedence per precision (without mixing components within a precision) and retain flat precisions that have no target-specific replacement.

[xieofxie]

An EP-only invocation cannot discover nested recipes: with the default device="auto", this leaves target_dir unset, while the auto-target branch below is disabled whenever ep is provided. For example, run_eval.py --ep cpu now misses the cpu/cpu recipes added here and falls back to generated config, even though CPU has an unambiguous device. Please infer a unique device from the normalized EP (or select a sole matching device directory) before resolving the target path.