feat(iris): deliver QR/face metadata objects to AVCaptureMetadataOutput

[meatpaste]

The gap

Apps that scan codes rarely touch pixels themselves — they attach an AVCaptureMetadataOutput and wait for AVMetadataMachineReadableCodeObjects. Nothing in the injected session produces those, since there's no real capture connection behind it. The practical result today: point sim cam at a QR code and the app renders the injected feed perfectly, then its scanner never fires. No error, no timeout, just a dead-end.

This runs detection over the frame the injector has already delivered and calls the app's metadata delegate directly.

sim cam start --image qr.png     # the app's QR scanner now fires with the decoded payload

What's hooked

setMetadataObjectsDelegate:queue:, availableMetadataObjectTypes, and the metadataObjectTypes pair. setMetadataObjectTypes: is recorded rather than forwarded — the real setter validates against its own (empty) available list and raises NSInvalidArgumentException.

Three things worth knowing

Vision doesn't work in the Simulator. Every VNImageRequestHandler fails with "Could not create inference context" — barcodes and faces alike. Detection therefore uses CIDetector, which covers QR codes and faces only. availableMetadataObjectTypes advertises exactly those two rather than the full symbology set, because apps intersect their requested types against that list; claiming EAN or PDF417 would just make them wait on callbacks that can never arrive.

AVMetadataObject's -dealloc segfaults on an instance built with class_createInstance — reproducibly, even on one with nothing assigned. The synthesised objects are therefore pooled: a fixed ring allocated once and recycled, never released, so that dealloc is never reached and memory stays bounded. An object is valid for the callback it arrives in, which is the lifetime the real API promises anyway.

Detection is throttled to 200ms (IRIS_RECOGNITION_INTERVAL) and single-flight on its own serial queue — frames arrive at up to 120fps and scanning every one costs milliseconds for nothing.

Also fixes a crash unrelated to this feature

-[AVCaptureDeviceInput portsWithMediaType:sourceDeviceType:sourceDevicePosition:] fell through to AVFoundation's implementation on the synthesised input, which walks state class_createInstance never set up and segfaults inside -[AVCaptureDeviceInput multiCamPorts]. Any app calling input.ports(for:) hits this — it's how I found it. MSCFakeCaptureInput now answers typed port lookups itself, returning the video port for video requests and an empty array otherwise (the documented "no such port" answer, which callers already handle).

Verification

End to end against a real Flutter app — the multicamera plugin driving an iPad Pro simulator on Xcode 26.6 / macOS 26.5:

[IrisInject] Captured metadata delegate=<multicamera.CameraHandle: 0x1063f46c0> …
[IrisInject] Metadata types requested: 1 (org.iso.QRCode)
[IrisInject] Recognised 1 metadata object(s)
→ app's onBarcodesScanned callback: [AIRLOCK-KIOSK-TEST-42]

One caveat on that app: like the libraries in patches/, it hard-codes #if targetEnvironment(simulator) and returns before touching AVFoundation, so it needed the same one-line guard relaxation to see the injected camera at all. Worth a patches/ entry if that's useful — happy to add one.

Docs updated: a new Metadata objects section in docs/SIM_CAM_ARCHITECTURE.md and a short note in the README.

[annurdien]

Code Review

Author: meatpaste · Diff: +460 / −0 across 7 files · Single commit

---

Overall Verdict

Strong PR. Well-researched, well-documented, solves a real gap in the injection layer. The code is careful around AVFoundation's sharp edges (dealloc segfaults, missing ports, Vision unavailability) and the workarounds are sound. The architecture doc and commit message are unusually thorough — every non-obvious decision is explained with root causes.

A few issues worth discussing before merge — nothing blocking, but some have correctness implications under concurrency.

---

Issues

🔴 High — Race on pooled metadata objects

FakeCaptureObjects.mm — MSCPooledMetadataObject

The pool uses a ring buffer of size 16. An object is mutated (ivars overwritten) in objectWithType: on the recognition queue, then read by the delegate on delegateQueue. With a pool size of 16 and recognition throttled to 200ms, the window is narrow — but the problem is structural, not probabilistic:

1. MSCPooledMetadataObject returns an object from the pool (under lock).
2. The caller writes to its ivars after the lock is released.
3. Meanwhile the delegate from a previous detection may still be reading the same object's ivars if the delegate callback was slow.

The 200ms throttle makes collision unlikely in practice, but if delegateQueue is congested (e.g. main queue under load), a callback from frame N could overlap with ivar mutation for frame N+16. The pool size being 16 gives ~3.2 seconds of runway at 200ms cadence, which is generous — but the invariant is fragile.

Suggestion: Either:

• Double the pool to 32 (buys 6.4s of delegate lag — effectively "never happens" territory), or
• Copy the object ivars under the pool lock (move mutation inside MSCPooledMetadataObject by passing the values), or
• Add a lightweight flag/generation counter so the delegate can detect staleness.

🟡 Medium — recognitionInFlight flag spans two lock acquisitions (TOCTOU)

CaptureHooks.mm — runRecognition

// First lock region: read inFlight
os_unfair_lock_lock(&gState.lock);
bool inFlight = gState.recognitionInFlight;
// ...
os_unfair_lock_unlock(&gState.lock);

// Gap — another thread could read inFlight=false here too

// Second lock region: set inFlight = true
os_unfair_lock_lock(&gState.lock);
gState.recognitionInFlight = true;
os_unfair_lock_unlock(&gState.lock);

If deliverFrame fires twice in quick succession (e.g. timer coalescing), both could read inFlight = false before either sets it to true, launching two concurrent detection passes. Mitigated by the serial recognitionQueue (work is serialised even if dispatched twice), but "single-flight" isn't strictly enforced — two blocks can be enqueued.

Suggestion: Merge the two lock regions into one:

os_unfair_lock_lock(&gState.lock);
// ... read all state ...
bool shouldRun = !inFlight && (now - last >= IRIS_RECOGNITION_INTERVAL);
if (shouldRun) {
    gState.recognitionInFlight = true;
    gState.lastRecognition = now;
}
os_unfair_lock_unlock(&gState.lock);
if (!shouldRun) return;

🟡 Medium — Superfluous __bridge roundtrip

FakeCaptureObjects.mm — pool allocation:

id instance = (__bridge id)(__bridge void *)class_createInstance(cls, 0);

class_createInstance already returns id. The double-bridge (__bridge id)(__bridge void *) is a no-op — harmless but confusing. Likely survived from an earlier version that used __bridge_transfer.

Suggestion: Simplify to id instance = class_createInstance(cls, 0);

🟡 Medium — Missing comment on iris_availableMetadataObjectTypes

iris_setMetadataObjectsDelegate:queue: chains to the original (correct). iris_setMetadataObjectTypes: deliberately does NOT chain (correct, with a clear comment). But iris_availableMetadataObjectTypes also doesn't chain and has no comment explaining why. The asymmetry looks like a bug to a reader who doesn't have the PR description in front of them.

🟢 Low — Face detection faceIDs are sequential per-frame, not stable across frames

NSInteger faceID = 1;
for (CIFeature *feature in [detector featuresInImage:image]) {
    MSCFakeFaceObject *object = [MSCFakeFaceObject objectWithFaceID:faceID++ ...];

Real AVMetadataFaceObject.faceID is a tracking ID that persists across frames. Here, face 1 in frame N and face 1 in frame N+1 may not be the same person. Fine for v1 (CIDetector doesn't provide tracking IDs), but worth noting as a known limitation in the arch docs.

🟢 Low — IRIS_RECOGNITION_INTERVAL comment says "Vision"

IrisConstants.h:

// Vision runs over the injected frames to synthesise AVCaptureMetadataOutput

The implementation uses CIDetector (Core Image), not Vision. The PR description itself explains why Vision doesn't work — the comment should match.

🟢 Low — kCMTimeZero is deprecated

Both fake objects return kCMTimeZero from -duration. Deprecated since iOS 12. Use CMTimeMake(0, 1). No functional impact.

---

Things Done Well

Area	Detail
Crash workarounds	The -dealloc segfault workaround (pooling) and the portsWithMediaType: fix are both well-researched. The pool approach is clean — bounded memory, no lifecycle surprises.
Throttling	200ms cadence + single-flight + serial queue is the right design. Not over-engineered.
Swizzle hygiene	setMetadataObjectsDelegate:queue: correctly chains to the original. setMetadataObjectTypes: correctly doesn't (with a clear comment). The getter returns recorded state, not the real value — correct.
Documentation	The architecture doc addition is excellent — the mermaid diagram, the three constraints section, and the note about setMetadataObjectTypes: being intercepted. Commit message is thorough.
Bonus crash fix	The portsWithMediaType: fix is the right scope for this PR — it's closely related (same fake-object subsystem) and would block testing this feature.
Defensive coding	Nil checks on delegate/queue/output, respondsToSelector: before calling, fallback to dispatch_get_main_queue() when queue is nil.

---

Recommendation

Approve with requests:

1. Merge the two lock regions in runRecognition to actually enforce single-flight (🟡).
2. Fix the "Vision" comment in IrisConstants.h (🟢 — trivial).
3. Consider the pool race (🔴) — either increase pool size or restructure mutation. The current window is generous but the invariant is brittle.

The rest (bridge no-op, kCMTimeZero deprecation, faceID instability) can be follow-up work.

[meatpaste]

Thanks — genuinely useful review, and you were right on every point. All seven addressed in 78ce039.

🔴 Pool race

You called this structural, and it is, so I've fixed the structure rather than widened the window. Slots are now checked out and back in instead of handed round a ring:

• MSCAcquirePooledMetadataObject marks a slot in use under the lock.
• MSCReleasePooledMetadataObjects returns it, called from the delegate block after the callback returns.

So a slot is never handed out again while anything could still be reading it — that's the invariant, not a probability bound. It also happens to be exactly the lifetime the real API promises ("valid for the duration of the callback"), so the semantics line up rather than being an approximation of them.

I skipped the three suggestions deliberately, and it's worth saying why: doubling the pool only buys more runway, and copying ivars under the pool lock doesn't actually help — the delegate doesn't take that lock, so the write and the read still aren't ordered against each other. The generation counter would work but puts the burden on the reader.

Under pressure the pool grows to a cap (16 → 64) and then logs and drops rather than failing quietly, so a delegate that stops returning costs detections instead of memory or correctness.

🟡 recognitionInFlight TOCTOU

Fixed as suggested — every precondition and the claim now share one lock region. I folded the requested-types check in there too, so there's no claim-then-unclaim path.

🟡 __bridge roundtrip

Correct, a no-op, and you're right that it's a leftover from the __bridge_transfer version. Now just class_createInstance(cls, 0), with the comment saying the +1 is deliberately abandoned rather than implying the cast is doing something.

🟡 Missing comment on iris_availableMetadataObjectTypes

Fair — the asymmetry did look accidental. Commented: the real getter reports what the absent hardware supports, i.e. nothing, so an app intersecting against it never attaches a scanner at all.

🟢 faceID, 🟢 "Vision" comment, 🟢 kCMTimeZero

All done rather than deferred — they were one-liners. faceID's non-stability is noted at the accessor and as a Known limitation in the arch doc; the constants comment now says Core Image; kCMTimeZero → CMTimeMake(0, 1).

Re-verified

Against the same real Flutter app (multicamera plugin), live QR feed:

Recognitions:      138
Pool exhausted:    0
Metadata delegate: 4
Types requested:   1 (org.iso.QRCode)

138 detections with zero exhaustion is the meaningful number — the cap is 64, so if release weren't happening it would have run dry after ~13s at the 200ms cadence. The app's barcode callback fired throughout and its backend lookup answered 200, so the objects survive the delegate handoff intact.

[annurdien]

@meatpaste , could you attach your manual test? A screen recording would be appreciated.

[meatpaste]

Sure, i've got a Claude skill to record demo videos 🤣

What it shows, filmed against the actual PR branch (feat/metadata-output-barcode-scanning, built via Iris/Scripts/build.sh):

1. The multicamera example app on an iPhone 17 Pro simulator (iOS 26.5), no camera views yet.
2. FrameHost --image qr.png injecting a QR encoding AIRLOCK-KIOSK-TEST-42.
3. Tapping + starts a capture session — the injected frame renders in the preview (which already worked).
4. Barcodes: [AIRLOCK-KIOSK-TEST-42] appears underneath — the app's AVCaptureMetadataOutput delegate firing, which is the thing the PR adds.

The narration explicitly separates "the feed rendered" from "the delegate fired", since that's the distinction a reviewer needs to see.

iris-qr-metadata-demo.mp4

[annurdien]

LGTM! thank you @meatpaste