Add ArmIKLayer, AudioVisemeDriver, and VMC Protocol support for device-driven avatars

[arkavo-com]

Summary

Library work for the Mac "no-device avatar" path tracked in arkavo-org/Creator#40 (VMagicMirror parity). Three additive subsystems, no renderer or shader changes.

Closes #433. Refs arkavo-org/Creator#31, arkavo-org/Creator#33, arkavo-org/Creator#36, arkavo-org/Spatial#89.

ArmIKLayer (Animation/Layers/ArmIKLayer.swift)

• AnimationLayer at priority 7 (above ArmCounterbalanceLayer) that solves shoulder and elbow rotations toward world-space HandTargets with TwoBoneIKSolver, plus per-finger FingerCurl.
• Never reads the live arm pose: the chain is rebuilt from rest transforms captured in initialize(with:) under the parent's current world matrix, so deltas are base-pose relative and a constant target gives a constant pose across frames. This matters because the compositor composes base * delta without resetting bones between frames.
• Releasing a target or curl writes one explicit identity delta so the arm returns to rest instead of freezing.
• Finger curl is a model-space axis rotation converted into each phalanx's rest frame (inverse(W) * delta * W), so it holds on rigs whose finger bones are not hand-aligned.

AudioVisemeAnalyzer / AudioVisemeDriver (Audio/)

• PCM [Float] in, VRM viseme weights (aa ih ou ee oh) out. Formant heuristic: energy-weighted spectral centroids in two bands scored against canonical F1/F2 pairs in log-frequency space, with noise gate and loudness curve. Direct DFT on the bins of interest, no Accelerate or AVFoundation.
• Driver keeps a rolling window fed from the audio thread (push), smooths per viseme through FilterManager, and writes the controller like ARKitFaceDriver (apply(to:) / update(samples:controller:)).

VMC Protocol (VMC/)

• OSCPacket / OSCMessage / OSCBundle: OSC 1.0 decode and encode, nested bundles, all standard type tags.
• VMCDriver: /VMC/Ext/Bone/Pos, /Root/Pos, /Blend/Val + /Blend/Apply (commit semantics), /OK, /T. Rotations are composed rest-relative (rest * inverse(restWorld) * delta * restWorld). Unity to model space is a reflection of X for every loaded model because the loader already rotates VRM 0.x to +Z-facing; flipZ is kept for raw 0.x data. Maps Unity HumanBodyBones names including the thumb chain shift (Unity Proximal/Intermediate/Distal → VRM Metacarpal/Proximal/Distal) and VRM 0.x blend aliases (Joy, A, Blink_L, ...).
• VMCEncoder: inverse mapping for sending; a frame built from a model round-trips through VMCDriver to the same pose.
• VMCReceiver: thin UDP NWListener, port 0 supported for tests.

Docs

• New DocC article DeviceDrivenAvatars linked from the catalog; README bullets.

Test plan

• ArmIKLayerTests (15): hand-on-target for both arms within 5 mm, stability over consecutive frames and after target changes, unreachable clamping, weight blending, elbow hint plane, finger curl by position for left/right/thumb/fist, release restores rest, priority above counterbalance.
• AudioVisemeTests (12): each prototype vowel classifies to its viseme from synthetic two-tone input, silence and gate, formant tracking, volume saturation, driver write/decay/gain/small-buffer accumulation.
• OSCPacketTests (10): hand-built byte layouts for a ,sf message and a #bundle, round trips for all types, error paths.
• VMCDriverTests (16): name mappings both directions, coordinate flips, Unity yaw 90° on the left upper arm lands the hand 0.5 m forward, rest-relative correctness on a twisted rest bone, blend commit-on-Apply, staleness, encoder round trip, UDP loopback on an ephemeral port.
• Full suite swift test --parallel --num-workers 14 -j 16 --disable-sandbox
• No .metal changes, so no metallib rebuild or AvatarSample_U.png regeneration.

Design notes for reviewers

• Priority 7 with .blend(weight) intentionally replaces the counterbalance brace's additive arm deltas while a target is set.
• The viseme classifier is a heuristic, not MFCC plus calibration. Its job is to keep the mouth alive when no tracker is connected; Creator should prefer a tracked mouth when one exists.
• ADR-0047 in Spatial rejected VMC on the phone; this PR is the Creator-side bridge that ADR anticipated.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FgZDdzrNEtPmbMupyyUTXj

[github-actions]

✅ Claude Code Review ran on a4aa4ef — result: success.
Review completed. Tool permission denials: 0. Turns: 9. Duration: 48s.

The completed review reported no tool permission denials.
Findings are posted as review comments. Run log.

[gitguardian]

️✅ There are no secrets present in this pull request anymore.

If these secrets were true positive and are still valid, we highly recommend you to revoke them.
While these secrets were previously flagged, we no longer have a reference to the
specific commits where they were detected. Once a secret has been leaked into a git
repository, you should consider it compromised, even if it was deleted immediately.
Find here more information about risks.

---

^{_{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.}}

[gitar-bot]

Code Review ✅ Approved 3 resolved / 3 findings

Comprehensive device-driven avatar support with ArmIKLayer for IK-solved arm poses, AudioVisemeDriver for mic-driven lip-sync, and VMC Protocol implementation (OSC codec, driver, encoder, receiver). VMCReceiver idle connection leak and stale ready signal consumption were fixed; onError timing issue was dismissed. No open issues remain.

✅ 3 resolved

✅ Quality: VMCReceiver never releases idle UDP connections

📄 Sources/VRMMetalKit/VMC/VMCReceiver.swift:128-139 📄 Sources/VRMMetalKit/VMC/VMCReceiver.swift:164-168
accept appends every incoming NWConnection to connections, but entries are only removed from remove(), which fires solely on .failed/.cancelled connection states. A healthy UDP sender's connection stays .ready indefinitely, so a long-lived receiver that sees senders on changing source ports accumulates connection objects that are never torn down until stop(). For the typical single-sender VMC case this is one connection, so impact is low; consider capping or aging out connections if multiple/rotating senders are expected.

✅ Edge Case: VMCReceiver.start()/waitUntilReady can consume a stale ready signal

📄 Sources/VRMMetalKit/VMC/VMCReceiver.swift:56 📄 Sources/VRMMetalKit/VMC/VMCReceiver.swift:108-113 📄 Sources/VRMMetalKit/VMC/VMCReceiver.swift:115-126
stop() resets readySignalled=false and boundPort=nil but does not drain the ready DispatchSemaphore. If start() signalled ready on a prior session and waitUntilReady was never called (or timed out), the leftover semaphore count causes the next waitUntilReady to return immediately, potentially observing boundPort == nil (false) before the new listener is actually ready, or a stale value. Re-create the semaphore in start() (or drain it in stop()) so readiness signaling is scoped per listener lifecycle.

✅ Edge Case: onError fires before listener identity guard on .failed

📄 Sources/VRMMetalKit/VMC/VMCReceiver.swift:107-114
In the listener .failed handler, self.onError?(...) is called before the guard self.listener === listener check, whereas the same guard correctly protects the boundPort/readySignalled mutations. A stale listener from a previous start/stop cycle (or the old listener during a restart) that transitions to .failed will emit a spurious error to the caller's onError callback even though it is no longer the active listener. Move the onError call after the identity guard so only the current listener reports failures.

_{Was this helpful? React with 👍 / 👎 | Gitar}