feat: synthetic input simulation layer (MCP + AltTester) - #9889
Draft
pravusjif wants to merge 13 commits into
Draft
feat: synthetic input simulation layer (MCP + AltTester)#9889pravusjif wants to merge 13 commits into
pravusjif wants to merge 13 commits into
Conversation
Stage 1 — pure refactor: the MCP-private input plumbing moves into a new DCL.SyntheticInput assembly (McpEcsRequest→EcsRequest, McpMovementOverride→ SyntheticMovementIntent, McpPointerEventIntent/System→Synthetic*), with a SyntheticInputPlugin registering the delivering systems and a SyntheticInputAgent facade both MCP tools and future AltTester probes call. walk/click_entity/look_at become thin front-ends over it. Stage 2 — new capabilities through the same production pipelines: - global SDK input events: PrepareGlobalInputEventsSystem appends synthetic button edges (deduped against real ones) and is pinned between the raycast and pointer-events processing - hover-only aim holds, aimless (global) button gestures, screen-point aims - camera look: held Cinemachine delta + look-at via CameraLookAtIntent - synthetic movement now respects scene InputModifier locks by default (ignoreInputModifiers escape hatch on the walk tool) - new MCP tools: hover_entity, press_input, click_at, camera_look Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… tools Layered UI simulation in DCL.SyntheticInput/UiSimulation: - semantic path (primary): UiDiscovery lists/resolves uGUI elements by path/id/AltId, SdkUiResolver addresses SDK scene UI by CRDT id, and UiInteractionSimulator synthesizes their events directly (ExecuteEvents for uGUI, UI Toolkit SendEvent for scene UI — two-frame SDK clicks so the single pointer-event slot never loses the press), all behind an occlusion pre-check so covered elements can't be clicked through - device path (fidelity): always-enabled DclAutomation mouse+keyboard reach both InputActionAsset graphs; UiVirtualDeviceGestureSystem replays move/click/drag/key gestures one state per frame, suppressing OS-cursor warps via the frame-stamped SyntheticCursorState while it steers - new MCP tools: ui_list, ui_click (semantic or device), ui_set_text (inputs + SDK dropdowns), ui_scroll, ui_drag - one UiAutomationServices session constructed in DynamicWorldContainer, owned by SyntheticInputPlugin, shared by the MCP front-end Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- WorldAutomationProbe / UiAutomationProbe (#if ALTTESTER, assembly DCL.SyntheticInput — the CallStaticMethod wire contract): tests drive the exact same SyntheticInputAgent/UiAutomationServices the MCP tools drive. Multi-frame gestures use start/poll via AltOperationRegistry (CallStaticMethod is synchronous on the main thread); nothing throws towards the test. - SyntheticInputPlugin installs the probes (static-latch pattern); the layer now also enables on --alttester alone in ALTTESTER builds (no MCP server). - link.xml preserves the reflection-only probe types against IL2CPP High stripping, backfilling the two existing AltTester probes. - New docs/synthetic-input-simulation.md (architecture + usage from both front-ends); mcp-automation.md tool catalog + implementation map updated; automation-testing.md probe table fixed (PerfSampler assembly is DCL.Plugins) and extended; mcp-scene-iteration skill + sdk-skills creator copy updated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
🚦 CI Status
Build skipped — no changes detected under No C# files changed — lint ratchet skipped. Waiting for tests to start… |
…notion of auto-improving skill, not needed for now
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue: #9393
Synthetic Input Simulation Layer
A driver-agnostic input simulation layer (
Explorer/Assets/DCL/SyntheticInput/, newDCL.SyntheticInputassembly) that lets automation — the embedded MCP server and AltTester tests — execute every input a human can, through the production input pipelines (collisions, occlusion, distance gates, scene input locks and CRDT write-back are all the real ones). Full architecture:docs/synthetic-input-simulation.md.What it covers
PetHoverEnter/PetHoverLeave)InputAction(IA_PRIMARY, IA_SECONDARY, IA_ACTION_3…) with the real-key fan-out — entity-bound when hovering a qualified entity, scene-root broadcast otherwiseStructure
McpEcsRequest→EcsRequest,McpMovementOverride→SyntheticMovementIntent,McpPointerEventSystem→SyntheticPointerEventSystem; its test suite migrated rename-only). MCP tools are now thin front-ends over theSyntheticInputAgent/UiAutomationServicesfacades.camera_look,click_at,hover_entity,press_input,ui_list,ui_click,ui_set_text,ui_scroll,ui_drag.WorldAutomationProbe,UiAutomationProbe,#if ALTTESTER, assemblyDCL.SyntheticInput): tests drive the exact same code path viaCallStaticMethod, with a start/poll API for multi-frame gestures. Preserved against IL2CPP stripping inlink.xml(the two pre-existing probes backfilled).--mcp/--mcp-port(any build) or--alttester(ALTTESTER builds); nothing is constructed in retail without flags.Behavior changes to review
InputModifierlocks by default (like WASD); thewalktool gains anignoreInputModifiersescape.PrepareGlobalInputEventsSystemnow accepts synthetic button edges (deduped against real same-frame presses) and is pinned[UpdateAfter(PlayerOriginatedRaycastSystem)]/[UpdateBefore(ProcessPointerEventsSystem)]— two previously-unordered systems; behavior-neutral for real input.ProcessPointerEventsSystemnow removes each action edge from theGlobalInputEventsbuffer the moment it lands entity-bound on a hovered, qualified entity — per action, same frame — andWritePointerEventResultsSystemalways writes the (pre-suppressed) buffer. Verified in-world with buffer instrumentation: an aimed press produces the entity result and zero root writes; an unaimed press broadcasts. NewProcessPointerEventsSystemShouldcovers the synthetic edge, a real key press, and the no-hover control. (Three live runs first misdiagnosed suppression as broken — the scene-side measurement was at fault:isTriggered(action, type, engine.RootEntity)never reads the root becauseRootEntityis0and the SDK'sif (entity)guard is falsy for it; an upstream js-sdk-toolchain fix is worth filing.)maxDistancetargets (HoverFeedbackUtils): the leave is gated on the ending hover's own qualification instead of being re-qualified against the ray of the frame the hover ended on — real users also never sawPET_HOVER_LEAVEon tight-range targets before this.