Vivid Mesh Shader Unity Plugin

Experimental Windows x64 native plugins for compiling and dispatching D3D12 mesh shaders from VividRP without passing the shader program through Unity ShaderLab.

The project builds two independent DLLs:

DLL	Purpose	Loaded by
VividMeshShaderCompiler.dll	Compiles HLSL stages to DXIL with DXC and resolves Unity-style include paths.	Unity Editor only
VividMeshShader.dll	Creates D3D12 root signatures and mesh-shader PSOs, then records DispatchMesh into Unity's active command list.	Editor and Windows player

The implementation is intentionally narrow and mostly stateless at the API boundary. Unity owns the render graph and frame buffers; the runtime plugin owns only immutable shader objects, queued dispatch snapshots, and the references needed to keep submitted GPU resources alive.

Requirements

• Windows x64
• CMake 3.20 or newer
• A C++20-capable MSVC toolchain (Visual Studio 2022 is recommended)
• Windows SDK with D3D12 headers and libraries
• Microsoft DirectX Shader Compiler native package containing:
  • dxcapi.h
  • dxcompiler.lib
  • dxcompiler.dll
  • dxil.dll
• A D3D12 device supporting Shader Model 6.5 and the Mesh Shader feature tier

The runtime plugin currently requires Unity's IUnityGraphicsD3D12v8 interface.

Build

Point VMS_DXC_ROOT at the root of an extracted Microsoft.Direct3D.DXC package. The root may use either the NuGet build/native/... layout or direct include, lib/x64, and bin/x64 directories.

cmake -S . -B build `
  -G "Visual Studio 17 2022" `
  -A x64 `
  -DVMS_DXC_ROOT="C:\path\to\Microsoft.Direct3D.DXC"

cmake --build build --config Release

Build outputs are written to build/bin:

build/bin/
  VividMeshShader.dll
  VividMeshShaderCompiler.dll
  dxcompiler.dll
  dxil.dll

To produce a redistributable directory, including the DXC license files when they are present in the package:

cmake --install build --config Release --prefix build/install

VividRP deployment

Copy the runtime DLL to:

Packages/com.vivid.render-pipelines/
  Runtime/SubSystem/Plugin/MeshShader/Plugins/x86_64/
    VividMeshShader.dll

Copy the compiler and its DXC runtime dependencies to:

Packages/com.vivid.render-pipelines/
  Editor/SubSystem/Plugin/MeshShader/Plugins/x86_64/
    VividMeshShaderCompiler.dll
    dxcompiler.dll
    dxil.dll

Keep the DXC license files next to the redistributed compiler binaries. Unity plugin importer metadata is maintained by VividRP and is not generated by this repository.

Shader authoring workflow

Mesh-shader source remains a normal .hlsl file, so it can include VividRP and CoreRP shader definitions directly. A small .vms JSON manifest selects the source, entry points, profiles, and compile options:

{
  "source": "Packages/com.vivid.render-pipelines/Shaders/Core/Private/GPUDriven/VisibilityBufferMeshShader.hlsl",
  "amplificationEntry": "AmplificationMain",
  "amplificationProfile": "as_6_5",
  "meshEntry": "MeshMain",
  "meshProfile": "ms_6_5",
  "pixelEntry": "PixelMain",
  "pixelProfile": "ps_6_5",
  "rootLayoutVersion": 1,
  "debug": false,
  "disableOptimizations": false
}

VividRP's scripted importer performs the following steps:

1. Reads the referenced HLSL text.
2. Maps Unity package prefixes to physical package directories.
3. Tracks transitive #include and #include_with_pragmas dependencies.
4. Calls VividMeshShaderCompiler.dll once for each AS, MS, and PS stage.
5. Stores the immutable DXIL blobs in a VividMeshShaderProgramAsset.
6. Passes those blobs and the render state to VividMeshShader.dll at runtime.

The compiler uses HLSL 2021, strict language checking, and -O3 by default. The manifest options add embedded debug information (-Zi -Qembed_debug) or disable optimization (-Od). Compilation diagnostics are returned to the Unity asset importer.

Runtime flow

VividMeshShaderProgramAsset
  -> VMS_CreateShaderObjectFromDxil
  -> D3D12 root signature + mesh-shader PSO
  -> VMS_CreateDispatchBatchRequest
  -> CommandBuffer.IssuePluginEventAndData
  -> plugin render callback
  -> ID3D12GraphicsCommandList6::DispatchMesh

VMS_CreateDispatchBatchRequest snapshots the shader handles, root constants, and six D3D12 buffer references. Event data is a monotonically increasing token, not a pointer to the request allocation. The callback atomically takes ownership of the registered request, which makes late callbacks safe after cancellation.

Adjacent renderer-list dispatches can be combined in one request. The current batch limit is 64 dispatches.

Root signature contract

The runtime plugin creates a fixed root signature. HLSL used with the plugin must match this layout:

Root parameter	HLSL binding	Contents
0	b0	20 32-bit constants: 4x4 view-projection matrix, renderer-list index, maximum request count, and two padding values
1	t0	Visible meshlet render requests
2	t1	Indirect arguments
3	t2	Instance data
4	t3	Meshlet data
5	t4	Shared vertex data
6	t5	Shared index data

The amplification shader reads the indirect argument buffer, clamps the request count, and emits the mesh dispatch dimensions. The native call itself records DispatchMesh(1, 1, 1) for each batch entry.

The plugin asks Unity to transition all six buffers to shader-readable states before dispatch. It restores the indirect argument buffer to D3D12_RESOURCE_STATE_INDIRECT_ARGUMENT afterward and reports all final states back to Unity.

Render-event behavior

The dispatch event accesses Unity's active D3D12 command list. It is configured with:

• SyncWorkerThreads, because split-job workers must stop before native code replaces the graphics PSO and root bindings.
• ModifiesCommandBuffersState, so Unity invalidates its cached command-list bindings after the callback.

A following state-boundary event is a no-op state marker. It does not request FlushCommandBuffers and does not perform another SyncWorkerThreads wait. Consequently it does not explicitly submit the active command buffers or add a second worker-thread synchronization point.

Resource and device lifetime

• Native pointers obtained from GraphicsBuffer.GetNativeBufferPtr() are converted to ID3D12Resource references when a request is created.
• Do not assume that a cached native pointer remains valid after SetData or SetBufferData; Unity may rotate the underlying D3D12 allocation. VividRP resolves these resource pointers for each queued batch.
• A successfully recorded batch retains its shader objects and D3D12 resources until Unity's frame fence reaches the captured value.
• If Unity does not expose a usable frame fence, submitted resources are retained conservatively until device reset or plugin shutdown.
• Pending requests are owned by a native token registry. Callback execution and explicit request destruction use single-owner removal, preventing double free.
• Requests whose events never execute are canceled when their shader object is destroyed, when the D3D12 device resets or shuts down, or when the plugin unloads. A late callback for a canceled token is a no-op.
• Shader objects and requests carry a device generation. Work created for an old D3D12 device is rejected after reset instead of using stale COM objects.

Native APIs

The authoritative ABI declarations are in src/Plugin.h and src/Compiler.h.

Runtime ABI: VMS_ABI_VERSION == 2

• Support and diagnostics: VMS_GetSupportStatus, VMS_GetDispatchFailureCount, and VMS_GetLastError.
• Shader objects: VMS_CreateShaderObjectFromDxil and VMS_DestroyShaderObject.
• Dispatch requests: VMS_CreateDispatchBatchRequest, VMS_DestroyDispatchRequest, and the render-event accessors.
• VMS_CreateShaderObject remains exported for ABI compatibility but runtime HLSL compilation is disabled; it returns zero with an explanatory error.

Compiler ABI: VMSC_ABI_VERSION == 1

• VMSC_Compile returns an owned result handle even for normal compilation failures, allowing the caller to retrieve DXC diagnostics.
• Read the result through VMSC_GetResultSuccess, VMSC_GetResultData, VMSC_GetResultSize, and VMSC_GetResultDiagnostics.
• Release every non-zero result handle with VMSC_DestroyResult.

Both ABIs are x64-only and validate structure sizes and versions before using caller data.

Current limitations

• Experimental VividRP integration; not a general-purpose Unity mesh-shader API.
• Windows x64 and D3D12 only.
• Requires amplification, mesh, and pixel DXIL stages.
• Fixed root signature and resource ordering.
• Maximum four render targets and 64 dispatches per native batch.
• No Unity ShaderLab variant or keyword processing.
• Runtime HLSL compilation is intentionally disabled.
• The current VividRP VisibilityBuffer path temporarily disables occlusion culling while mesh shaders are active, and alpha-tested buckets use the legacy draw path.

Diagnostics and validation

When integration fails:

1. Check VMS_GetSupportStatus() before creating shader objects.
2. Compare the managed and native ABI versions.
3. Read VMS_GetLastError() after a zero handle or dispatch failure.
4. Check DXC diagnostics emitted by the .vms asset importer.
5. In RenderDoc, verify that the expected DispatchMesh calls and root bindings appear in the VisibilityBuffer pass.
6. Stress device and managed lifetimes by repeatedly entering and exiting Unity Play Mode.

After changing native code, rebuild the Release target, copy the appropriate DLL to VividRP, and verify that the source and destination SHA-256 hashes match.

Source layout

src/Plugin.cpp    Runtime Unity/D3D12 plugin
src/Plugin.h      Runtime ABI
src/Compiler.cpp  DXC compiler bridge and include resolver
src/Compiler.h    Compiler ABI
PluginAPI/        Unity native plugin interface headers
CMakeLists.txt    Build and install configuration

License

This project is licensed under the MIT License, copyright 2026 af8a2a.

Third-party components retain their own licenses. Unity's native plugin interface headers are covered by PluginAPI/LICENSE.md. DXC redistribution licenses are installed when their files are available under VMS_DXC_ROOT.