Allow building TLAS from instances buffer on Vulkan and DX12

[AntonWagnerCau]

Connections
Part of the ray tracing tracking issue #6762, orthogonal to #9835

Description
Currently the only way to build a TLAS is CommandEncoder::build_acceleration_structures which serializes each TlasInstance on the CPU. For scenes with very large or GPU-produces instance sets that per-instance CPU work quickly adds up and becomes a bottleneck (See #9835).

This adds CommandEncoder::build_tlas_from_instances_buffer which build a TLAS directly from a buffer of instance records in the backend's native layout.

Further additions:

wgt::RawTlasInstance which mirrors the VkAccelerationStructureInstanceKHR / D3D12_RAYTRACING_INSTANCE_DESC layouts.

wgpu::util::TlasInstancePacker which is an optional reusable compute helper that packs the GPU per-instance transforms/indices/custom data into the RawTlasInstance records, which should be usable in many engine use-cases.

The instance buffer can be filled by a compute shader or a copy command in a previous submit, a previous encoder in the same submit or previously in the same encoder as the tlas build command.

Because the BLASes can't be recovered from the opaque buffer, the caller lists them in dependencies.

This is Vulkan/DX12 only
The raw layout differs on outside of these, so the build is rejected with a validation error on other backends.
Extending it to Metal should not be particularly challenging and would mostly involve defining a MetalRawTlasInstance and replacing the backend guards with branching. The packer in utils would also need to account for this.
The main reason for not proposing it for all backends in this PR is that i would have no way to test it myself.

Validation added (mirroring the CPU/BLAS paths): feature gate, backend guard, TLAS_INPUT usage, count ≤ max_instances, buffer-size, offset 16-byte alignment, and NeedsInitializedMemory registration for the instance buffer.

The API is a safe fn with a runtime backend guard rather than unsafe, and it rides on EXPERIMENTAL_RAY_QUERY rather than a new feature happy to change either if you'd prefer.

Trace recording (device/trace/record.rs) panic!s for this command since it has no serialized Command representation (same as the existing TransitionResources arm). The player mirrors it with unimplemented!.

Testing

New GPU tests in tests/tests/wgpu-gpu/ray_tracing/as_build_from_buffer.rs run on Vulkan + DX12:

Positive: single instance round-trips (hit / instance index / custom data / t); packer output; multi-instance with two BLASes at different depths (verifies per-instance transform, ordering, custom data, and that each blas_index resolves to the correct BLAS address); rebuild of the same TLAS with fewer instances; non-zero offset; empty (count == 0).

Validation asserting specific specific errors: too many instances, missing TLAS_INPUT usage, buffer too small, unaligned offset.

Further i tested it in my own engine, building heavily instanced scenes. Rebuilding after inserting, deleting, modifying instances. While measuring a noticeable performance improvement over the CPU path.

Squash or Rebase?

Squash

Checklist

• I self-reviewed and fully understand this PR.
• WebGPU implementations built with wgpu may be affected behaviorally.
• Validation and feature gates are in place to confine behavioral changes.
• Tests demonstrate the validation and altered logic works.
• CHANGELOG.md entries for the user-facing effects of this change are present.
• The PR is minimal, and doesn't make sense to land as multiple PRs.
• Commits are logically scoped and individually reviewable.
• The PR description has enough context to understand the motivation and solution implemented.

[AntonWagnerCau]

If it makes the PR less complex to review i can also split off the packer and reduce the tests to be just the validation errors. That would drop a lot of additional lines without affecting the functionality.

[Vecvec]

@AntonWagnerCau, the as_hal methods on the acceleration structures were added to try and allow for non-portable building. Are those functions insufficient for your use-case?

[AntonWagnerCau]

@Vecvec
Using as_hal + mark_acceleration_structures_built would technically allow building the AS in this way but it does pull up a lot of quite ugly handling on the caller's side.

The instance-packing compute dispatch and the build need separate encoders, since one would be raw and rest of the frame (at least in my case) wouldn't, with the caller hand-emitting the input barriers and restoring buffer states to what the usage tracker expects, including the cross-frame WAR hazard on the instance buffer.

The usage tracker would have to be handled manually aswell and the BLAS dependencies aren't recorded anywhere leaving the BLAS unvalidated and the keep-alive responsibilities on the caller side.

Scratch sizing/alignement/liftimes would also all fall on the caller to query and handle. From briefly looking into it getting the scratch offset alignment isn't trivial for a live device either but could probably be handled somehow..

If the missing metal compatibility is the blocker and you otherwise believe this to be a valuable addition i can try and add it to the PR, probably relying on tests and the ci/cd to verify on mac for me and maybe someone else in the community to actually run it before it gets merged.
Metal compatibility would actually strengthen the need for a wgpu side packing util so the caller does not have to worry about which backend's RawTlasInstance needs to be produced.

[Vecvec]

Even with Metal compatibility for the packer, the build function itself (assuming, for example, a user misses the util. function) still has a surprising change of behavior on different platforms. What are you trying to use this to do?

[AntonWagnerCau]

I have a rather large scene with about 1.7M instances in total. If i change an instance transform i have to call build_acceleration_structures.
Just the recording of the TlasInstances on the CPU takes ~90ms, with more additional costs down the line.
All the data is already on the GPU and hal already takes GPU resident buffers for it's AS construction. So i'm trying to close the gap so we don't have to dip to the CPU for the build.

I can see the concerns with missing the packer and trying to submit self constructed instance_buffers without accounting for portability.

To adress this i could make the RawInstance structs private and force a packing on wgpu side. This way a user can only submit structured gpu inputs and would have no access to the instances buffer itself. So the user wouldn't call i.e.:

        packer.pack(
            device,
            &mut encoder,
            &wgpu::util::TlasInstancePackParams {
                transforms: &transforms,
                blas_indices: &blas_indices,
                blas_addresses: &blas_adresses,
                custom_data: &custom_data,
                instances: &instances_buffer,
                count,
            },
        );

        encoder.build_tlas_from_instances_buffer(
            &self.tlas,
            wgpu::TlasInstancesBuffer {
                buffer: &instances_buffer,
                offset: 0,
                count,
                dependencies: &blas_deps,
            },
        );

but rather

encoder.build_tlas_from_gpu_instances(
    &self.tlas,
    wgpu::TlasGpuInstances {
        transforms: &self.transforms,
        blas_indices: &self.blas_indices,
        custom_data: &self.custom_data,
        count,
        dependencies: &blas_deps,
    },
);

This would add a small compute pipeline, probably into wgpu-core or hal though, which i don't know how desirable this is and was originally the reason i added the packer to util.

[Vecvec]

I'm still worried about code size and complexity. Are your transforms generated on the GPU? If not, it might be possible to rework the TlasInstance conversion to not require dynamic dispatch.

[AntonWagnerCau]

No my transforms are not GPU generated.
But i don't think it can get to sub-ms time on the cpu no matter how deeply it's rewritten.
On 1.7M instances there's about 200 MB of data (Instance Records, Transforms, Custom Data) which even if entirely bandwidth bound on RAM is around 4 ms. (Though please correct me if i'm missing something here)

Using this PR's code i measured 5 µs CPU time (and ~0.1-0.2 ms GPU time for packing) in the same setup.

I'm still open to trying to reduce the complexity and code size. But with the packer becoming mandatory for portability the only LOCs that are easily trimmable are the tests, which make up around 700 lines and can probably be reduced to making sure the correct errors are thrown on misuse.

[Vecvec]

How many instances do you update per build?

[AntonWagnerCau]

It varies but usually less than a hundred

[Vecvec]

wgpu currently tracks lowest_unmodified but doesn't use it, it could probably be used to only convert and update the instances which were changed (validation might still be needed for all instances [though I would have to check]).

[AntonWagnerCau]

using lowest_unmodified as the dirty tracking would require knowing ahead of time some subset of instances which frequently change so they can be put to the front of the instances buffer with performance degrading the higher up the modified instance is

[AntonWagnerCau]

If complexity and portability remain the main concerns i'd be willing to basically redo this PR but with metal compat and smaller chunked commits. I'd just need to know wether the compute pipeline for packing in core/hal would be acceptable and where you'd prefer it