diff --git a/API.md b/API.md index 088c5613..99afe983 100644 --- a/API.md +++ b/API.md @@ -117,6 +117,7 @@ GltfState containing a state for visualization in GltfView * [.animationIndices](#GltfState+animationIndices) * [.animationTimer](#GltfState+animationTimer) * [.variant](#GltfState+variant) + * [.needsRedraw](#GltfState+needsRedraw) * [.renderingParameters](#GltfState+renderingParameters) * [.morphing](#GltfState+renderingParameters.morphing) * [.skinning](#GltfState+renderingParameters.skinning) @@ -132,6 +133,7 @@ GltfState containing a state for visualization in GltfView * [.environmentRotation](#GltfState+renderingParameters.environmentRotation) * [.useDirectionalLightsWithDisabledIBL](#GltfState+renderingParameters.useDirectionalLightsWithDisabledIBL) * [.internalMSAA](#GltfState+renderingParameters.internalMSAA) + * [.floatingPointFramebuffer](#GltfState+renderingParameters.floatingPointFramebuffer) * _static_ * [.ToneMaps](#GltfState.ToneMaps) * [.KHR_PBR_NEUTRAL](#GltfState.ToneMaps.KHR_PBR_NEUTRAL) @@ -239,6 +241,12 @@ animation timer allows to control the animation time ### gltfState.variant KHR_materials_variants +**Kind**: instance property of [GltfState](#GltfState) + + +### gltfState.needsRedraw +Indicates whether the view needs to be redrawn, currently used to indicate new sorting orders for gaussian splatting + **Kind**: instance property of [GltfState](#GltfState) @@ -262,6 +270,7 @@ parameters used to configure the rendering * [.environmentRotation](#GltfState+renderingParameters.environmentRotation) * [.useDirectionalLightsWithDisabledIBL](#GltfState+renderingParameters.useDirectionalLightsWithDisabledIBL) * [.internalMSAA](#GltfState+renderingParameters.internalMSAA) + * [.floatingPointFramebuffer](#GltfState+renderingParameters.floatingPointFramebuffer) @@ -278,7 +287,7 @@ skin / skeleton #### renderingParameters.clearColor -clear color expressed as list of ints in the range [0, 255] +clear color expressed as list of floats in the range [0, 1] **Kind**: static property of [renderingParameters](#GltfState+renderingParameters) @@ -351,6 +360,12 @@ If this is set to true, directional lights will be generated if IBL is disabled #### renderingParameters.internalMSAA MSAA used for cases which are not handled by the browser (e.g. Transmission) +**Kind**: static property of [renderingParameters](#GltfState+renderingParameters) + + +#### renderingParameters.floatingPointFramebuffer +Use RGBA16F floating-point main framebuffer instead of RGBA8 + **Kind**: static property of [renderingParameters](#GltfState+renderingParameters) @@ -1010,7 +1025,9 @@ Fit view to updated canvas size without changing rotation if distance is incorre ## ResourceLoaderUtils -Utility class providing static helper methods for resource loading operations, such as extracting file extensions, resolving folder paths, normalizing relative paths, and detecting absolute URLs. +Utility class providing static helper methods for resource loading operations, +such as extracting file extensions, resolving folder paths, normalizing relative +paths, and detecting absolute URLs. **Kind**: global class @@ -1026,7 +1043,8 @@ Utility class providing static helper methods for resource loading operations, s Extracts the file extension from a filename. **Kind**: static method of [ResourceLoaderUtils](#ResourceLoaderUtils) -**Returns**: string \| undefined - The lowercase file extension (without the leading dot), or `undefined` if the filename has no extension. +**Returns**: string \| undefined - The lowercase file extension (without the leading dot), + or `undefined` if the filename has no extension. | Param | Type | Description | | --- | --- | --- | @@ -1038,7 +1056,8 @@ Extracts the file extension from a filename. Returns the directory portion of a file path, including the trailing slash. **Kind**: static method of [ResourceLoaderUtils](#ResourceLoaderUtils) -**Returns**: string - The path up to and including the last `/`, or an empty string if no `/` is present. +**Returns**: string - The path up to and including the last `/`, or an empty string + if no `/` is present. | Param | Type | Description | | --- | --- | --- | @@ -1047,7 +1066,10 @@ Returns the directory portion of a file path, including the trailing slash. ### ResourceLoaderUtils.cleanRelativePath(relativePath) ⇒ string -Normalizes a relative URL path by resolving `.` and `..` segments. - Strips a leading `./` prefix. - Collapses `/./` sequences to `/`. - Resolves `/../` sequences by removing the preceding path segment. +Normalizes a relative URL path by resolving `.` and `..` segments. +- Strips a leading `./` prefix. +- Collapses `/./` sequences to `/`. +- Resolves `/../` sequences by removing the preceding path segment. **Kind**: static method of [ResourceLoaderUtils](#ResourceLoaderUtils) **Returns**: string - The normalized path with dot segments resolved. @@ -1059,7 +1081,8 @@ Normalizes a relative URL path by resolving `.` and `..` segments. - Strips a le ### ResourceLoaderUtils.isAbsoluteUrl(url) ⇒ boolean -Determines whether a URL is absolute (i.e. contains a scheme such as `http:` or `data:`). A URL is considered absolute when it contains a `:` that appears before any `/`. +Determines whether a URL is absolute (i.e. contains a scheme such as `http:` or `data:`). +A URL is considered absolute when it contains a `:` that appears before any `/`. **Kind**: static method of [ResourceLoaderUtils](#ResourceLoaderUtils) **Returns**: boolean - `true` if the URL is absolute, `false` otherwise. diff --git a/README.md b/README.md index ed089363..60f04a3c 100644 --- a/README.md +++ b/README.md @@ -30,6 +30,7 @@ Formerly hosted together with the example frontend at the [glTF Sample Viewer](h - [x] glTF 2.0 - [x] [KHR_animation_pointer](https://github.com/KhronosGroup/glTF/tree/main/extensions/2.0/Khronos/KHR_animation_pointer) - [x] [KHR_draco_mesh_compression](https://github.com/KhronosGroup/glTF/tree/main/extensions/2.0/Khronos/KHR_draco_mesh_compression) +- [x] [KHR_gaussian_splatting](https://github.com/KhronosGroup/glTF/blob/main/extensions/2.0/Khronos/KHR_gaussian_splatting/README.md) - [x] [KHR_lights_punctual](https://github.com/KhronosGroup/glTF/tree/main/extensions/2.0/Khronos/KHR_lights_punctual) - [x] [KHR_materials_anisotropy](https://github.com/KhronosGroup/glTF/tree/main/extensions/2.0/Khronos/KHR_materials_anisotropy) - [x] [KHR_materials_clearcoat](https://github.com/KhronosGroup/glTF/tree/main/extensions/2.0/Khronos/KHR_materials_clearcoat) diff --git a/source/GltfState/gltf_state.js b/source/GltfState/gltf_state.js index d4b76038..bd279724 100644 --- a/source/GltfState/gltf_state.js +++ b/source/GltfState/gltf_state.js @@ -31,6 +31,9 @@ class GltfState { /** KHR_materials_variants */ this.variant = undefined; + /** Indicates whether the view needs to be redrawn, currently used to indicate new sorting orders for gaussian splatting */ + this.needsRedraw = false; + /** parameters used to configure the rendering */ this.renderingParameters = { /** morphing between vertices */ @@ -55,15 +58,19 @@ class GltfState { KHR_materials_specular: true, /** KHR_materials_iridescence adds a thin-film iridescence effect */ KHR_materials_iridescence: true, + /** KHR_materials_diffuse_transmission */ KHR_materials_diffuse_transmission: true, /** KHR_materials_anisotropy defines microfacet grooves in the surface, stretching the specular reflection on the surface */ KHR_materials_anisotropy: true, /** KHR_materials_dispersion defines configuring the strength of the angular separation of colors (chromatic abberation)*/ KHR_materials_dispersion: true, - KHR_materials_emissive_strength: true + /** KHR_materials_emissive_strength allows configuring the strength of the emissive component */ + KHR_materials_emissive_strength: true, + /** KHR_gaussian_splatting */ + KHR_gaussian_splatting: true }, - /** clear color expressed as list of ints in the range [0, 255] */ - clearColor: [58, 64, 74, 255], + /** clear color expressed as list of floats in the range [0, 1] */ + clearColor: [0.22, 0.25, 0.29, 1], /** exposure factor */ exposure: 1.0, /** KHR_lights_punctual */ @@ -92,7 +99,9 @@ class GltfState { /** If this is set to true, directional lights will be generated if IBL is disabled */ useDirectionalLightsWithDisabledIBL: false, /** MSAA used for cases which are not handled by the browser (e.g. Transmission)*/ - internalMSAA: 4 + internalMSAA: 4, + /** Use RGBA16F floating-point main framebuffer instead of RGBA8 */ + floatingPointFramebuffer: true }; // retain a reference to the view with which the state was created, so that it can be validated @@ -226,6 +235,17 @@ GltfState.DebugOutput = { SINGLE_SCATTER_COLOR: "Single-Scatter Color", /** output for the pre scatter pass, which collects all lighting contribution for scattering */ PRE_SCATTER_PASS: "Pre-Scatter Pass" + }, + + gaussianSplatting: { + /** output the spherical harmonics degree 0 */ + SH_DEGREE_0: "SH Degree 0", + /** output the spherical harmonics degree 0-1 */ + SH_DEGREE_1: "SH Degree 1", + /** output the spherical harmonics degree 0-2 */ + SH_DEGREE_2: "SH Degree 2", + /** output the spherical harmonics degree 0-3 */ + SH_DEGREE_3: "SH Degree 3" } }; diff --git a/source/Renderer/renderer.js b/source/Renderer/renderer.js index ca0cbc51..ac2fe9c2 100644 --- a/source/Renderer/renderer.js +++ b/source/Renderer/renderer.js @@ -19,6 +19,11 @@ import cubemapVertShader from "./shaders/cubemap.vert"; import cubemapFragShader from "./shaders/cubemap.frag"; import scatterShader from "./shaders/scatter.frag"; import specularGlossinesShader from "./shaders/specular_glossiness.frag"; +import splatVertexShader from "./shaders/splat.vert"; +import splatFragmentShader from "./shaders/splat.frag"; +import fullscreenVertShader from "./shaders/fullscreen.vert"; +import tonemapMainFragShader from "./shaders/tonemap_main.frag"; +import splatCompositeFragShader from "./shaders/splat_composite.frag"; import { gltfLight } from "../gltf/light.js"; import { jsToGl } from "../gltf/utils.js"; import { gltfMaterial } from "../gltf/material.js"; @@ -61,6 +66,11 @@ class gltfRenderer { shaderSources.set("cubemap.vert", cubemapVertShader); shaderSources.set("cubemap.frag", cubemapFragShader); shaderSources.set("specular_glossiness.frag", specularGlossinesShader); + shaderSources.set("splat.vert", splatVertexShader); + shaderSources.set("splat.frag", splatFragmentShader); + shaderSources.set("fullscreen.vert", fullscreenVertShader); + shaderSources.set("tonemap_main.frag", tonemapMainFragShader); + shaderSources.set("splat_composite.frag", splatCompositeFragShader); this.shaderCache = new ShaderCache(shaderSources, this.webGl); @@ -86,6 +96,28 @@ class gltfRenderer { this.maxVertAttributes = undefined; this.instanceBuffer = undefined; + + this.splatVBO = undefined; + this.currentSortBuffer = undefined; + + this.mainTexture = undefined; + this.mainTextureHDR = undefined; + this.mainDepthTexture = undefined; + this.mainTonemapTexture = undefined; + this.mainFramebuffer = undefined; + + this.framebufferFormat = undefined; + this.framebufferType = undefined; + + // Tracks last value of state.renderingParameters.floatingPointFramebuffer so we + // only re-attach when the setting actually changes. + this._floatingPointFramebuffer = undefined; + + // Splat isolation framebuffer: same colour format as mainFramebuffer, + // shares mainDepthTexture for depth-test (no depth writes). + this.splatColorTexture = undefined; + this.splatColorTextureHDR = undefined; + this.splatFramebuffer = undefined; } ///////////////////////////////////////////////////////////////////// @@ -181,6 +213,113 @@ class gltfRenderer { context.viewport(0, 0, this.opaqueFramebufferWidth, this.opaqueFramebufferHeight); context.bindFramebuffer(context.FRAMEBUFFER, null); + const quatVertices = new Float32Array([-1, -1, 1, -1, -1, 1, 1, 1]); + this.splatVBO = context.createBuffer(); + context.bindBuffer(context.ARRAY_BUFFER, this.splatVBO); + context.bufferData(context.ARRAY_BUFFER, quatVertices, context.STATIC_DRAW); + context.bindBuffer(context.ARRAY_BUFFER, null); + + this.currentSortBuffer = context.createBuffer(); + + this.framebufferFormat = context.supports_EXT_color_buffer_half_float + ? context.RGBA16F : context.RGBA; + this.framebufferType = context.supports_EXT_color_buffer_half_float + ? context.HALF_FLOAT : context.UNSIGNED_BYTE; + + const hdrW = Math.max(this.currentWidth, 1); + const hdrH = Math.max(this.currentHeight, 1); + + // Main color texture + this.mainTexture = context.createTexture(); + context.bindTexture(context.TEXTURE_2D, this.mainTexture); + context.texParameteri(context.TEXTURE_2D, context.TEXTURE_MIN_FILTER, context.NEAREST); + context.texParameteri(context.TEXTURE_2D, context.TEXTURE_MAG_FILTER, context.NEAREST); + context.texParameteri(context.TEXTURE_2D, context.TEXTURE_WRAP_S, context.CLAMP_TO_EDGE); + context.texParameteri(context.TEXTURE_2D, context.TEXTURE_WRAP_T, context.CLAMP_TO_EDGE); + context.texImage2D(context.TEXTURE_2D, 0, context.RGBA, + hdrW, hdrH, 0, context.RGBA, context.UNSIGNED_BYTE, null); + context.bindTexture(context.TEXTURE_2D, null); + + if (context.supports_EXT_color_buffer_half_float) { + this.mainTextureHDR = context.createTexture(); + context.bindTexture(context.TEXTURE_2D, this.mainTextureHDR); + context.texParameteri(context.TEXTURE_2D, context.TEXTURE_MIN_FILTER, context.NEAREST); + context.texParameteri(context.TEXTURE_2D, context.TEXTURE_MAG_FILTER, context.NEAREST); + context.texParameteri(context.TEXTURE_2D, context.TEXTURE_WRAP_S, context.CLAMP_TO_EDGE); + context.texParameteri(context.TEXTURE_2D, context.TEXTURE_WRAP_T, context.CLAMP_TO_EDGE); + context.texImage2D(context.TEXTURE_2D, 0, context.RGBA16F, + hdrW, hdrH, 0, context.RGBA, context.HALF_FLOAT, null); + context.bindTexture(context.TEXTURE_2D, null); + } + + // Main depth texture + this.mainDepthTexture = context.createTexture(); + context.bindTexture(context.TEXTURE_2D, this.mainDepthTexture); + context.texParameteri(context.TEXTURE_2D, context.TEXTURE_MIN_FILTER, context.NEAREST); + context.texParameteri(context.TEXTURE_2D, context.TEXTURE_MAG_FILTER, context.NEAREST); + context.texParameteri(context.TEXTURE_2D, context.TEXTURE_WRAP_S, context.CLAMP_TO_EDGE); + context.texParameteri(context.TEXTURE_2D, context.TEXTURE_WRAP_T, context.CLAMP_TO_EDGE); + context.texImage2D(context.TEXTURE_2D, 0, context.DEPTH_COMPONENT24, + hdrW, hdrH, 0, context.DEPTH_COMPONENT, context.UNSIGNED_INT, null); + context.bindTexture(context.TEXTURE_2D, null); + + this.mainFramebuffer = context.createFramebuffer(); + context.bindFramebuffer(context.FRAMEBUFFER, this.mainFramebuffer); + context.framebufferTexture2D(context.FRAMEBUFFER, context.COLOR_ATTACHMENT0, + context.TEXTURE_2D, this.mainTexture, 0); + context.framebufferTexture2D(context.FRAMEBUFFER, context.DEPTH_ATTACHMENT, + context.TEXTURE_2D, this.mainDepthTexture, 0); + + // Tonemap flag texture: R8UI, one flag per pixel. + // 0 = linear only, 1 = sRGB gamma only, 2 = tonemap + sRGB. + this.mainTonemapTexture = context.createTexture(); + context.bindTexture(context.TEXTURE_2D, this.mainTonemapTexture); + context.texParameteri(context.TEXTURE_2D, context.TEXTURE_MIN_FILTER, context.NEAREST); + context.texParameteri(context.TEXTURE_2D, context.TEXTURE_MAG_FILTER, context.NEAREST); + context.texParameteri(context.TEXTURE_2D, context.TEXTURE_WRAP_S, context.CLAMP_TO_EDGE); + context.texParameteri(context.TEXTURE_2D, context.TEXTURE_WRAP_T, context.CLAMP_TO_EDGE); + context.texImage2D(context.TEXTURE_2D, 0, context.R8UI, + hdrW, hdrH, 0, context.RED_INTEGER, context.UNSIGNED_BYTE, null); + context.bindTexture(context.TEXTURE_2D, null); + + context.framebufferTexture2D(context.FRAMEBUFFER, context.COLOR_ATTACHMENT1, + context.TEXTURE_2D, this.mainTonemapTexture, 0); + context.drawBuffers([context.COLOR_ATTACHMENT0, context.COLOR_ATTACHMENT1]); + context.bindFramebuffer(context.FRAMEBUFFER, null); + + this.splatColorTexture = context.createTexture(); + context.bindTexture(context.TEXTURE_2D, this.splatColorTexture); + context.texParameteri(context.TEXTURE_2D, context.TEXTURE_MIN_FILTER, context.NEAREST); + context.texParameteri(context.TEXTURE_2D, context.TEXTURE_MAG_FILTER, context.NEAREST); + context.texParameteri(context.TEXTURE_2D, context.TEXTURE_WRAP_S, context.CLAMP_TO_EDGE); + context.texParameteri(context.TEXTURE_2D, context.TEXTURE_WRAP_T, context.CLAMP_TO_EDGE); + context.texImage2D(context.TEXTURE_2D, 0, context.RGBA, + hdrW, hdrH, 0, context.RGBA, context.UNSIGNED_BYTE, null); + context.bindTexture(context.TEXTURE_2D, null); + + if (context.supports_EXT_color_buffer_half_float) { + this.splatColorTextureHDR = context.createTexture(); + context.bindTexture(context.TEXTURE_2D, this.splatColorTextureHDR); + context.texParameteri(context.TEXTURE_2D, context.TEXTURE_MIN_FILTER, context.NEAREST); + context.texParameteri(context.TEXTURE_2D, context.TEXTURE_MAG_FILTER, context.NEAREST); + context.texParameteri(context.TEXTURE_2D, context.TEXTURE_WRAP_S, context.CLAMP_TO_EDGE); + context.texParameteri(context.TEXTURE_2D, context.TEXTURE_WRAP_T, context.CLAMP_TO_EDGE); + context.texImage2D(context.TEXTURE_2D, 0, context.RGBA16F, + hdrW, hdrH, 0, context.RGBA, context.HALF_FLOAT, null); + context.bindTexture(context.TEXTURE_2D, null); + } + + // Initially attach the LDR colour texture; re-attached together with + // mainFramebuffer when _floatingPointFramebuffer is resolved. + this.splatFramebuffer = context.createFramebuffer(); + context.bindFramebuffer(context.FRAMEBUFFER, this.splatFramebuffer); + context.framebufferTexture2D(context.FRAMEBUFFER, context.COLOR_ATTACHMENT0, + context.TEXTURE_2D, this.splatColorTexture, 0); + context.framebufferTexture2D(context.FRAMEBUFFER, context.DEPTH_ATTACHMENT, + context.TEXTURE_2D, this.mainDepthTexture, 0); + context.drawBuffers([context.COLOR_ATTACHMENT0]); + context.bindFramebuffer(context.FRAMEBUFFER, null); + this.maxVertAttributes = context.getParameter(context.MAX_VERTEX_ATTRIBS); this.initialized = true; @@ -250,26 +389,143 @@ class gltfRenderer { ); this.webGl.context.bindTexture(this.webGl.context.TEXTURE_2D, null); this.webGl.context.bindFramebuffer(this.webGl.context.FRAMEBUFFER, null); + + // Resize main color texture (always RGBA/UNSIGNED_BYTE). + this.webGl.context.bindTexture(this.webGl.context.TEXTURE_2D, this.mainTexture); + this.webGl.context.texImage2D( + this.webGl.context.TEXTURE_2D, + 0, + this.webGl.context.RGBA, + this.currentWidth, + this.currentHeight, + 0, + this.webGl.context.RGBA, + this.webGl.context.UNSIGNED_BYTE, + null + ); + + // Resize HDR color texture (RGBA16F/HALF_FLOAT, only when available). + if (this.mainTextureHDR) { + this.webGl.context.bindTexture( + this.webGl.context.TEXTURE_2D, + this.mainTextureHDR + ); + this.webGl.context.texImage2D( + this.webGl.context.TEXTURE_2D, + 0, + this.webGl.context.RGBA16F, + this.currentWidth, + this.currentHeight, + 0, + this.webGl.context.RGBA, + this.webGl.context.HALF_FLOAT, + null + ); + } + + this.webGl.context.bindTexture( + this.webGl.context.TEXTURE_2D, + this.mainDepthTexture + ); + this.webGl.context.texImage2D( + this.webGl.context.TEXTURE_2D, + 0, + this.webGl.context.DEPTH_COMPONENT24, + this.currentWidth, + this.currentHeight, + 0, + this.webGl.context.DEPTH_COMPONENT, + this.webGl.context.UNSIGNED_INT, + null + ); + this.webGl.context.bindTexture(this.webGl.context.TEXTURE_2D, null); + + // Resize tonemap flag texture (R8UI). + this.webGl.context.bindTexture( + this.webGl.context.TEXTURE_2D, + this.mainTonemapTexture + ); + this.webGl.context.texImage2D( + this.webGl.context.TEXTURE_2D, + 0, + this.webGl.context.R8UI, + this.currentWidth, + this.currentHeight, + 0, + this.webGl.context.RED_INTEGER, + this.webGl.context.UNSIGNED_BYTE, + null + ); + this.webGl.context.bindTexture(this.webGl.context.TEXTURE_2D, null); + + // Resize splat isolation colour textures. + this.webGl.context.bindTexture( + this.webGl.context.TEXTURE_2D, + this.splatColorTexture + ); + this.webGl.context.texImage2D( + this.webGl.context.TEXTURE_2D, + 0, + this.webGl.context.RGBA, + this.currentWidth, + this.currentHeight, + 0, + this.webGl.context.RGBA, + this.webGl.context.UNSIGNED_BYTE, + null + ); + if (this.splatColorTextureHDR) { + this.webGl.context.bindTexture( + this.webGl.context.TEXTURE_2D, + this.splatColorTextureHDR + ); + this.webGl.context.texImage2D( + this.webGl.context.TEXTURE_2D, + 0, + this.webGl.context.RGBA16F, + this.currentWidth, + this.currentHeight, + 0, + this.webGl.context.RGBA, + this.webGl.context.HALF_FLOAT, + null + ); + } + this.webGl.context.bindTexture(this.webGl.context.TEXTURE_2D, null); } } } // frame state clearFrame(clearColor) { + // Convert clear color from sRGB to linear since we will always transfer to sRGB by default + const linearClearColor = clearColor.map((c) => Math.pow(c, 2.2)); this.webGl.context.bindFramebuffer(this.webGl.context.FRAMEBUFFER, this.opaqueFramebuffer); - this.webGl.context.clearColor(...clearColor); + this.webGl.context.clearColor(...linearClearColor); this.webGl.context.clear(GL.COLOR_BUFFER_BIT | GL.DEPTH_BUFFER_BIT); this.webGl.context.bindFramebuffer( this.webGl.context.FRAMEBUFFER, this.opaqueFramebufferMSAA ); - this.webGl.context.clearColor(...clearColor); + this.webGl.context.clearColor(...linearClearColor); this.webGl.context.clear(GL.COLOR_BUFFER_BIT | GL.DEPTH_BUFFER_BIT); this.webGl.context.bindFramebuffer(this.webGl.context.FRAMEBUFFER, this.scatterFramebuffer); this.webGl.context.clearColor(0, 0, 0, 0); this.webGl.context.clear(GL.COLOR_BUFFER_BIT | GL.DEPTH_BUFFER_BIT); + // Main framebuffer: clear color (attachment 0) and depth separately, + // then zero the tonemap flag attachment (attachment 1) via clearBufferuiv. + if (this.mainFramebuffer) { + const gl = this.webGl.context; + gl.bindFramebuffer(gl.FRAMEBUFFER, this.mainFramebuffer); + // Clear color attachment 0 to the scene background colour. + gl.clearBufferfv(gl.COLOR, 0, linearClearColor); + // Clear tonemap flag attachment to 1 (linear to sRGB by default). + gl.clearBufferuiv(gl.COLOR, 1, new Uint32Array([1, 1, 1, 1])); + // Clear depth. + gl.clearBufferfv(gl.DEPTH, 0, new Float32Array([1.0])); + } this.webGl.context.bindFramebuffer(this.webGl.context.FRAMEBUFFER, null); - this.webGl.context.clearColor(...clearColor); + this.webGl.context.clearColor(...linearClearColor); this.webGl.context.clear(GL.COLOR_BUFFER_BIT | GL.DEPTH_BUFFER_BIT); this.webGl.context.bindFramebuffer(this.webGl.context.FRAMEBUFFER, null); } @@ -298,7 +554,8 @@ class gltfRenderer { state.gltf.materials[primitive.material].alphaMode !== "BLEND" && (state.gltf.materials[primitive.material].extensions === undefined || state.gltf.materials[primitive.material].extensions - .KHR_materials_transmission === undefined) + .KHR_materials_transmission === undefined) && + primitive.extensions?.KHR_gaussian_splatting === undefined ); let counter = 0; @@ -330,17 +587,19 @@ class gltfRenderer { // transparent drawables need sorting before they can be drawn this.transparentDrawables = drawables.filter( ({ primitive }) => - state.gltf.materials[primitive.material].alphaMode === "BLEND" && - (state.gltf.materials[primitive.material].extensions === undefined || - state.gltf.materials[primitive.material].extensions - .KHR_materials_transmission === undefined) + (state.gltf.materials[primitive.material].alphaMode === "BLEND" && + (state.gltf.materials[primitive.material].extensions === undefined || + state.gltf.materials[primitive.material].extensions + .KHR_materials_transmission === undefined)) || + primitive.extensions?.KHR_gaussian_splatting !== undefined ); this.transmissionDrawables = drawables.filter( ({ primitive }) => state.gltf.materials[primitive.material].extensions !== undefined && state.gltf.materials[primitive.material].extensions.KHR_materials_transmission !== - undefined + undefined && + primitive.extensions?.KHR_gaussian_splatting === undefined ); this.scatterDrawables = drawables.filter( @@ -349,7 +608,8 @@ class gltfRenderer { state.gltf.materials[primitive.material].extensions.KHR_materials_volume_scatter !== undefined && state.gltf.materials[primitive.material].extensions.KHR_materials_volume !== - undefined + undefined && + primitive.extensions?.KHR_gaussian_splatting === undefined ); } @@ -503,7 +763,7 @@ class gltfRenderer { this.viewProjectionMatrix, state, this.shaderCache, - ["LINEAR_OUTPUT 1"] + ["TRANSMISSION_PASS 1"] ); let drawableCounter = 0; @@ -511,6 +771,7 @@ class gltfRenderer { const drawable = instance[0]; let renderpassConfiguration = {}; renderpassConfiguration.linearOutput = true; + renderpassConfiguration.transmission = true; renderpassConfiguration.frameBufferSize = [ this.opaqueFramebufferWidth, this.opaqueFramebufferHeight @@ -541,6 +802,7 @@ class gltfRenderer { for (const drawable of this.transparentDrawables) { let renderpassConfiguration = {}; renderpassConfiguration.linearOutput = true; + renderpassConfiguration.transmission = true; renderpassConfiguration.frameBufferSize = [ this.opaqueFramebufferWidth, this.opaqueFramebufferHeight @@ -582,26 +844,54 @@ class gltfRenderer { this.webGl.context.generateMipmap(this.webGl.context.TEXTURE_2D); } - // Render to canvas - this.webGl.context.bindFramebuffer(this.webGl.context.FRAMEBUFFER, null); + // Re-attach mainFramebuffer and splatFramebuffer COLOR_ATTACHMENT0 if + // the floating-point toggle changed. + const wantFP = + state.renderingParameters.floatingPointFramebuffer !== false && + this.mainTextureHDR !== undefined; + if (wantFP !== this._floatingPointFramebuffer) { + this._floatingPointFramebuffer = wantFP; + const gl = this.webGl.context; + const colorTex = wantFP ? this.mainTextureHDR : this.mainTexture; + const splatColorTex = wantFP ? this.splatColorTextureHDR : this.splatColorTexture; + gl.bindFramebuffer(gl.FRAMEBUFFER, this.mainFramebuffer); + gl.framebufferTexture2D( + gl.FRAMEBUFFER, + gl.COLOR_ATTACHMENT0, + gl.TEXTURE_2D, + colorTex, + 0 + ); + if (this.splatFramebuffer) { + gl.bindFramebuffer(gl.FRAMEBUFFER, this.splatFramebuffer); + gl.framebufferTexture2D( + gl.FRAMEBUFFER, + gl.COLOR_ATTACHMENT0, + gl.TEXTURE_2D, + splatColorTex ?? this.splatColorTexture, + 0 + ); + } + gl.bindFramebuffer(gl.FRAMEBUFFER, null); + } + + this.webGl.context.bindFramebuffer(this.webGl.context.FRAMEBUFFER, this.mainFramebuffer); this.webGl.context.viewport(aspectOffsetX, aspectOffsetY, aspectWidth, aspectHeight); - // Render environment - const fragDefines = []; - this.pushFragParameterDefines(fragDefines, state); + // Environment (always linear in this pass) this.environmentRenderer.drawEnvironmentMap( this.webGl, this.viewProjectionMatrix, state, this.shaderCache, - fragDefines + ["LINEAR_OUTPUT 1"] ); let drawableCounter = 0; for (const instance of Object.values(this.opaqueDrawables)) { const drawable = instance[0]; let renderpassConfiguration = {}; - renderpassConfiguration.linearOutput = false; + renderpassConfiguration.linearOutput = true; renderpassConfiguration.frameBufferSize = [this.currentWidth, this.currentHeight]; const instanceOffset = instanceWorldTransforms[drawableCounter]; drawableCounter++; @@ -628,7 +918,7 @@ class gltfRenderer { ); for (const drawable of this.transmissionDrawables.filter((a) => a.depth <= 0)) { let renderpassConfiguration = {}; - renderpassConfiguration.linearOutput = false; + renderpassConfiguration.linearOutput = true; renderpassConfiguration.frameBufferSize = [this.currentWidth, this.currentHeight]; let sampledTextures = {}; sampledTextures.transmissionSampleTexture = this.opaqueRenderTexture; @@ -650,18 +940,306 @@ class gltfRenderer { state.gltf, this.transparentDrawables ); + this.needsRedraw = false; + + // ── Transparent drawables (non-splat and splat) in depth-sorted order ─── + // Splats are rendered one at a time into the dedicated splatFramebuffer + // (which shares mainDepthTexture for depth-testing) and + // immediately composited back into mainFramebuffer before the next + // drawable is processed. This preserves correct depth-sorted blending + // order between splats and regular transparent geometry. for (const drawable of this.transparentDrawables.filter((a) => a.depth <= 0)) { - let renderpassConfiguration = {}; - renderpassConfiguration.linearOutput = false; - renderpassConfiguration.frameBufferSize = [this.currentWidth, this.currentHeight]; - this.drawPrimitive( - state, - renderpassConfiguration, - drawable.primitive, - drawable.node, - this.viewProjectionMatrix - ); + if ( + drawable.primitive.extensions?.KHR_gaussian_splatting !== undefined && + state.renderingParameters.enabledExtensions.KHR_gaussian_splatting + ) { + if (currentCamera.type !== "perspective") { + console.warn("Splat rendering with non-perspective projection is undefined"); + } + // Isolate this splat in its own framebuffer pass. + if (this.splatFramebuffer) { + const gl = this.webGl.context; + + // Clear only the colour attachment – depth is shared with + // mainFramebuffer and must not be touched here. + gl.bindFramebuffer(gl.FRAMEBUFFER, this.splatFramebuffer); + gl.clearBufferfv(gl.COLOR, 0, [0, 0, 0, 0]); + gl.viewport(aspectOffsetX, aspectOffsetY, aspectWidth, aspectHeight); + + this.drawSplat( + state, + drawable.primitive, + drawable.node, + this.projMatrix, + this.viewMatrix + ); + + // Composite into mainFramebuffer. + gl.bindFramebuffer(gl.FRAMEBUFFER, this.mainFramebuffer); + gl.viewport(aspectOffsetX, aspectOffsetY, aspectWidth, aspectHeight); + this.splatCompositePass(state, drawable.primitive.linear); + } + } else { + let renderpassConfiguration = {}; + renderpassConfiguration.linearOutput = true; + renderpassConfiguration.frameBufferSize = [this.currentWidth, this.currentHeight]; + this.drawPrimitive( + state, + renderpassConfiguration, + drawable.primitive, + drawable.node, + this.viewProjectionMatrix + ); + } } + + // ── Final tonemapping pass → canvas ─────────────────────────────────── + this.tonemapPass(state, aspectOffsetX, aspectOffsetY, aspectWidth, aspectHeight); + + state.needsRedraw = this.needsRedraw; + } + + drawSplat(state, primitive, node, projectionMatrix, viewMatrix) { + if (primitive.skip) return; + // Request an async worker sort each frame (no-op if the previous sort + // has not yet finished or no worker is available). + const modelViewMatrix = mat4.multiply(mat4.create(), viewMatrix, node.worldTransform); + primitive.requestSort(modelViewMatrix); + if (primitive.sortPending) { + this.needsRedraw = true; + } + let defines = primitive.defines.slice(); + if (primitive.linear === true) { + defines.push("LINEAR_OUTPUT 1"); + } + + // Debug views + if (state.renderingParameters.debugOutput !== GltfState.DebugOutput.NONE) { + if ( + state.renderingParameters.debugOutput === + GltfState.DebugOutput.gaussianSplatting.SH_DEGREE_0 + ) { + defines = defines.filter( + (define) => !define.startsWith("HAS_GAUSSIAN_SPLATTING_DEGREE") + ); + } else if ( + state.renderingParameters.debugOutput === + GltfState.DebugOutput.gaussianSplatting.SH_DEGREE_1 + ) { + defines = defines.filter( + (define) => + define !== "HAS_GAUSSIAN_SPLATTING_DEGREE_2 1" && + define !== "HAS_GAUSSIAN_SPLATTING_DEGREE_3 1" + ); + } else if ( + state.renderingParameters.debugOutput === + GltfState.DebugOutput.gaussianSplatting.SH_DEGREE_2 + ) { + defines = defines.filter( + (define) => define !== "HAS_GAUSSIAN_SPLATTING_DEGREE_3 1" + ); + } + } + + const fragmentHash = this.shaderCache.selectShader("splat.frag", defines); + const vertexHash = this.shaderCache.selectShader("splat.vert", defines); + if (fragmentHash && vertexHash) { + this.shader = this.shaderCache.getShaderProgram(fragmentHash, vertexHash); + } + + if (this.shader === undefined) { + return; + } + + this.webGl.context.useProgram(this.shader.program); + + this.webGl.context.uniformMatrix4fv( + this.shader.getUniformLocation("u_ModelMatrix"), + false, + node.worldTransform + ); + + const rotationMatrixInv = mat3.create(); + mat3.fromQuat(rotationMatrixInv, node.worldQuaternion); + mat3.transpose(rotationMatrixInv, rotationMatrixInv); + this.shader.updateUniform("u_ModelRotationInv", rotationMatrixInv); + + this.webGl.context.uniformMatrix4fv( + this.shader.getUniformLocation("u_ViewMatrix"), + false, + viewMatrix + ); + this.webGl.context.uniformMatrix4fv( + this.shader.getUniformLocation("u_ProjectionMatrix"), + false, + projectionMatrix + ); + this.webGl.context.uniform2i( + this.shader.getUniformLocation("u_FramebufferSize"), + this.currentWidth, + this.currentHeight + ); + // The projection matrix stores the focal length in the first and second element of the diagonal. + // We need to convert from NDC space to screen space, which is done by multiplying with the framebuffer dimensions and dividing by 2, since NDC goes from -1 to 1. + this.webGl.context.uniform2f( + this.shader.getUniformLocation("u_FocalLength"), + this.projMatrix[0] * this.currentWidth * 0.5, + this.projMatrix[5] * this.currentHeight * 0.5 + ); + this.shader.updateUniform("u_TextureWidth", primitive.splatTextureWidth); + this.shader.updateUniform("u_Camera", this.currentCameraPosition); + + this.webGl.context.enable(GL.BLEND); + this.webGl.context.blendFuncSeparate( + GL.ONE, + GL.ONE_MINUS_SRC_ALPHA, + GL.ONE, + GL.ONE_MINUS_SRC_ALPHA + ); + this.webGl.context.blendEquation(GL.FUNC_ADD); + this.webGl.context.depthMask(false); + + let textureIndex = 0; + + let location = this.shader.getUniformLocation(primitive.positionTextureInfo.samplerName); + this.webGl.setTexture(location, state.gltf, primitive.positionTextureInfo, textureIndex++); + + location = this.shader.getUniformLocation(primitive.rotationTextureInfo.samplerName); + this.webGl.setTexture(location, state.gltf, primitive.rotationTextureInfo, textureIndex++); + + location = this.shader.getUniformLocation(primitive.scaleTextureInfo.samplerName); + this.webGl.setTexture(location, state.gltf, primitive.scaleTextureInfo, textureIndex++); + + location = this.shader.getUniformLocation(primitive.opacityTextureInfo.samplerName); + this.webGl.setTexture(location, state.gltf, primitive.opacityTextureInfo, textureIndex++); + + location = this.shader.getUniformLocation(primitive.shArray.samplerName); + this.webGl.setTexture(location, state.gltf, primitive.shArray, textureIndex++); + + this.webGl.context.bindBuffer(this.webGl.context.ARRAY_BUFFER, this.splatVBO); + location = this.shader.getAttributeLocation("a_position"); + this.webGl.context.enableVertexAttribArray(location); + this.webGl.context.vertexAttribPointer(location, 2, GL.FLOAT, false, 0, 0); + this.webGl.context.bindBuffer(this.webGl.context.ARRAY_BUFFER, null); + + this.webGl.context.bindBuffer(this.webGl.context.ARRAY_BUFFER, this.currentSortBuffer); + this.webGl.context.bufferData( + this.webGl.context.ARRAY_BUFFER, + primitive.sortOrder, + this.webGl.context.DYNAMIC_DRAW + ); + location = this.shader.getAttributeLocation("a_instance_sort_index"); + this.webGl.context.enableVertexAttribArray(location); + this.webGl.context.vertexAttribIPointer(location, 1, GL.UNSIGNED_INT, 0, 0); + this.webGl.context.vertexAttribDivisor(location, 1); + this.webGl.context.bindBuffer(this.webGl.context.ARRAY_BUFFER, null); + + this.webGl.context.drawArraysInstanced(GL.TRIANGLE_STRIP, 0, 4, primitive.sortOrder.length); + + this.webGl.context.vertexAttribDivisor(location, 0); + this.webGl.context.disableVertexAttribArray(location); + this.webGl.context.depthMask(true); + } + + tonemapPass(state, aspectOffsetX, aspectOffsetY, aspectWidth, aspectHeight) { + const gl = this.webGl.context; + + const fragDefines = []; + this.pushFragParameterDefines(fragDefines, state); + + const fragHash = this.shaderCache.selectShader("tonemap_main.frag", fragDefines); + const vertHash = this.shaderCache.selectShader("fullscreen.vert", []); + if (!fragHash || !vertHash) return; + const shader = this.shaderCache.getShaderProgram(fragHash, vertHash); + if (!shader) return; + + gl.bindFramebuffer(gl.FRAMEBUFFER, null); + gl.viewport(aspectOffsetX, aspectOffsetY, aspectWidth, aspectHeight); + + gl.useProgram(shader.program); + shader.updateUniform("u_Exposure", state.renderingParameters.exposure, false); + + const hdrLoc = shader.getUniformLocation("u_MainSampler"); + gl.activeTexture(GL.TEXTURE0); + gl.bindTexture( + gl.TEXTURE_2D, + this._floatingPointFramebuffer ? this.mainTextureHDR : this.mainTexture + ); + gl.uniform1i(hdrLoc, 0); + + const tonemapLoc = shader.getUniformLocation("u_TonemapSampler"); + gl.activeTexture(GL.TEXTURE1); + gl.bindTexture(GL.TEXTURE_2D, this.mainTonemapTexture); + gl.uniform1i(tonemapLoc, 1); + + gl.bindBuffer(gl.ARRAY_BUFFER, this.splatVBO); + const posLoc = shader.getAttributeLocation("a_position"); + gl.enableVertexAttribArray(posLoc); + gl.vertexAttribPointer(posLoc, 2, GL.FLOAT, false, 0, 0); + gl.bindBuffer(gl.ARRAY_BUFFER, null); + + gl.disable(gl.DEPTH_TEST); + gl.disable(gl.BLEND); + gl.drawArrays(GL.TRIANGLE_STRIP, 0, 4); + + gl.enable(gl.DEPTH_TEST); + gl.disableVertexAttribArray(posLoc); + } + + // Composites the splat isolation framebuffer onto the currently-bound mainFramebuffer. + splatCompositePass(state, isLinear) { + const gl = this.webGl.context; + + const fragDefines = []; + this.pushFragParameterDefines(fragDefines, state); + if (isLinear) { + fragDefines.push("LINEAR_OUTPUT 1"); + } + const fragHash = this.shaderCache.selectShader("splat_composite.frag", fragDefines); + const vertHash = this.shaderCache.selectShader("fullscreen.vert", []); + if (!fragHash || !vertHash) return; + const shader = this.shaderCache.getShaderProgram(fragHash, vertHash); + if (!shader) return; + + gl.useProgram(shader.program); + + const splatTex = this._floatingPointFramebuffer + ? this.splatColorTextureHDR + : this.splatColorTexture; + const samplerLoc = shader.getUniformLocation("u_SplatSampler"); + gl.activeTexture(GL.TEXTURE0); + gl.bindTexture(gl.TEXTURE_2D, splatTex ?? this.splatColorTexture); + gl.uniform1i(samplerLoc, 0); + + shader.updateUniform("u_Exposure", state.renderingParameters.exposure, false); + + // Bind the fullscreen quad VBO (a_position attribute expected by fullscreen.vert). + gl.bindBuffer(gl.ARRAY_BUFFER, this.splatVBO); + const posLoc = shader.getAttributeLocation("a_position"); + gl.enableVertexAttribArray(posLoc); + gl.vertexAttribPointer(posLoc, 2, GL.FLOAT, false, 0, 0); + gl.bindBuffer(gl.ARRAY_BUFFER, null); + + gl.enable(gl.BLEND); + this.webGl.context.blendFuncSeparate( + GL.SRC_ALPHA, + GL.ONE_MINUS_SRC_ALPHA, + GL.ONE, + GL.ONE_MINUS_SRC_ALPHA + ); + this.webGl.context.blendEquation(GL.FUNC_ADD); + + // No depth test or depth write – this is a pure compositing pass. + gl.disable(gl.DEPTH_TEST); + gl.depthMask(false); + + gl.drawArrays(GL.TRIANGLE_STRIP, 0, 4); + + // Restore state. + gl.disable(gl.BLEND); + gl.enable(gl.DEPTH_TEST); + gl.depthMask(true); + gl.disableVertexAttribArray(posLoc); } // vertices with given material @@ -718,6 +1296,10 @@ class gltfRenderer { { fragDefines.push("LINEAR_OUTPUT 1"); } + if (renderpassConfiguration.transmission) + { + fragDefines.push("TRANSMISSION_PASS 1"); + } // POINTS, LINES, LINE_LOOP, LINE_STRIP if (primitive.mode < 4) { @@ -794,10 +1376,12 @@ class gltfRenderer { this.webGl.context.enable(GL.BLEND); this.webGl.context.blendFuncSeparate(GL.SRC_ALPHA, GL.ONE_MINUS_SRC_ALPHA, GL.ONE, GL.ONE_MINUS_SRC_ALPHA); this.webGl.context.blendEquation(GL.FUNC_ADD); + this.webGl.context.depthMask(false); } else { this.webGl.context.disable(GL.BLEND); + this.webGl.context.depthMask(true); } @@ -1146,6 +1730,7 @@ class gltfRenderer { break; case GltfState.ToneMaps.NONE: default: + fragDefines.push("TONEMAP_NONE 1"); break; } diff --git a/source/Renderer/shader.js b/source/Renderer/shader.js index f33ff1a7..958f957f 100644 --- a/source/Renderer/shader.js +++ b/source/Renderer/shader.js @@ -154,6 +154,18 @@ class gltfShader { } break; } + case GL.UNSIGNED_INT: { + if ( + Array.isArray(value) || + value instanceof Uint32Array || + value instanceof Int32Array + ) { + this.gl.context.uniform1uiv(uniform.loc, value); + } else { + this.gl.context.uniform1ui(uniform.loc, value); + } + break; + } case GL.INT_VEC2: this.gl.context.uniform2iv(uniform.loc, value); break; diff --git a/source/Renderer/shaders/cubemap.frag b/source/Renderer/shaders/cubemap.frag index 352da8b9..6bb4d6ad 100644 --- a/source/Renderer/shaders/cubemap.frag +++ b/source/Renderer/shaders/cubemap.frag @@ -9,7 +9,8 @@ uniform float u_EnvBlurNormalized; uniform int u_MipCount; uniform samplerCube u_GGXEnvSampler; -out vec4 FragColor; +layout(location = 0) out vec4 FragColor; +layout(location = 1) out uint toneMapFlag; in vec3 v_TexCoords; @@ -18,10 +19,6 @@ void main() vec4 color = textureLod(u_GGXEnvSampler, v_TexCoords, u_EnvBlurNormalized * float(u_MipCount - 1)); color.rgb *= u_EnvIntensity; color.a = 1.0; - -#ifdef LINEAR_OUTPUT FragColor = color.rgba; -#else - FragColor = vec4(toneMap(color.rgb), color.a); -#endif + toneMapFlag = 2u; } diff --git a/source/Renderer/shaders/fullscreen.vert b/source/Renderer/shaders/fullscreen.vert new file mode 100644 index 00000000..fd902190 --- /dev/null +++ b/source/Renderer/shaders/fullscreen.vert @@ -0,0 +1,13 @@ +precision highp float; + +// Fullscreen quad vertex shader. +// Expects the splatVBO quad: positions in [-1, 1] x [-1, 1]. +in vec2 a_position; + +out vec2 v_uv; + +void main() +{ + v_uv = a_position * 0.5 + 0.5; + gl_Position = vec4(a_position, 0.0, 1.0); +} diff --git a/source/Renderer/shaders/pbr.frag b/source/Renderer/shaders/pbr.frag index 2777a58a..690b9b2c 100644 --- a/source/Renderer/shaders/pbr.frag +++ b/source/Renderer/shaders/pbr.frag @@ -30,7 +30,8 @@ precision highp float; #endif -out vec4 g_finalColor; +layout(location = 0) out vec4 g_finalColor; +layout(location = 1) out uint toneMapFlag; void main() @@ -48,11 +49,13 @@ void main() } baseColor.a = 1.0; #endif -#ifdef LINEAR_OUTPUT +#ifdef TRANSMISSION_PASS // If used for transmission, we need to invert exposure and tone mapping, so the original color is computed in the general render pass g_finalColor = vec4(toneMapInverse(baseColor.rgb), baseColor.a); + toneMapFlag = 2u; #else - g_finalColor = vec4(linearTosRGB(baseColor.rgb), baseColor.a); + g_finalColor = baseColor; + toneMapFlag = 1u; #endif // PBR Flow. This else goes all the way to the end of the file #else @@ -397,14 +400,14 @@ void main() baseColor.a = 1.0; #endif -#ifdef LINEAR_OUTPUT + g_finalColor = vec4(color.rgb, baseColor.a); -#else - g_finalColor = vec4(toneMap(color), baseColor.a); -#endif + toneMapFlag = 2u; + #else // In case of missing data for a debug view, render a checkerboard. + toneMapFlag = 0u; g_finalColor = vec4(1.0); { float frequency = 0.02; diff --git a/source/Renderer/shaders/specular_glossiness.frag b/source/Renderer/shaders/specular_glossiness.frag index 5ba6572a..45038d7a 100644 --- a/source/Renderer/shaders/specular_glossiness.frag +++ b/source/Renderer/shaders/specular_glossiness.frag @@ -10,7 +10,8 @@ precision highp float; #include -out vec4 g_finalColor; +layout(location = 0) out vec4 g_finalColor; +layout(location = 1) out uint toneMapFlag; // Specular Glossiness uniform vec3 u_SpecularFactor; @@ -142,9 +143,7 @@ void main() #endif -#ifdef MATERIAL_UNLIT - color = baseColor.rgb; -#elif defined(NOT_TRIANGLE) && !defined(HAS_NORMAL_VEC3) +#if defined(NOT_TRIANGLE) && !defined(HAS_NORMAL_VEC3) //Points or Lines with no NORMAL attribute SHOULD be rendered without lighting and instead use the sum of the base color value and the emissive value. color = f_emissive + baseColor.rgb; #else @@ -162,14 +161,12 @@ void main() baseColor.a = 1.0; #endif -#ifdef LINEAR_OUTPUT g_finalColor = vec4(color.rgb, baseColor.a); -#else - g_finalColor = vec4(toneMap(color), baseColor.a); -#endif + toneMapFlag = 2u; #else // In case of missing data for a debug view, render a checkerboard. + toneMapFlag = 0u; g_finalColor = vec4(1.0); { float frequency = 0.02; diff --git a/source/Renderer/shaders/splat.frag b/source/Renderer/shaders/splat.frag new file mode 100644 index 00000000..76bf5a1a --- /dev/null +++ b/source/Renderer/shaders/splat.frag @@ -0,0 +1,23 @@ +precision highp float; + +in vec4 v_color; // (r, g, b, opacity) +in vec2 v_uv; // pixel-space offset from splat centre +in vec3 v_conic; // (A, B, C) coefficients of the conic section defining the splat's shape + +layout(location = 0) out vec4 g_finalColor; + +void main() { + + // Equation 4 + float exponent = -0.5 * (v_conic.x * v_uv.x * v_uv.x + v_conic.z * v_uv.y * v_uv.y) - v_conic.y * v_uv.x * v_uv.y; + + if (exponent > 0.0) { + discard; + } + + float alpha = min(0.99, exp(exponent) * v_color.a); // opacity modulated by Gaussian falloff + if (alpha < 1.0 / 255.0) { + discard; + } + g_finalColor = max(vec4(v_color.rgb * alpha, alpha), vec4(0.0)); // premultiplied-alpha output +} diff --git a/source/Renderer/shaders/splat.vert b/source/Renderer/shaders/splat.vert new file mode 100644 index 00000000..0a47514c --- /dev/null +++ b/source/Renderer/shaders/splat.vert @@ -0,0 +1,297 @@ +// Original CUDA kernel https://github.com/graphdeco-inria/diff-gaussian-rasterization/blob/9c5c2028f6fbee2be239bc4c9421ff894fe4fbe0/cuda_rasterizer/forward.cu + +precision highp float; +precision highp int; + +in vec2 a_position; +in uint a_instance_sort_index; + +out vec4 v_color; +out vec2 v_uv; +out vec3 v_conic; + +uniform mat4 u_ProjectionMatrix; +uniform mat4 u_ViewMatrix; +uniform mat4 u_ModelMatrix; +uniform mat3 u_ModelRotationInv; +uniform uint u_TextureWidth; +uniform ivec2 u_FramebufferSize; +uniform vec2 u_FocalLength; +uniform vec3 u_Camera; + +#ifdef POSITION_IS_INTEGER +uniform mediump isampler2D u_POSITIONSampler; +#elif defined(POSITION_IS_UINTEGER) +uniform mediump usampler2D u_POSITIONSampler; +#else +uniform sampler2D u_POSITIONSampler; +#endif + +#ifdef ROTATION_IS_INTEGER +uniform mediump isampler2D u_ROTATIONSampler; +#else +uniform sampler2D u_ROTATIONSampler; +#endif + +#ifdef SCALE_IS_UINTEGER +uniform mediump usampler2D u_SCALESampler; +#else +uniform sampler2D u_SCALESampler; +#endif + +#ifdef OPACITY_IS_UINTEGER +uniform mediump usampler2D u_OPACITYSampler; +#else +uniform sampler2D u_OPACITYSampler; +#endif + +uniform mediump sampler2DArray u_SHCoefficientsSampler; + +#define SH_C0 0.28209479177387814 + +// Gaussian radius multiplier: how many standard deviations to extend the screen-space quad. +// exp(-0.5 * SPLAT_SIGMA²) gives the minimum opacity at the quad boundary. +// 3.33 → 1/255 (mathematically exact), 3.0 → 1/90 (23% smaller quads, ~40% fewer fragments). +#define SPLAT_SIGMA 3.0 + +#ifdef HAS_GAUSSIAN_SPLATTING_DEGREE_1 +#define SH_C1_0 -0.4886025119029199 +#define SH_C1_1 0.4886025119029199 +#define SH_C1_2 -0.4886025119029199 + + +#ifdef HAS_GAUSSIAN_SPLATTING_DEGREE_2 +#define SH_C2_0 1.0925484305920792 +#define SH_C2_1 -1.0925484305920792 +#define SH_C2_2 0.31539156525252005 +#define SH_C2_3 -1.0925484305920792 +#define SH_C2_4 0.5462742152960396 + + +#ifdef HAS_GAUSSIAN_SPLATTING_DEGREE_3 +#define SH_C3_0 -0.5900435899266435 +#define SH_C3_1 2.890611442640554 +#define SH_C3_2 -0.4570457994644658 +#define SH_C3_3 0.3731763325901154 +#define SH_C3_4 -0.4570457994644658 +#define SH_C3_5 1.445305721320277 +#define SH_C3_6 -0.5900435899266435 +#endif + +#endif +#endif + +mat3 computeC(vec3 scale, vec4 rotation) +{ + + float qx = rotation.x; + float qy = rotation.y; + float qz = rotation.z; + float qw = rotation.w; + float sx = scale.x; + float sy = scale.y; + float sz = scale.z; + + mat3 C = mat3( + sx * (1.0 - 2.0*(qy*qy + qz*qz)), sx * (2.0*(qx*qy + qw*qz)), sx * (2.0*(qx*qz - qw*qy)), + sy * (2.0*(qx*qy - qw*qz)), sy * (1.0 - 2.0*(qx*qx + qz*qz)), sy * (2.0*(qy*qz + qw*qx)), + sz * (2.0*(qx*qz + qw*qy)), sz * (2.0*(qy*qz - qw*qx)), sz * (1.0 - 2.0*(qx*qx + qy*qy)) + ); + + return C; +} + +vec3 computeCameraCovariance(mat3 world_covariance, vec3 view_splat_center) +{ + float x = view_splat_center.x; + float y = view_splat_center.y; + float z = view_splat_center.z; + + mat3 J = mat3( + u_FocalLength.x / z, 0.0, -(u_FocalLength.x * x) / (z * z), + 0.0, u_FocalLength.y / z, -(u_FocalLength.y * y) / (z * z), + 0, 0, 0 + ); + + mat3 W = transpose(mat3(u_ViewMatrix)); + + // Equation 5 + mat3 T = W * J; + + mat3 cov = transpose(T) * transpose(world_covariance) * T; + + // Low-pass filter (EWA Splatting) + cov[0][0] += 0.3; + cov[1][1] += 0.3; + + return vec3(cov[0][0], cov[0][1], cov[1][1]); +} + +void main() +{ + vec3 splat_center; + ivec2 texelCoord = ivec2(a_instance_sort_index % u_TextureWidth, a_instance_sort_index / u_TextureWidth); +#if defined(POSITION_IS_INTEGER) || defined(POSITION_IS_UINTEGER) + splat_center = vec3(texelFetch(u_POSITIONSampler, texelCoord, 0).xyz); +#ifdef POSITION_NEEDS_NORMALIZATION +#ifdef POSITION_IS_UINTEGER + splat_center = splat_center / 65535.0; // normalize unsigned short +#else + splat_center = max(splat_center / 32767.0, -1.0); // normalize signed short +#endif +#endif +#else + splat_center = texelFetch(u_POSITIONSampler, texelCoord, 0).xyz; +#endif + + vec4 rotation; +#if defined(ROTATION_IS_INTEGER) + rotation = vec4(texelFetch(u_ROTATIONSampler, texelCoord, 0).xyzw); +#ifdef ROTATION_NEEDS_NORMALIZATION + rotation = max(rotation / 32767.0, -1.0); // normalize signed short +#endif +#else + rotation = texelFetch(u_ROTATIONSampler, texelCoord, 0); +#endif + + vec3 scale; +#if defined(SCALE_IS_UINTEGER) + scale = vec3(texelFetch(u_SCALESampler, texelCoord, 0).xyz); +#ifdef SCALE_NEEDS_NORMALIZATION + scale = scale / 65535.0; // normalize unsigned short +#endif +#else + scale = texelFetch(u_SCALESampler, texelCoord, 0).xyz; +#endif + + float opacity; +#if defined(OPACITY_IS_UINTEGER) + opacity = float(texelFetch(u_OPACITYSampler, texelCoord, 0).x); +#ifdef OPACITY_NEEDS_NORMALIZATION + opacity = opacity / 65535.0; // normalize unsigned short +#endif +#else + opacity = texelFetch(u_OPACITYSampler, texelCoord, 0).x; +#endif + + splat_center = (u_ModelMatrix * vec4(splat_center, 1.0)).xyz; + + vec4 view_splat_center = u_ViewMatrix * vec4(splat_center, 1.0); + vec4 clip_splat_center = u_ProjectionMatrix * view_splat_center; + clip_splat_center /= clip_splat_center.w; // perspective division + + mat3 C = computeC(scale, rotation); + mat3 M = mat3(u_ModelMatrix); + mat3 world_covariance = M * C * transpose(C) * transpose(M); + vec3 camera_covariance = computeCameraCovariance(world_covariance, view_splat_center.xyz); + + float a = camera_covariance.x; // Variance x + float b = camera_covariance.y; // Covariance xy + float c = camera_covariance.z; // Variance y + + float det = (a * c - b * b); + if (det == 0.0) { + gl_Position = vec4(0.0); + return; + } + + float det_inv = 1.0 / det; + // Calculate the inverse of the covariance matrix + v_conic = vec3(c * det_inv, -b * det_inv, a * det_inv); + + // pow(e, pow(-SPLAT_SIGMA, 2) * -0.5) gives the boundary opacity. + // sqrt(a) and sqrt(c) are the standard deviations in x and y, so multiplying + // by SPLAT_SIGMA gives the quad half-size in pixels at that opacity threshold. + vec2 quad_pixel_size = vec2(SPLAT_SIGMA * sqrt(a), SPLAT_SIGMA * sqrt(c)); // screen space half quad height and width + vec2 quad_ndc_size = quad_pixel_size / vec2(u_FramebufferSize) * 2.0; // in ndc space + clip_splat_center.xy = clip_splat_center.xy + a_position * quad_ndc_size; + + // Discard splats whose projected size exceeds half the screen — + // they are almost certainly behind or very close to the camera and + // would cause extreme overdraw with negligible visual contribution. + float min_screen = float(min(u_FramebufferSize.x, u_FramebufferSize.y)); + float max_quad_size = max(quad_pixel_size.x, quad_pixel_size.y); + if (max_quad_size > min_screen * 0.5) + { + gl_Position = vec4(0.0, 0.0, 2.0, 1.0); + return; + } + v_uv = a_position * quad_pixel_size; + gl_Position = clip_splat_center; + + // Degree 0 + vec3 sh0 = texelFetch(u_SHCoefficientsSampler, ivec3(texelCoord, 0), 0).rgb; + +#ifdef HAS_GAUSSIAN_SPLATTING_DEGREE_1 + // Degree 1 + vec3 sh1_0 = texelFetch(u_SHCoefficientsSampler, ivec3(texelCoord, 1), 0).rgb; + vec3 sh1_1 = texelFetch(u_SHCoefficientsSampler, ivec3(texelCoord, 2), 0).rgb; + vec3 sh1_2 = texelFetch(u_SHCoefficientsSampler, ivec3(texelCoord, 3), 0).rgb; + +#ifdef HAS_GAUSSIAN_SPLATTING_DEGREE_2 + // Degree 2 + vec3 sh2_0 = texelFetch(u_SHCoefficientsSampler, ivec3(texelCoord, 4), 0).rgb; + vec3 sh2_1 = texelFetch(u_SHCoefficientsSampler, ivec3(texelCoord, 5), 0).rgb; + vec3 sh2_2 = texelFetch(u_SHCoefficientsSampler, ivec3(texelCoord, 6), 0).rgb; + vec3 sh2_3 = texelFetch(u_SHCoefficientsSampler, ivec3(texelCoord, 7), 0).rgb; + vec3 sh2_4 = texelFetch(u_SHCoefficientsSampler, ivec3(texelCoord, 8), 0).rgb; + +#ifdef HAS_GAUSSIAN_SPLATTING_DEGREE_3 + // Degree 3 + vec3 sh3_0 = texelFetch(u_SHCoefficientsSampler, ivec3(texelCoord, 9), 0).rgb; + vec3 sh3_1 = texelFetch(u_SHCoefficientsSampler, ivec3(texelCoord, 10), 0).rgb; + vec3 sh3_2 = texelFetch(u_SHCoefficientsSampler, ivec3(texelCoord, 11), 0).rgb; + vec3 sh3_3 = texelFetch(u_SHCoefficientsSampler, ivec3(texelCoord, 12), 0).rgb; + vec3 sh3_4 = texelFetch(u_SHCoefficientsSampler, ivec3(texelCoord, 13), 0).rgb; + vec3 sh3_5 = texelFetch(u_SHCoefficientsSampler, ivec3(texelCoord, 14), 0).rgb; + vec3 sh3_6 = texelFetch(u_SHCoefficientsSampler, ivec3(texelCoord, 15), 0).rgb; +#endif +#endif +#endif + + vec3 color = sh0 * SH_C0; + +#ifdef HAS_GAUSSIAN_SPLATTING_DEGREE_1 + + vec3 view_dir_world = normalize(splat_center - u_Camera); + vec3 view_dir = u_ModelRotationInv * view_dir_world; // local-frame direction + + float x = view_dir.x; + float y = view_dir.y; + float z = view_dir.z; + + color += SH_C1_0 * y * sh1_0 + + SH_C1_1 * z * sh1_1 + + SH_C1_2 * x * sh1_2; + +#ifdef HAS_GAUSSIAN_SPLATTING_DEGREE_2 + float xx = x * x; + float yy = y * y; + float zz = z * z; + float xy = x * y; + float yz = y * z; + float xz = x * z; + + color += SH_C2_0 * xy * sh2_0 + + SH_C2_1 * yz * sh2_1 + + SH_C2_2 * (2.0 * zz - xx - yy) * sh2_2 + + SH_C2_3 * xz * sh2_3 + + SH_C2_4 * (xx - yy) * sh2_4; + +#ifdef HAS_GAUSSIAN_SPLATTING_DEGREE_3 + color += SH_C3_0 * y * (3.0 * xx - yy) * sh3_0 + + SH_C3_1 * xy * z * sh3_1 + + SH_C3_2 * y * (4.0 * zz - xx - yy) * sh3_2 + + SH_C3_3 * z * (2.0 * zz - 3.0 * xx - 3.0 * yy) * sh3_3 + + SH_C3_4 * x * (4.0 * zz - xx - yy) * sh3_4 + + SH_C3_5 * z * (xx - yy) * sh3_5 + + SH_C3_6 * x * (xx - 3.0 * yy) * sh3_6; +#endif + +#endif +#endif + + color += 0.5; + + v_color = vec4(color, opacity); +} diff --git a/source/Renderer/shaders/splat_composite.frag b/source/Renderer/shaders/splat_composite.frag new file mode 100644 index 00000000..ba9d0050 --- /dev/null +++ b/source/Renderer/shaders/splat_composite.frag @@ -0,0 +1,38 @@ +precision highp float; + +#include + +// Composites the splat isolation framebuffer onto the main framebuffer. +// Outputs: +// location 0 – color (blended onto main) +// location 1 – toneMapFlag 1u = "linearToSRGB" for the tonemap_main pass + +uniform sampler2D u_SplatSampler; + +in vec2 v_uv; + +layout(location = 0) out vec4 g_finalColor; +layout(location = 1) out uint g_toneMapFlag; + +void main() { + vec4 splat = texture(u_SplatSampler, v_uv); + + // Discard fully-transparent fragments so the toneMapFlag texture is not + // overwritten for pixels that were never touched by a splat. +#ifndef TONEMAP_NONE + vec3 multiplied = splat.rgb * splat.a; + float maxComponent = max(max(multiplied.r, multiplied.g), multiplied.b); + if (maxComponent < 1.0 / 255.0) { + discard; + } +#endif + splat.rgb = clamp(splat.rgb, vec3(0.0), vec3(1.0)); +#ifndef LINEAR_OUTPUT + splat.rgb = sRGBToLinear(splat.rgb); +#endif + g_finalColor = splat; + // The composited content is linear; tonemap_main.frag should apply + // linearToSRGB (flag value 0) in its final step. + g_toneMapFlag = 1u; + +} diff --git a/source/Renderer/shaders/tonemap_main.frag b/source/Renderer/shaders/tonemap_main.frag new file mode 100644 index 00000000..c242c4b6 --- /dev/null +++ b/source/Renderer/shaders/tonemap_main.frag @@ -0,0 +1,30 @@ +precision highp float; + +// Final tonemapping pass. +// Applies tonemapping for mesh pixels, linearToSRGB for splat pixels, +// and mixes between them based on splatCoverage. +// +// The TONEMAP_* define is injected at compile time to match the selected tone mapper, +// mirroring what pbr.frag does. The u_Exposure uniform is provided by tonemapping.glsl. + +#include + +uniform sampler2D u_MainSampler; +uniform lowp usampler2D u_TonemapSampler; + +in vec2 v_uv; + +out vec4 g_finalColor; + +void main() +{ + uint splatCoverage = texture(u_TonemapSampler, v_uv).r; + vec4 color = texture(u_MainSampler, v_uv); + + if (splatCoverage == 2u) { + color.rgb = toneMap(color.rgb); + } else if (splatCoverage == 1u) { + color.rgb = linearTosRGB(color.rgb); + } + g_finalColor = vec4(color.rgb, color.a); +} diff --git a/source/gltf/gltf.js b/source/gltf/gltf.js index 8daf7545..1d13d776 100644 --- a/source/gltf/gltf.js +++ b/source/gltf/gltf.js @@ -21,6 +21,7 @@ import { gltfVariant } from "./variant.js"; const allowedExtensions = [ "KHR_animation_pointer", "KHR_draco_mesh_compression", + "KHR_gaussian_splatting", "KHR_lights_image_based", "KHR_lights_punctual", "KHR_materials_anisotropy", diff --git a/source/gltf/primitive.js b/source/gltf/primitive.js index 2a0eec1b..f6a18702 100644 --- a/source/gltf/primitive.js +++ b/source/gltf/primitive.js @@ -34,10 +34,256 @@ class gltfPrimitive extends GltfObject { this.hasTexcoord = false; this.hasColor = false; + // Gaussian Splatting + this.hasDegree1 = false; + this.hasDegree2 = false; + this.hasDegree3 = false; + this.linear = true; + this.positionTextureInfo = undefined; + this.rotationTextureInfo = undefined; + this.scaleTextureInfo = undefined; + this.opacityTextureInfo = undefined; + this.sphericalHarmonicsTextureInfo = undefined; + this.sortOrder = undefined; + this.splatTextureWidth = undefined; + + // Worker-based sort state + this.sortWorker = undefined; + this.sortWorkerReady = false; + this.sortPending = false; + + // Store the view matrix if sort is currently running + this.queuedViewMatrix = undefined; + // Compare against a stored value to detect whether a redraw is needed. + this.lastSortViewMatrix = undefined; + // The primitive centroid is used for depth sorting. this.centroid = undefined; } + //Currently only support types relevant for gaussian splatting + // If an alignment of 4 bytes is not possible for a given format, the correct alignment is returned. + _getInternalTextureFormat(componentType, componentCount, normalized = false) { + if (componentType === GL.FLOAT) { + switch (componentCount) { + case 1: // OPACITIY + return { internalFormat: GL.R32F, format: GL.RED }; + case 3: // POSITION, SCALE, Spherical Harmonics + return { internalFormat: GL.RGB32F, format: GL.RGB }; + case 4: // ROTATION + return { internalFormat: GL.RGBA32F, format: GL.RGBA }; + } + } + if (componentType === GL.UNSIGNED_BYTE) { + switch (componentCount) { + case 1: // OPACITY + return { internalFormat: GL.R8, format: GL.RED, alignment: 1 }; // Opacity is always normalized + case 3: // POSITION, SCALE + return { + internalFormat: normalized ? GL.RGB8 : GL.RGB8UI, + format: normalized ? GL.RGB : GL.RGB_INTEGER, + alignment: 1 + }; + } + } + if (componentType === GL.BYTE) { + switch (componentCount) { + case 3: // POSITION + return { + internalFormat: normalized ? GL.RGB8_SNORM : GL.RGB8I, + format: normalized ? GL.RGB : GL.RGB_INTEGER, + alignment: 1 + }; + case 4: // ROTATION + return { internalFormat: GL.RGBA8_SNORM, format: GL.RGBA }; // Rotation is always normalized + } + } + + // There is no normalized format for unsigned short and short. Needs to be resolved in the shader + if (componentType === GL.UNSIGNED_SHORT) { + switch (componentCount) { + case 1: // OPACITY + return { internalFormat: GL.R16UI, format: GL.RED_INTEGER, alignment: 2 }; // Opacity is always normalized + case 3: // POSITION, SCALE + return { internalFormat: GL.RGB16UI, format: GL.RGB_INTEGER, alignment: 2 }; + } + } + if (componentType === GL.SHORT) { + switch (componentCount) { + case 3: // POSITION + return { internalFormat: GL.RGB16I, format: GL.RGB_INTEGER, alignment: 2 }; + case 4: // ROTATION + return { internalFormat: GL.RGBA16I, format: GL.RGBA_INTEGER }; + } + } + console.error( + "Unsupported texture format for componentType:", + componentType, + "and componentCount:", + componentCount + ); + return undefined; + } + + _createDataTexture(gltf, webGlContext, attributeName, accessor) { + if (accessor === undefined) { + return undefined; + } + let texture = webGlContext.createTexture(); + webGlContext.bindTexture(webGlContext.TEXTURE_2D, texture); + // Set texture format and upload data. + const componentType = accessor.componentType; + const componentCount = accessor.getComponentCount(accessor.type); + const formats = this._getInternalTextureFormat( + componentType, + componentCount, + accessor.normalized + ); + if ( + formats.format === GL.RED_INTEGER || + formats.format === GL.RGB_INTEGER || + formats.format === GL.RGBA_INTEGER + ) { + if (componentType === GL.UNSIGNED_BYTE || componentType === GL.UNSIGNED_SHORT) { + this.defines.push(`${attributeName}_IS_UINTEGER 1`); + } else { + this.defines.push(`${attributeName}_IS_INTEGER 1`); + } + if (accessor.normalized) { + // Only shorts do not support normalized integer formats, so we need to normalize them manually in the shader. + this.defines.push(`${attributeName}_NEEDS_NORMALIZATION 1`); + } + } else { + this.defines.push(`${attributeName}_IS_FLOAT 1`); + } + const size = Math.ceil(Math.sqrt(accessor.count)); + const data = accessor.getDeinterlacedView(gltf); + const paddedData = new data.constructor(size * size * componentCount); + paddedData.set(data); + + webGlContext.pixelStorei(webGlContext.UNPACK_ALIGNMENT, formats.alignment ?? 4); + webGlContext.texImage2D( + webGlContext.TEXTURE_2D, + 0, //level + formats.internalFormat, + size, + size, + 0, //border + formats.format, + accessor.componentType, + paddedData + ); + webGlContext.pixelStorei(webGlContext.UNPACK_ALIGNMENT, 4); // restore default + // Ensure mipmapping is disabled and the sampler is configured correctly. + webGlContext.texParameteri(GL.TEXTURE_2D, GL.TEXTURE_WRAP_S, GL.CLAMP_TO_EDGE); + webGlContext.texParameteri(GL.TEXTURE_2D, GL.TEXTURE_WRAP_T, GL.CLAMP_TO_EDGE); + webGlContext.texParameteri(GL.TEXTURE_2D, GL.TEXTURE_MIN_FILTER, GL.NEAREST); + webGlContext.texParameteri(GL.TEXTURE_2D, GL.TEXTURE_MAG_FILTER, GL.NEAREST); + + // Now we add the morph target texture as a gltf texture info resource, so that + // we can just call webGl.setTexture(..., gltfTextureInfo, ...) in the renderer. + const image = new gltfImage( + undefined, // uri + GL.TEXTURE_2D, // type + 0, // mip level + undefined, // buffer view + undefined, // name + ImageMimeType.GLTEXTURE, // mimeType + texture // image + ); + gltf.images.push(image); + + gltf.samplers.push( + new gltfSampler(GL.NEAREST, GL.NEAREST, GL.CLAMP_TO_EDGE, GL.CLAMP_TO_EDGE, undefined) + ); + + const tex = new gltfTexture( + gltf.samplers.length - 1, + gltf.images.length - 1, + GL.TEXTURE_2D + ); + // The webgl texture is already initialized -> this flag informs + // webgl.setTexture about this. + tex.initialized = true; + + gltf.textures.push(tex); + + const textureInfo = new gltfTextureInfo(gltf.textures.length - 1, 0, true); + textureInfo.samplerName = `u_${attributeName}Sampler`; //TODO Check if this works + textureInfo.generateMips = false; + return textureInfo; + } + + _createDataTextureArray( + gltf, + webGlContext, + data, + width, + textureCount, + componentCount, + samplerName + ) { + let texture = webGlContext.createTexture(); + webGlContext.bindTexture(webGlContext.TEXTURE_2D_ARRAY, texture); + // Set texture format and upload data. + // Use 16-bit half-precision floats: half the bandwidth of RGB32F with negligible + // quality loss for SH coefficients. WebGL2 accepts Float32Array with FLOAT type + // when the internal format is a 16F format — the driver converts on upload. + let internalFormat = componentCount === 4 ? webGlContext.RGBA16F : webGlContext.RGB16F; + let format = componentCount === 4 ? webGlContext.RGBA : webGlContext.RGB; + let type = webGlContext.FLOAT; + webGlContext.texImage3D( + webGlContext.TEXTURE_2D_ARRAY, + 0, //level + internalFormat, + width, + width, + textureCount, //Layer count + 0, //border + format, + type, + data + ); + // Ensure mipmapping is disabled and the sampler is configured correctly. + webGlContext.texParameteri(GL.TEXTURE_2D_ARRAY, GL.TEXTURE_WRAP_S, GL.CLAMP_TO_EDGE); + webGlContext.texParameteri(GL.TEXTURE_2D_ARRAY, GL.TEXTURE_WRAP_T, GL.CLAMP_TO_EDGE); + webGlContext.texParameteri(GL.TEXTURE_2D_ARRAY, GL.TEXTURE_MIN_FILTER, GL.NEAREST); + webGlContext.texParameteri(GL.TEXTURE_2D_ARRAY, GL.TEXTURE_MAG_FILTER, GL.NEAREST); + + // Now we add the morph target texture as a gltf texture info resource, so that + // we can just call webGl.setTexture(..., gltfTextureInfo, ...) in the renderer. + const image = new gltfImage( + undefined, // uri + GL.TEXTURE_2D_ARRAY, // type + 0, // mip level + undefined, // buffer view + undefined, // name + ImageMimeType.GLTEXTURE, // mimeType + texture // image + ); + gltf.images.push(image); + + gltf.samplers.push( + new gltfSampler(GL.NEAREST, GL.NEAREST, GL.CLAMP_TO_EDGE, GL.CLAMP_TO_EDGE, undefined) + ); + + const tex = new gltfTexture( + gltf.samplers.length - 1, + gltf.images.length - 1, + GL.TEXTURE_2D_ARRAY + ); + // The webgl texture is already initialized -> this flag informs + // webgl.setTexture about this. + tex.initialized = true; + + gltf.textures.push(tex); + + const textureInfo = new gltfTextureInfo(gltf.textures.length - 1, 0, true); + textureInfo.generateMips = false; + textureInfo.samplerName = samplerName; + return textureInfo; + } + initGl(gltf, webGlContext) { // Use the default glTF material. if (this.material === undefined) { @@ -92,6 +338,7 @@ class gltfPrimitive extends GltfObject { break; } let knownAttribute = true; + let isTexture = false; switch (attribute) { case "POSITION": this.skip = false; @@ -123,11 +370,43 @@ class gltfPrimitive extends GltfObject { case "WEIGHTS_1": this.hasWeights = true; break; + case "KHR_gaussian_splatting:ROTATION": + case "KHR_gaussian_splatting:SCALE": + case "KHR_gaussian_splatting:OPACITY": + isTexture = true; + break; + case "KHR_gaussian_splatting:SH_DEGREE_0_COEF_0": + isTexture = true; + break; + case "KHR_gaussian_splatting:SH_DEGREE_1_COEF_0": + case "KHR_gaussian_splatting:SH_DEGREE_1_COEF_1": + case "KHR_gaussian_splatting:SH_DEGREE_1_COEF_2": + isTexture = true; + this.hasDegree1 = true; + break; + case "KHR_gaussian_splatting:SH_DEGREE_2_COEF_0": + case "KHR_gaussian_splatting:SH_DEGREE_2_COEF_1": + case "KHR_gaussian_splatting:SH_DEGREE_2_COEF_2": + case "KHR_gaussian_splatting:SH_DEGREE_2_COEF_3": + case "KHR_gaussian_splatting:SH_DEGREE_2_COEF_4": + isTexture = true; + this.hasDegree2 = true; + break; + case "KHR_gaussian_splatting:SH_DEGREE_3_COEF_0": + case "KHR_gaussian_splatting:SH_DEGREE_3_COEF_1": + case "KHR_gaussian_splatting:SH_DEGREE_3_COEF_2": + case "KHR_gaussian_splatting:SH_DEGREE_3_COEF_3": + case "KHR_gaussian_splatting:SH_DEGREE_3_COEF_4": + case "KHR_gaussian_splatting:SH_DEGREE_3_COEF_5": + case "KHR_gaussian_splatting:SH_DEGREE_3_COEF_6": + isTexture = true; + this.hasDegree3 = true; + break; default: knownAttribute = false; console.log("Unknown attribute: " + attribute); } - if (knownAttribute) { + if (knownAttribute && !isTexture) { const idx = this.attributes[attribute]; this.glAttributes.push({ attribute: attribute, @@ -138,6 +417,157 @@ class gltfPrimitive extends GltfObject { } } + // Gaussian Splatting + if (this.extensions?.KHR_gaussian_splatting !== undefined) { + const extension = this.extensions.KHR_gaussian_splatting; + if (extension.kernel !== "ellipse") { + console.warn( + `Unsupported kernel type for Gaussian Splatting: ${extension.kernel}. Using ellipse kernel.` + ); + } + if (extension.colorSpace === "srgb_rec709_display") { + this.linear = false; + } else if (extension.colorSpace !== "lin_rec709_display") { + console.warn( + `Unsupported color space for Gaussian Splatting: ${extension.colorSpace}. Using linear Rec. 709 display.` + ); + } + if (extension.projection !== undefined && extension.projection !== "perspective") { + console.warn( + `Unsupported projection type for Gaussian Splatting: ${extension.projection}. Using perspective projection.` + ); + } + if ( + extension.sortingMethod !== undefined && + extension.sortingMethod !== "cameraDistance" + ) { + console.warn( + `Unsupported sorting method for Gaussian Splatting: ${extension.sortingMethod}. Using camera distance.` + ); + } + if (this.hasDegree2 && !this.hasDegree1) { + console.warn( + "Degree 2 SH Coefficients provided without Degree 1. This is not supported and Degree 2 coefficients will be ignored." + ); + this.hasDegree2 = false; + } + if (this.hasDegree3 && (!this.hasDegree1 || !this.hasDegree2)) { + console.warn( + "Degree 3 SH Coefficients provided without Degree 1 or Degree 2. This is not supported and Degree 3 coefficients will be ignored." + ); + this.hasDegree3 = false; + } + + const max2DTextureSize = Math.pow(webGlContext.getParameter(GL.MAX_TEXTURE_SIZE), 2); + const vertexCount = + gltf.accessors[this.attributes["KHR_gaussian_splatting:SH_DEGREE_0_COEF_0"]].count; + this.initSortWorker(gltf, vertexCount); + this.splatTextureWidth = Math.ceil(Math.sqrt(vertexCount)); + const singleTextureSize = Math.pow(this.splatTextureWidth, 2); + + if (vertexCount > max2DTextureSize) { + console.error("Vertex count exceeds maximum 2D texture size."); + this.skip = true; + return; + } + + this.positionTextureInfo = this._createDataTexture( + gltf, + webGlContext, + "POSITION", + gltf.accessors[this.attributes.POSITION] + ); + + this.rotationTextureInfo = this._createDataTexture( + gltf, + webGlContext, + "ROTATION", + gltf.accessors[this.attributes["KHR_gaussian_splatting:ROTATION"]] + ); + + if (this.rotationTextureInfo === undefined) { + console.error( + "Rotation attribute is required for Gaussian Splatting but not found. Skipping primitive." + ); + this.skip = true; + return; + } + + this.scaleTextureInfo = this._createDataTexture( + gltf, + webGlContext, + "SCALE", + gltf.accessors[this.attributes["KHR_gaussian_splatting:SCALE"]] + ); + + if (this.scaleTextureInfo === undefined) { + console.error( + "Scale attribute is required for Gaussian Splatting but not found. Skipping primitive." + ); + this.skip = true; + return; + } + + this.opacityTextureInfo = this._createDataTexture( + gltf, + webGlContext, + "OPACITY", + gltf.accessors[this.attributes["KHR_gaussian_splatting:OPACITY"]] + ); + + if (this.opacityTextureInfo === undefined) { + console.error( + "Opacity attribute is required for Gaussian Splatting but not found. Skipping primitive." + ); + this.skip = true; + return; + } + + if (this.attributes["KHR_gaussian_splatting:SH_DEGREE_0_COEF_0"] === undefined) { + console.error( + "SH Degree 0 Coefficient 0 attribute is required for Gaussian Splatting but not found. Skipping primitive." + ); + this.skip = true; + return; + } + + let textureAtlasSize = 1; + if (this.hasDegree1) { + this.defines.push("HAS_GAUSSIAN_SPLATTING_DEGREE_1 1"); + textureAtlasSize += 3; + if (this.hasDegree2) { + this.defines.push("HAS_GAUSSIAN_SPLATTING_DEGREE_2 1"); + textureAtlasSize += 5; + if (this.hasDegree3) { + this.defines.push("HAS_GAUSSIAN_SPLATTING_DEGREE_3 1"); + textureAtlasSize += 7; + } + } + } + const shData = new Float32Array(singleTextureSize * textureAtlasSize * 3); + const shAttributes = Object.keys(this.attributes) + .filter((attr) => attr.startsWith("KHR_gaussian_splatting:SH_DEGREE_")) + .sort(); + for (let i = 0; i < shAttributes.length; i++) { + const accessor = gltf.accessors[this.attributes[shAttributes[i]]]; + const data = accessor.getDeinterlacedView(gltf); + shData.set(data, i * singleTextureSize * 3); + } + + this.shArray = this._createDataTextureArray( + gltf, + webGlContext, + shData, + this.splatTextureWidth, + textureAtlasSize, + 3, + "u_SHCoefficientsSampler" + ); + + this.sortOrder = new Uint32Array(vertexCount); + for (let i = 0; i < vertexCount; i++) this.sortOrder[i] = i; + } + // MORPH TARGETS if (this.targets !== undefined && this.targets.length > 0) { const max2DTextureSize = Math.pow(webGlContext.getParameter(GL.MAX_TEXTURE_SIZE), 2); @@ -237,83 +667,15 @@ class gltfPrimitive extends GltfObject { } // Add the morph target texture. - // We have to create a WebGL2 texture as the format of the - // morph target texture has to be explicitly specified - // (gltf image would assume uint8). - let texture = webGlContext.createTexture(); - webGlContext.bindTexture(webGlContext.TEXTURE_2D_ARRAY, texture); - // Set texture format and upload data. - let internalFormat = webGlContext.RGBA32F; - let format = webGlContext.RGBA; - let type = webGlContext.FLOAT; - let data = morphTargetTextureArray; - webGlContext.texImage3D( - webGlContext.TEXTURE_2D_ARRAY, - 0, //level - internalFormat, - width, + this.morphTargetTextureInfo = this._createDataTextureArray( + gltf, + webGlContext, + morphTargetTextureArray, width, - targetCount * attributes.length, //Layer count - 0, //border - format, - type, - data - ); - // Ensure mipmapping is disabled and the sampler is configured correctly. - webGlContext.texParameteri( - GL.TEXTURE_2D_ARRAY, - GL.TEXTURE_WRAP_S, - GL.CLAMP_TO_EDGE - ); - webGlContext.texParameteri( - GL.TEXTURE_2D_ARRAY, - GL.TEXTURE_WRAP_T, - GL.CLAMP_TO_EDGE - ); - webGlContext.texParameteri(GL.TEXTURE_2D_ARRAY, GL.TEXTURE_MIN_FILTER, GL.NEAREST); - webGlContext.texParameteri(GL.TEXTURE_2D_ARRAY, GL.TEXTURE_MAG_FILTER, GL.NEAREST); - - // Now we add the morph target texture as a gltf texture info resource, so that - // we can just call webGl.setTexture(..., gltfTextureInfo, ...) in the renderer. - const morphTargetImage = new gltfImage( - undefined, // uri - GL.TEXTURE_2D_ARRAY, // type - 0, // mip level - undefined, // buffer view - undefined, // name - ImageMimeType.GLTEXTURE, // mimeType - texture // image + targetCount * attributes.length, + 4, + "u_MorphTargetsSampler" ); - gltf.images.push(morphTargetImage); - - gltf.samplers.push( - new gltfSampler( - GL.NEAREST, - GL.NEAREST, - GL.CLAMP_TO_EDGE, - GL.CLAMP_TO_EDGE, - undefined - ) - ); - - const morphTargetTexture = new gltfTexture( - gltf.samplers.length - 1, - gltf.images.length - 1, - GL.TEXTURE_2D_ARRAY - ); - // The webgl texture is already initialized -> this flag informs - // webgl.setTexture about this. - morphTargetTexture.initialized = true; - - gltf.textures.push(morphTargetTexture); - - this.morphTargetTextureInfo = new gltfTextureInfo( - gltf.textures.length - 1, - 0, - true - ); - this.morphTargetTextureInfo.samplerName = "u_MorphTargetsSampler"; - this.morphTargetTextureInfo.generateMips = false; } else { console.warn("Mesh of Morph targets too big. Cannot apply morphing."); } @@ -322,6 +684,119 @@ class gltfPrimitive extends GltfObject { this.computeCentroid(gltf); } + /** + * Spawn a mkkellogg WASM sort worker and hand it the splat centre positions. + * Called once during initGl for Gaussian Splatting primitives. + * @param {object} gltf + * @param {number} vertexCount + */ + initSortWorker(gltf, vertexCount) { + try { + this.sortWorker = new Worker( + new URL("./libs/mkkellogg-sort.worker.js", import.meta.url), //URL needs to be relative to rollup build + { type: "module" } + ); + } catch (err) { + console.warn("Failed to spawn sort worker:", err); + return; + } + + // Build stride-4 Float32Array (x, y, z, 1) from the POSITION accessor. + const posAccessor = gltf.accessors[this.attributes.POSITION]; + const rawPositions = posAccessor.getDeinterlacedView(gltf); + const posOp = new Float32Array(vertexCount * 4); + for (let i = 0; i < vertexCount; i++) { + posOp[i * 4 + 0] = rawPositions[i * 3 + 0]; + posOp[i * 4 + 1] = rawPositions[i * 3 + 1]; + posOp[i * 4 + 2] = rawPositions[i * 3 + 2]; + posOp[i * 4 + 3] = 1.0; + } + + this.sortWorker.onmessage = (e) => { + const { type } = e.data; + if (type === "ready") { + this.sortWorkerReady = true; + this.sortPending = false; + // Fire any sort that was queued while the worker was initialising. + if (this.queuedViewMatrix !== undefined) { + // Skip if the view matrix hasn't changed since the last dispatched sort. + if (this.lastSortViewMatrix !== undefined) { + let same = true; + for (let i = 0; i < 16; i++) { + if (this.lastSortViewMatrix[i] !== this.queuedViewMatrix[i]) { + same = false; + break; + } + } + if (same) return; + } + this.sortPending = true; + this.sortWorker.postMessage({ + type: "sort", + viewMatrix: this.queuedViewMatrix + }); + this.queuedViewMatrix = undefined; + } + } else if (type === "sorted") { + this.sortOrder = e.data.indices; + this.sortPending = false; + } else if (type === "error") { + console.error("Sort worker error:", e.data.message); + this.sortPending = false; + } + }; + + this.sortWorker.onerror = (err) => { + console.error( + "Sort worker uncaught error:", + err.message, + err.filename, + "line", + err.lineno, + err + ); + this.sortPending = false; + }; + + // Transfer positions buffer to the worker (zero-copy). + this.sortWorker.postMessage({ type: "init", posOp: posOp, splatCount: vertexCount }, [ + posOp.buffer + ]); + this.sortPending = true; + } + + /** + * Request an asynchronous back-to-front sort of the splat indices. + * Safe to call every frame — the request is dropped while a previous sort + * is still in flight. + * @param {Float32Array} modelViewMatrix Column-major 4×4 model-view matrix (view * world). + */ + requestSort(modelViewMatrix) { + if (this.sortWorker === undefined) { + return; + } + if (!this.sortWorkerReady || this.sortPending) { + // Worker is busy — keep the latest matrix so it sorts immediately once ready + this.queuedViewMatrix = new Float32Array(modelViewMatrix); + return; + } + // Skip if the view matrix hasn't changed since the last dispatched sort. + if (this.lastSortViewMatrix !== undefined) { + let same = true; + for (let i = 0; i < 16; i++) { + if (this.lastSortViewMatrix[i] !== modelViewMatrix[i]) { + same = false; + break; + } + } + if (same) return; + } + const vm = new Float32Array(modelViewMatrix); + this.lastSortViewMatrix = vm; + this.sortPending = true; + this.sortWorker.postMessage({ type: "sort", viewMatrix: vm }); + } + computeCentroid(gltf) { const positionsAccessor = gltf.accessors[this.attributes.POSITION]; const positions = positionsAccessor.getNormalizedDeinterlacedView(gltf); diff --git a/source/libs/mkkellogg-sort.worker.js b/source/libs/mkkellogg-sort.worker.js new file mode 100644 index 00000000..f98d9ccc --- /dev/null +++ b/source/libs/mkkellogg-sort.worker.js @@ -0,0 +1,164 @@ +/** + * mkkellogg-sort.worker.js + * + * Web Worker that runs the mkkellogg counting-sort WASM extracted from + * @mkkellogg/gaussian-splats-3d. + * + * Exported function (from sorter.cpp): + * sortIndexes( + * indexes*, // i32 – input/output index array + * centers*, // i32 – float positions stride-4 (x,y,z,w) + * precomputed*, // i32 – pass 0 (unused when usePrecomputedDistances=false) + * mappedDistances*, // i32 – scratch int array, size=renderCount + * frequencies*, // i32 – scratch uint array, size=distanceMapRange + * modelViewProj*, // i32 – view matrix (4×4 column-major float) + * indexesOut*, // i32 – output sorted index array + * sceneIndexes*, // i32 – pass 0 (unused when dynamicMode=false) + * transforms*, // i32 – pass 0 (unused when dynamicMode=false) + * distanceMapRange, // i32 – histogram bin count (32768) + * sortCount, // i32 – splats to sort (= splatCount) + * renderCount, // i32 – rendered splat count (= splatCount) + * splatCount, // i32 – total splat count + * usePrecomputedDistances, // i32 – 0 (false) + * useIntegerSort, // i32 – 0 (false → float path) + * dynamicMode // i32 – 0 (false → single scene) + * ) + * + * Messages in: + * { type:'init', posOp: Float32Array, splatCount: number } + * { type:'sort', viewMatrix: Float32Array[16] } + * + * Messages out: + * { type:'ready' } + * { type:'sorted', indices: Uint32Array } (buffer transferred) + * { type:'error', message: string } + */ + +// Resolve the WASM URL relative to this worker file, compatible with both +// Rollup (import.meta.url is the worker script URL at runtime) and dev servers. +const wasmUrl = new URL('./sorter_no_simd_non_shared.wasm', import.meta.url).href; + +// ── Constants ────────────────────────────────────────────────────────────────── + +const DIST_MAP_RANGE = 32768; +const HEADER_PAGES = 1; // reserve 1 WASM page (64 KiB) for internal use +const PAGE_BYTES = 65536; + +// ── Module-level state ──────────────────────────────────────────────────────── + +let sortFn = null; // wasm export function +let memBuf = null; // WebAssembly.Memory + +// Byte offsets (set during init) +let pIndexes = 0, pCenters = 0, pMappedDist = 0, + pFrequencies = 0, pMVP = 0, pIndexesOut = 0; +let N = 0; + +// ── Helpers ─────────────────────────────────────────────────────────────────── + +/** Align a byte offset up to 16-byte boundary. */ +const align16 = (b) => (b + 15) & ~15; + +// ── Initialisation ──────────────────────────────────────────────────────────── + +async function init({ posOp, splatCount }) { + N = splatCount; + + // Layout each buffer (byte offsets in WebAssembly.Memory) + let off = HEADER_PAGES * PAGE_BYTES; + + pIndexes = off; off = align16(off + N * 4); // Uint32[N] + pCenters = off; off = align16(off + N * 4 * 4); // Float32[N*4] + pMappedDist = off; off = align16(off + N * 4); // Int32[N] + pFrequencies = off; off = align16(off + DIST_MAP_RANGE * 4); // Uint32[32768] + pMVP = off; off = align16(off + 16 * 4); // Float32[16] + pIndexesOut = off; off = align16(off + N * 4); // Uint32[N] + + const totalBytes = off; + const pages = Math.ceil(totalBytes / PAGE_BYTES) + 2; // +2 for safety + + memBuf = new WebAssembly.Memory({ initial: pages, maximum: pages + 4 }); + + let instance; + try { + const response = await fetch(wasmUrl); + ({ instance } = await WebAssembly.instantiateStreaming(response, { + env: { memory: memBuf }, + })); + } catch (err) { + // Fallback: fetch as ArrayBuffer (needed when streaming isn't available). + const ab = await (await fetch(wasmUrl)).arrayBuffer(); + ({ instance } = await WebAssembly.instantiate(ab, { + env: { memory: memBuf }, + })); + } + + sortFn = instance.exports.sortIndexes; + if (typeof instance.exports.__wasm_call_ctors === 'function') { + instance.exports.__wasm_call_ctors(); + } + + // Populate centres (stride-4): Float32 (x, y, z, opacity) — raw positions. + const fv = new Float32Array(memBuf.buffer); + fv.set(new Float32Array(posOp, 0, N * 4), pCenters >> 2); + + // Initialise indexes 0..N-1 + const uv = new Uint32Array(memBuf.buffer); + for (let i = 0; i < N; i++) uv[(pIndexes >> 2) + i] = i; + + self.postMessage({ type: 'ready' }); +} + +// ── Sort ────────────────────────────────────────────────────────────────────── + +function sort({ viewMatrix }) { + const uv = new Uint32Array(memBuf.buffer); + const fv = new Float32Array(memBuf.buffer); + + // Reset frequencies histogram to zero + uv.fill(0, pFrequencies >> 2, (pFrequencies >> 2) + DIST_MAP_RANGE); + + // Write view matrix into WASM memory, with the Z row negated. + // The standard lookAt puts -forward in row 2, so visible splats have negative + // distances (near ≈ −ε, far ≈ −∞). The WASM counting sort maps low values to + // low bins and places them at the END of indexesOut, giving front-to-back order. + // By negating row 2 (columns 2, 6, 10, 14) we flip all distances to positive + // (near ≈ +ε, far ≈ +∞), so the sort naturally produces back-to-front order + // with no O(N) reversal pass needed. + fv.set(viewMatrix, pMVP >> 2); + const base = pMVP >> 2; + fv[base + 2] = -fv[base + 2]; + fv[base + 6] = -fv[base + 6]; + fv[base + 10] = -fv[base + 10]; + fv[base + 14] = -fv[base + 14]; + + // Call the counting sort + // sortIndexes(indexes, centers, precomputed, mappedDist, frequencies, + // mvp, indexesOut, sceneIdxs, transforms, + // distMapRange, sortCount, renderCount, splatCount, + // usePrecomputed, useIntegerSort, dynamicMode) + sortFn( + pIndexes, pCenters, 0, + pMappedDist, pFrequencies, pMVP, + pIndexesOut, 0, 0, + DIST_MAP_RANGE, N, N, N, + 0, 0, 0, + ); + + const out = new Uint32Array(uv.subarray(pIndexesOut >> 2, (pIndexesOut >> 2) + N)); + self.postMessage({ type: 'sorted', indices: out }, [out.buffer]); +} + +// ── Message dispatcher ──────────────────────────────────────────────────────── + +self.onmessage = async (e) => { + try { + if (e.data.type === 'init') { + await init(e.data); + } else if (e.data.type === 'sort') { + sort(e.data); + } + } catch (err) { + self.postMessage({ type: 'error', message: String(err) }); + } +}; diff --git a/source/libs/sorter_no_simd_non_shared.wasm b/source/libs/sorter_no_simd_non_shared.wasm new file mode 100644 index 00000000..e96cbdf0 Binary files /dev/null and b/source/libs/sorter_no_simd_non_shared.wasm differ diff --git a/thirdparty/GaussianSplats3D/BUILD.md b/thirdparty/GaussianSplats3D/BUILD.md new file mode 100644 index 00000000..fd68a429 --- /dev/null +++ b/thirdparty/GaussianSplats3D/BUILD.md @@ -0,0 +1,4 @@ +To build `sorter_no_simd_non_shared.wasm` + +1. [Install emscripten SDK](https://emscripten.org/docs/getting_started/downloads.html#) +2. Run `compile_wasm_no_simd_non_shared.sh` or copy its content into your shell to build the WASM file diff --git a/thirdparty/GaussianSplats3D/LICENSE b/thirdparty/GaussianSplats3D/LICENSE new file mode 100644 index 00000000..6370ba2d --- /dev/null +++ b/thirdparty/GaussianSplats3D/LICENSE @@ -0,0 +1,21 @@ +The MIT License (MIT) + +Copyright (c) 2023 Mark Kellogg + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. \ No newline at end of file diff --git a/thirdparty/GaussianSplats3D/compile_wasm_no_simd_non_shared.sh b/thirdparty/GaussianSplats3D/compile_wasm_no_simd_non_shared.sh new file mode 100644 index 00000000..cdabf646 --- /dev/null +++ b/thirdparty/GaussianSplats3D/compile_wasm_no_simd_non_shared.sh @@ -0,0 +1 @@ +em++ -std=c++11 sorter_no_simd.cpp -Os -s WASM=1 -s SIDE_MODULE=2 -o sorter_no_simd_non_shared.wasm -s IMPORTED_MEMORY=1 \ No newline at end of file diff --git a/thirdparty/GaussianSplats3D/sorter_no_simd.cpp b/thirdparty/GaussianSplats3D/sorter_no_simd.cpp new file mode 100644 index 00000000..c07fc71a --- /dev/null +++ b/thirdparty/GaussianSplats3D/sorter_no_simd.cpp @@ -0,0 +1,156 @@ +#include +#include + +#ifdef __cplusplus +#define EXTERN extern "C" +#else +#define EXTERN +#endif + +#define computeMatMul4x4ThirdRow(a, b, out) \ + out[0] = a[2] * b[0] + a[6] * b[1] + a[10] * b[2] + a[14] * b[3]; \ + out[1] = a[2] * b[4] + a[6] * b[5] + a[10] * b[6] + a[14] * b[7]; \ + out[2] = a[2] * b[8] + a[6] * b[9] + a[10] * b[10] + a[14] * b[11]; \ + out[3] = a[2] * b[12] + a[6] * b[13] + a[10] * b[14] + a[14] * b[15]; + +EXTERN EMSCRIPTEN_KEEPALIVE void sortIndexes(unsigned int* indexes, void* centers, void* precomputedDistances, + int* mappedDistances, unsigned int * frequencies, float* modelViewProj, + unsigned int* indexesOut, unsigned int* sceneIndexes, float* transforms, + unsigned int distanceMapRange, unsigned int sortCount, unsigned int renderCount, + unsigned int splatCount, bool usePrecomputedDistances, bool useIntegerSort, + bool dynamicMode) { + + int maxDistance = -2147483640; + int minDistance = 2147483640; + + float fMVPTRow3[4]; + int iMVPTRow3[4]; + unsigned int sortStart = renderCount - sortCount; + if (useIntegerSort) { + int* intCenters = (int*)centers; + if (usePrecomputedDistances) { + int* intPrecomputedDistances = (int*)precomputedDistances; + for (unsigned int i = sortStart; i < renderCount; i++) { + int distance = intPrecomputedDistances[indexes[i]]; + mappedDistances[i] = distance; + if (distance > maxDistance) maxDistance = distance; + if (distance < minDistance) minDistance = distance; + } + } else { + if (dynamicMode) { + int lastTransformIndex = -1; + for (unsigned int i = sortStart; i < renderCount; i++) { + unsigned int realIndex = indexes[i]; + unsigned int indexOffset = 4 * realIndex; + unsigned int sceneIndex = sceneIndexes[realIndex]; + if ((int)sceneIndex != lastTransformIndex) { + float* transform = &transforms[sceneIndex * 16]; + computeMatMul4x4ThirdRow(modelViewProj, transform, fMVPTRow3); + iMVPTRow3[0] = (int)(fMVPTRow3[0] * 1000.0); + iMVPTRow3[1] = (int)(fMVPTRow3[1] * 1000.0); + iMVPTRow3[2] = (int)(fMVPTRow3[2] * 1000.0); + iMVPTRow3[3] = (int)(fMVPTRow3[3] * 1000.0); + lastTransformIndex = (int)sceneIndex; + } + int distance = + (int)((iMVPTRow3[0] * intCenters[indexOffset] + + iMVPTRow3[1] * intCenters[indexOffset + 1] + + iMVPTRow3[2] * intCenters[indexOffset + 2] + + iMVPTRow3[3] * intCenters[indexOffset + 3])); + mappedDistances[i] = distance; + if (distance > maxDistance) maxDistance = distance; + if (distance < minDistance) minDistance = distance; + } + } else { + iMVPTRow3[0] = (int)(modelViewProj[2] * 1000.0); + iMVPTRow3[1] = (int)(modelViewProj[6] * 1000.0); + iMVPTRow3[2] = (int)(modelViewProj[10] * 1000.0); + iMVPTRow3[3] = 1; + for (unsigned int i = sortStart; i < renderCount; i++) { + unsigned int indexOffset = 4 * (unsigned int)indexes[i]; + int distance = + (int)((iMVPTRow3[0] * intCenters[indexOffset] + + iMVPTRow3[1] * intCenters[indexOffset + 1] + + iMVPTRow3[2] * intCenters[indexOffset + 2])); + mappedDistances[i] = distance; + if (distance > maxDistance) maxDistance = distance; + if (distance < minDistance) minDistance = distance; + } + } + } + } else { + float* floatCenters = (float*)centers; + if (usePrecomputedDistances) { + float* floatPrecomputedDistances = (float*)precomputedDistances; + for (unsigned int i = sortStart; i < renderCount; i++) { + int distance = (int)(floatPrecomputedDistances[indexes[i]] * 4096.0); + mappedDistances[i] = distance; + if (distance > maxDistance) maxDistance = distance; + if (distance < minDistance) minDistance = distance; + } + } else { + float* fMVP = (float*)modelViewProj; + float* floatTransforms = (float *)transforms; + + if (dynamicMode) { + int lastTransformIndex = -1; + for (unsigned int i = sortStart; i < renderCount; i++) { + unsigned int realIndex = indexes[i]; + unsigned int indexOffset = 4 * realIndex; + unsigned int sceneIndex = sceneIndexes[realIndex]; + if ((int)sceneIndex != lastTransformIndex) { + float* transform = &transforms[sceneIndex * 16]; + computeMatMul4x4ThirdRow(modelViewProj, transform, fMVPTRow3); + lastTransformIndex = (int)sceneIndex; + } + int distance = + (int)((fMVPTRow3[0] * floatCenters[indexOffset] + + fMVPTRow3[1] * floatCenters[indexOffset + 1] + + fMVPTRow3[2] * floatCenters[indexOffset + 2] + + fMVPTRow3[3] * floatCenters[indexOffset + 3]) * 4096.0); + mappedDistances[i] = distance; + if (distance > maxDistance) maxDistance = distance; + if (distance < minDistance) minDistance = distance; + } + } else { + for (unsigned int i = sortStart; i < renderCount; i++) { + unsigned int indexOffset = 4 * (unsigned int)indexes[i]; + int distance = + (int)((fMVP[2] * floatCenters[indexOffset] + + fMVP[6] * floatCenters[indexOffset + 1] + + fMVP[10] * floatCenters[indexOffset + 2]) * 4096.0); + mappedDistances[i] = distance; + if (distance > maxDistance) maxDistance = distance; + if (distance < minDistance) minDistance = distance; + } + } + } + } + + float distancesRange = (float)maxDistance - (float)minDistance; + float rangeMap = (float)(distanceMapRange - 1) / distancesRange; + + for (unsigned int i = sortStart; i < renderCount; i++) { + unsigned int frequenciesIndex = (int)((float)(mappedDistances[i] - minDistance) * rangeMap); + mappedDistances[i] = frequenciesIndex; + frequencies[frequenciesIndex] = frequencies[frequenciesIndex] + 1; + } + + unsigned int cumulativeFreq = frequencies[0]; + for (unsigned int i = 1; i < distanceMapRange; i++) { + unsigned int freq = frequencies[i]; + cumulativeFreq += freq; + frequencies[i] = cumulativeFreq; + } + + for (int i = (int)sortStart - 1; i >= 0; i--) { + indexesOut[i] = indexes[i]; + } + + for (int i = (int)renderCount - 1; i >= (int)sortStart; i--) { + unsigned int frequenciesIndex = mappedDistances[i]; + unsigned int freq = frequencies[frequenciesIndex]; + indexesOut[renderCount - freq] = indexes[i]; + frequencies[frequenciesIndex] = freq - 1; + } +} diff --git a/thirdparty/Thirdparty.md b/thirdparty/Thirdparty.md new file mode 100644 index 00000000..e7926356 --- /dev/null +++ b/thirdparty/Thirdparty.md @@ -0,0 +1,13 @@ +# Thirdparty + +This folder contains all third-party code that is not directly used in the distribution package. + +Libraries used directly are located in the `source/libs` folder or are included as npm packages. + +The following libraries are used: + +### GaussianSplats3D WASM sorter + +The `GaussianSplats3D` folder contains the C++ code to compile the WASM file `/source/libs/sorter_no_simd_non_shared.wasm`. + +The files were copied from the [original repository](https://github.com/mkkellogg/GaussianSplats3D) for reference.