Skip to content
nvpro-samplesPublic

About

Lightweight Vulkan denoiser (DLSS-style API, Slang shaders) based on Intel Open Image Denoise

Resources

Stars

10 stars

Watchers

0 watching

Forks

Latest commit

 

History

28 Commits

Folders and files

Repository files navigation

vk_oidn

Lightweight Vulkan denoiser based on Intel® Open Image Denoise, designed to be embedded in Vulkan renderers like DLSS: the application enables a few device features, creates a denoiser, and records cmdDenoise() into its own command buffers. The library never submits work or waits on the host.

  • Shaders written in Slang, using tensor cores through VK_KHR_cooperative_matrix (with a fallback for other GPUs).
  • Fully fused U-Net layers (convolution + bias + ReLU + pooling / upsampling / concatenation).
  • Inputs and outputs as buffers (device addresses) or storage image views.
  • Pre-packed network weights embedded in the library (no runtime file loading).

A path-traced room at 4 samples per pixel, the vk_oidn result from those 4 samples, and a 4096-sample reference

A room path traced in vk_gltf_renderer at 1280x720: 4 samples per pixel (left), denoised by vk_oidn in high quality with albedo and normal guides (middle), and a 4096-sample reference (right). Measured on this scene at 1000x800, the denoised image gets within 1% error of the converged image at about 50 samples per pixel, where the path tracer alone needs about 3,200.

Documentation: architecture (how it works, with diagrams) · validation and performance (all measurements and how they were made, issues found, experiments).

Status: version 0.x: the API may still change between minor versions. Releases with the prebuilt SDK packages are on the releases page. Example: the oidn_denoiser sample of vk_mini_samples (path-traced Cornell box, denoiser with albedo and normal guides).

Usage

The denoiser takes the noisy color (HDR or LDR) and, optionally, two guide images of the first visible surface: its albedo and its normal. They help the network keep textures and geometric edges sharp.

The noisy color at 1 sample per pixel, the albedo and normal guides, and the vk_oidn output

The oidn_denoiser sample of vk_mini_samples: color at 1 sample per pixel, albedo, normal (as 0.5 + 0.5 n), and the denoised output.

#include <vk_oidn/vk_oidn.h>

// 1. Before creating the VkDevice: enable what vk_oidn needs
vkoidn::getRequiredDeviceExtensions(instance, physicalDevice, vkGetInstanceProcAddr, extensions);
vkoidn::enableRequiredFeatures(features2, features11, features12, features13,
                               hasCoopMat ? &coopMatFeatures : nullptr);

// 2. Context (once per device)
vkoidn::ContextCreateInfo contextInfo{instance, physicalDevice, device,
                                      vkGetInstanceProcAddr, vkGetDeviceProcAddr,
                                      extensions.data(), uint32_t(extensions.size())};
std::unique_ptr<vkoidn::Context> context;
vkoidn::Context::create(contextInfo, context);

// 3. Denoiser (per resolution/configuration): records the weight upload into cmd
vkoidn::DenoiserCreateInfo info;
info.width = width;  info.height = height;
info.hdr = true;     info.albedo = true;  info.normal = true;
info.quality = vkoidn::Quality::eHigh;
std::unique_ptr<vkoidn::Denoiser> denoiser;
vkoidn::Denoiser::create(*context, cmd, info, denoiser);

// 4. Every frame: records the dispatches into the application's command buffer
vkoidn::DenoiseInputs inputs;
inputs.color  = {colorAddress,  vkoidn::Format::eFloat, 3, 16}; // e.g. RGBA32F, pixel stride 16
inputs.albedo = {albedoAddress, vkoidn::Format::eFloat, 3, 16};
inputs.normal = {normalAddress, vkoidn::Format::eFloat, 3, 16};
inputs.output = {outputAddress, vkoidn::Format::eFloat, 3, 16};
denoiser->cmdDenoise(cmd, inputs);
// barrier: compute shader write -> your use of the output

// Alternatively, storage image views (VK_IMAGE_LAYOUT_GENERAL, RGBA32F/RGBA16F, alpha preserved):
inputs = {};
inputs.colorView  = {colorView};
inputs.albedoView = {albedoView};
inputs.normalView = {normalView};
inputs.outputView = {outputView, VK_FORMAT_R16G16B16A16_SFLOAT}; // format: exact FP16 rounding
denoiser->cmdDenoise(cmd, inputs);

See tests/denoise_test.cpp for a complete example. The validation can be reproduced with tests/scripts (image generator and validation script, see the validation document). It is also a command-line tool: vk_oidn_denoise_test --hdr color.pfm --alb albedo.pfm --nrm normal.pfm -o out.pfm (-r ref.pfm compares with a reference, -n 50 measures the GPU time, --bench the per-image latency, --images uses storage image views instead of buffers).

Validation: on 22 configurations (HDR/LDR/sRGB, albedo/normal/clean aux, all qualities, half images, fixed input scale, tiled), the output matches the reference of the official Open Image Denoise 2.5.1 CPU device, as accurately as OIDN's CUDA device. The validation layer reports no errors.

GPU time per image on an NVIDIA RTX PRO 4500 Blackwell, compared with the official Open Image Denoise 2.5.1 CUDA device: same filter (hdr + albedo + normal), same FP32 images in device memory, same method (50 denoisings back to back, after a warm-up batch), median of 5 runs:

GPU time per image: vk_oidn is 1.5 to 1.8 times faster than Open Image Denoise 2.5.1's CUDA device at 1920x1080 and 3840x2160
Resolution Quality vk_oidn OIDN 2.5.1 CUDA Speedup
1280x720 high 2.48 ms 3.56 ms 1.43x
1920x1080 high 5.65 ms 8.56 ms 1.51x
3840x2160 high 24.39 ms 35.89 ms 1.47x
1280x720 fast 1.35 ms 2.46 ms 1.82x
1920x1080 fast 3.18 ms 5.80 ms 1.83x
3840x2160 fast 13.67 ms 24.29 ms 1.78x

The output is as accurate as the CUDA device's (average relative error against the OIDN CPU device: 0.00034 vs 0.00059 in high quality, 0.00114 vs 0.00108 in fast quality). The per-image latency (OIDN's oidnBenchmark metric: execute and wait for each image) gives the same picture, 1.42-1.47x in high quality and 1.69-1.73x in fast quality, and Nsight Systems, profiling the CUDA kernels and the Vulkan command buffers with the same tool, confirms the GPU times (1.48-1.54x and 1.82-1.86x, counting only the time the CUDA kernels run). The protocol, all results with their spread, the raw logs and the scripts to reproduce them are in docs/validation_and_performance.md and docs/benchmarks.

Adding vk_oidn to a project

Prebuilt SDK package (recommended): each release has a package per platform with the header, the static libraries (kernels and weights embedded) and the CMake package files. Nothing of vk_oidn is compiled; only the Vulkan headers are needed.

Package Contents
vk_oidn-<version>-windows-x64.zip vk_oidn.lib (Release) and vk_oidnd.lib (Debug), MSVC /MD runtime, Visual Studio 2022 or newer
vk_oidn-<version>-linux-x64.tar.gz libvk_oidn.a (Release, position independent), GCC 13 (Ubuntu 24.04)
include(FetchContent)
if(WIN32)
  set(VK_OIDN_PACKAGE vk_oidn-0.2.0-windows-x64.zip)
else()
  set(VK_OIDN_PACKAGE vk_oidn-0.2.0-linux-x64.tar.gz)
endif()
FetchContent_Declare(vk_oidn_sdk
  URL https://github.com/nvpro-samples/vk_oidn/releases/download/v0.2.0/${VK_OIDN_PACKAGE})
FetchContent_MakeAvailable(vk_oidn_sdk)
find_package(vk_oidn 0.2 CONFIG REQUIRED PATHS ${vk_oidn_sdk_SOURCE_DIR} NO_DEFAULT_PATH)
target_link_libraries(my_app PRIVATE vk_oidn::vk_oidn)

Applications built as RelWithDebInfo or MinSizeRel use the Release library. The checksums of the packages are in SHA256SUMS.txt of each release (use them with URL_HASH SHA256=...).

From the sources: FetchContent_Declare(vk_oidn GIT_REPOSITORY https://github.com/nvpro-samples/vk_oidn.git GIT_TAG v0.2.0) and FetchContent_MakeAvailable(vk_oidn) give the same vk_oidn::vk_oidn target, built with the application (requires Python 3 and slangc, see below).

Building

Build

Windows and Linux. Requires CMake 3.21+, a C++17 compiler, Python 3 and the Vulkan SDK (headers and slangc; tested with 1.4.363.0, Slang 2026.17). On Linux, the Vulkan loader library is also needed (libvulkan-dev).

cmake -S . -B build
cmake --build build --config Release
cmake --install build --config Release --prefix <sdk folder>   # creates the SDK package

The slangc of some Linux distribution packages is too old for the kernels: CMake then stops with a message. Set VULKAN_SDK to the LunarG SDK (source <sdk>/setup-env.sh), or VK_OIDN_SLANGC to a recent slangc from the Slang releases.

Every push is built on Windows and Linux by GitHub Actions, with warnings as errors. On Linux, the denoiser is also run on lavapipe (Mesa's CPU Vulkan driver) in 11 configurations, with buffers and with image views, and compared to the official Open Image Denoise CPU device, with the Khronos validation layer enabled (see docs/validation_and_performance.md). Each build also creates the SDK packages and links a small application against them (tests/package); for a version tag v*, the packages are attached to the release.

Shaders

All the kernels are written in Slang, in a single file, shaders/vk_oidn.slang. At build time, CMake compiles them to SPIR-V with slangc (from the Vulkan SDK) and embeds them in the library: an application ships no shader files.

Kernels Purpose
conv_coopmat_* 3x3 convolutions on the tensor cores (VK_KHR_cooperative_matrix, FP16 inputs), with the pooling and the concatenation of the skip connections fused in
conv_coopmat_up_* the convolutions that follow an upsampling, computed as four 2x2 convolutions on the low-resolution input (sub-pixel decomposition)
conv_simple convolutions without cooperative matrices, for the other GPUs
input_process, output_process, image_copy conversion between the application's images (buffers or storage images, FP32 or FP16) and the network's tensors
autoexposure_* automatic exposure of HDR inputs

The convolution kernels exist in variants: one or two blocks of output channels per workgroup (_nf1, _nf2), and FP32 or FP16 accumulation (_f16acc, used by the fast and balanced qualities). Every kernel is compiled twice: with FP16 float controls (round-to-nearest-even and denormal preservation, so that every GPU rounds the same way; Slang has no option for the rounding mode, which tools/spirv_float_controls.py adds to the SPIR-V), and without them (_nofc) for devices that do not support them.

The kernels are compiled with -O1 and use [unroll] hints rather than [ForceUnroll]: both were measurably faster (see docs/validation_and_performance.md). VK_OIDN_KERNEL_DIR=<build>/spv loads the compiled kernels from a directory instead of the embedded ones, to work on a kernel without relinking the application.

shaders/experimental/conv_cm2.comp is an experimental GLSL kernel for VK_NV_cooperative_matrix2, built when glslangValidator is available and used only with VK_OIDN_CM2=1; it is correct but slower than the Slang kernels.

Debugging environment variables

Variable Description
VK_OIDN_PROFILE=1 print per-layer GPU times when the denoiser is destroyed
VK_OIDN_COOPMAT=0/1 disable / force cooperative matrices (default: NVIDIA only)
VK_OIDN_CM2=1 experimental VK_NV_cooperative_matrix2 convolutions (slower, see the docs)
VK_OIDN_SUBPIXEL=0 disable the sub-pixel decomposition of the upsampling layers
VK_OIDN_CONV_VARIANT=mx2_nf1 force 1 (or 2) output blocks per workgroup in the convolutions (tuning)
VK_OIDN_KERNEL_DIR=<dir> load <dir>/<kernel>.spv instead of the embedded kernel when it exists (kernel development)

Weights

The weights/ folder contains the Open Image Denoise RT filter weights converted to a GPU-ready format by tools/pack_weights.py (layer graph + FP16 weight tiles in the layout of the convolution kernels). They are embedded in the library by default (VK_OIDN_EMBED_WEIGHTS, requires Python 3 at build time). See weights/README.md.

License

Apache License 2.0, see LICENSE and NOTICE. Derived from Intel Open Image Denoise; the network weights are from oidn-weights. vk_oidn is not affiliated with or endorsed by Intel Corporation.

About

Lightweight Vulkan denoiser (DLSS-style API, Slang shaders) based on Intel Open Image Denoise

Resources

Stars

10 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages