ipu6-native-isp

A native userspace ISP for Intel IPU6 cameras on Linux. It reads the sensor's raw Bayer stream directly from the IPU6 ISYS and does the entire image signal pipeline itself — demosaic, white balance, colour correction, denoise, tone mapping — on the GPU, then presents a normal webcam on /dev/video0 for Zoom, Meet, Chrome, etc.

No icamerasrc. No CamHAL. No PSYS firmware. No per-sensor .aiqb tuning files.

Developed and daily-driven on a Dell Precision (Meteor Lake) with the OV02E10 sensor. If you have a different IPU6 sensor, see Hardware support.

Why

Intel's IPU6 cameras don't expose a plain V4L2 device. The vendor path is:

icamerasrc (CamHAL + PSYS firmware + per-sensor .aiqb) -> GStreamer -> v4l2loopback

That stack is fragile in practice: not-negotiated (-4) errors, CamHAL/PSYS probes wedging processes into uninterruptible D-state, missing .aiqb tuning files, activation-latency races (the LED comes on after the app already gave up), and kernel-version fragility that forces you to pin an old kernel.

The key finding behind this project: the IPU6 ISYS exposes the sensor's raw Bayer frames as a standard V4L2 capture node. If we do the ISP ourselves, we get a reliable path we fully own — and, with a real ISP, image quality that can match the vendor stack.

ov02e10 (SGRBG10)  ->  Intel IVSC CSI (passthrough)  ->  IPU6 CSI2  ->  ISYS /dev/video3x (raw Bayer)
      -> [this project: V4L2 capture -> GPU ISP -> YUYV]  ->  v4l2loopback /dev/video0  ->  apps

What it does

• Raw Bayer capture from the IPU6 ISYS node (auto-discovered by walking the media graph).
• Two-pass GPU ISP (wgpu compute shader on the iGPU):
  • black-level subtraction, demosaic, white balance
  • colour correction matrix (CCM), tone map / gamma
  • gain-adaptive spatial denoise (luma/chroma split — denoises chroma harder, which kills colour grain without softening detail)
  • motion-gated temporal denoise — static areas average across frames (low-light grain drops), moving areas snap to the current frame (no ghosting)
• Auto-exposure / auto-white-balance driving the sensor's V4L2 controls, with center-weighted metering (exposes for the subject, not a bright window behind you).
• On-demand activation: keeps /dev/video0 alive with a placeholder, switches to the real sensor when an app opens the camera, and back when it closes — so the camera LED is only on when actually in use.
• Optional local (motion-gated) HDR exposure fusion (built; off by default).

Sustained ~30 fps @ 1080p on Meteor Lake Xe graphics.

Hardware support

Component	Tested
Laptop	Dell Precision (Meteor Lake)
Sensor	OmniVision OV02E10 (SGRBG10)
iGPU	Intel Arc / Xe (Meteor Lake, 8086:7d55)
Kernel	7.0.0-30-generic (Ubuntu)
Userspace	Ubuntu 24.04

Other IPU6 sensors may work: capture and demosaic are generic, but the CCM and AE/AWB tuning are measured against the OV02E10. A different sensor will likely need its own CCM and black level. Sensor discovery walks the media graph, so the ISYS node and Bayer order are not hardcoded.

Note on kernels: this was developed against 7.0.0-30. Earlier IPU6 work required pinning 6.17.x; the native path is what lets you unpin. Behaviour on other kernels is untested — reports welcome.

Requirements

• Intel IPU6 camera exposing a raw ISYS capture node
• v4l2loopback kernel module (0.15.4+ recommended)
• Rust toolchain (stable) with a working wgpu / Vulkan or GL backend on the iGPU
• media-ctl / v4l2-ctl (v4l-utils) for inspection

You do not need icamerasrc, CamHAL, or the Intel camera userspace packages.

Build

cargo build --release
# binaries: target/release/native-capture  (the ISP + capture)
#           target/release/ipu6-camera-daemon  (on-demand activation wrapper)

Run

# Load the loopback device (creates /dev/video0)
sudo modprobe v4l2loopback video_nr=0 card_label="IPU6 Native" exclusive_caps=1

# Managed mode: capture raw -> GPU ISP -> AE/AWB -> /dev/video0
native-capture --gpu --ae --managed /dev/video0

Then pick "IPU6 Native" as your camera in Zoom / Meet / Chrome.

Useful flags:

Flag	Meaning
--gpu	use the GPU ISP (vs the CPU fallback)
--ae	enable auto-exposure / gain / AWB
--managed	full managed loop (discovery, AE, denoise, output)
--record <dir>	dump raw + processed frames for tuning
--hdr / --hdr-motion <t>	enable local motion-gated HDR
--temporal <bool>	motion-gated temporal denoise (default on)

Run as a service (on-demand LED)

An example user systemd unit and install.sh are included. The daemon keeps a placeholder stream on /dev/video0 and only powers the sensor when an app opens it.

How it works

See docs/NATIVE_ISP_DESIGN.md for the full design: the media-graph route, the phased build, the ISP stages, and the quality roadmap.

Source layout:

src/
  bin/native_capture.rs  managed capture loop, AE tick, flags
  gpu_isp.rs             wgpu pipeline (demosaic/HDR/temporal + YUYV pack)
  isp.wgsl               the ISP compute shaders (2-pass)
  aeawb.rs               auto-exposure, center-weighted metering, AWB gains
  discovery.rs           media-graph walk to find the sensor -> ISYS route
  v4l2_capture.rs        raw Bayer V4L2 mmap capture
  v4l2_output.rs         YUYV write to v4l2loopback
tools/
  isp_tune.py            pure-numpy auto-tuner (matches ISP params to a reference)
  hdr_replay.py          offline HDR fusion / tonemap experiments

Tuning

tools/isp_tune.py is a dependency-light (pure NumPy) coordinate-descent tuner. Capture a reference image and a matching raw frame, and it optimises CCM / denoise / gamma to minimise a perceptual loss (Lab ΔE + luma-histogram + a high-frequency noise term). This is how the shipped OV02E10 colour matrix and gain-adaptive denoise strengths were derived.

Status

Daily-driver stable in single-exposure managed mode on the reference hardware. Low-light denoise (spatial + temporal) is live-validated. HDR is built but off by default (the motion-gate is ghost-free, but single-exposure is preferred for typical calls).

Related

• ipu6-camera-daemon — the lighter-weight alternative that wraps icamerasrc and bridges it to V4L2 with on-demand activation. Use that if you just want the vendor stack working in Zoom; use this if you want to own the whole imaging path (or CamHAL keeps failing you).

License

MIT © Gregory Havenga. See LICENSE.

Acknowledgements

Built by reverse-engineering the IPU6 ISYS media graph on real hardware. Not affiliated with or endorsed by Intel or OmniVision.