rockchip: gamescope

[felixjones]

Summary

Supersedes #2969, #2977, #2983, #3017

Adds the Rockchip RGA2 2D-engine stack (userspace + kernel driver) and brings
up Valve gamescope on top of it, so gamescope renders on-panel under either
GPU driver - the vendor ARM libMali blob or Mesa panfrost/PanVK - from the
same launch command. The work targets the Rockchip family (any SoC with an RGA
and a VOP2 display controller); the RK3576 (Anbernic RG Vita Pro) is the first
device enabled and tested.

Switch librga from the varphone fork to the maintained JeffyCN
linux-rga-multi source (im2d API), 1.3.1 to 1.10.5, for RGA2E multi-core
support. Applies to every Rockchip RGA part.

Mainline has no multi-RGA driver (only the old V4L2 rockchip-rga), so the
Rockchip BSP multi-RGA driver (RGA2/RGA2E/RGA3) is built out-of-tree to provide
/dev/rga. Source: rockchip-linux/kernel (mirror felixjones/linux-rga3-mirror);
patches port it to the mainline kernel API and add async fence mode - all
family-general. The RK3576-specific enablement is isolated to the device
tier: a devices/RK3576 device-tree patch adding the two RGA2-Pro nodes + their
IOMMUs, and one driver patch running those cores in IOMMU mode.
Developer guide: https://github.com/sz-jack-01/linux-rga/blob/rkr3.5/docs/Rockchip_Developer_Guide_RGA_EN.md

Gamescope changes are applied to Rockchip family as it introduces all the RGA hardware acceleration specific to those devices.

Copilot summary of technical changes below:

• ARM libMali Vulkan/EGL blob support (any libMali device; switched on absence
  of VK_EXT_physical_device_drm): drop the VK_EXT_robustness2 requirement
  (libMali/PanVK don't expose it, and gamescope only used it for null-descriptor
  binding, so bind inert blank views instead); CMA scanout (libMali can't export
  dma-bufs); the mali_buffer_sharing global (or GLES clients fail
  EGL_NOT_INITIALIZED); and join the pipeline-compile thread on shutdown (fixes a
  teardown SIGSEGV).
• panfrost/PanVK split render + display (any Mesa PanVK / split
  render-display SoC): scan out on the KMS display node with CMA output images
  when the render node isn't KMS-capable; bind the composite UBO with an explicit
  range (PanVK miscomputes VK_WHOLE_SIZE, otherwise a black screen).
• Rotated internal panel (any Rockchip VOP2, no hardware rotation):
  auto-enable the composite rotation shader (the portable but slower path) when the
  panel is mounted rotated; the orientation is read at runtime.
• Rockchip RGA composite & scanout offload (any RGA + VOP2 SoC): offload
  rotation, the nested->native upscale and multi-layer compositing to the RGA2 via
  librga; direct scanout is the default present path with async acquire/release
  fences so neither the present thread nor the VOP2 blocks; frames the RGA can't
  express fall back to the Vulkan composite. This is what removes the
  shader-rotation penalty above.
• Wayland subsurface video (driver/SoC-agnostic): advertise
  viewporter / subcompositor / single-pixel-buffer, composite a client's video
  subsurface on-panel, and route input to its toplevel. Built/tested with Steam
  Link, generic to any client using these protocols.

And finally: gamescope isn't built with anything yet - I'm going to add a Steamlink package that uses gamescope so look forward to that.

Testing

Tested exclusively on RG Vita Pro

This was tested and built around glmark2-es2-wayland, vkgears, and vkcube. I recommend installing those if you want to test this yourself.

To actually test this, release the EmulationStation/Sway session from the DRM so gamescope can own it:

systemctl stop essway.service sway.service

For panvk the following environment variables need to be set:

export PAN_I_WANT_A_BROKEN_VULKAN_DRIVER=1 MESA_VK_VERSION_OVERRIDE=1.3

libmali masks libGL.so.1 with /dev/null (days of debugging...) so that needs to be unmasked so gamescope can use it.

Here's a convenience script for running glmark2-es-wayland:

#!/bin/sh
export XDG_RUNTIME_DIR=/var/run/0-runtime-dir
mkdir -p "$XDG_RUNTIME_DIR"; chmod 700 "$XDG_RUNTIME_DIR"
export PAN_I_WANT_A_BROKEN_VULKAN_DRIVER=1 MESA_VK_VERSION_OVERRIDE=1.3
exec unshare -m --propagation private -- /bin/sh -c '
  for lib in /usr/lib/libGL.so.1 /usr/lib32/libGL.so.1; do
    real=$(readlink -f "$lib" 2>/dev/null)
    [ -n "$real" ] && umount "$real" 2>/dev/null
  done
  exec gamescope --expose-wayland -W 1920 -H 1080 -r 60 -- glmark2-es2-wayland
'

And yes --expose-wayland is required.

Additional video hardware RGA stuff was tested and built around the Steamlink app, a future PR.

Additional Context

Again, uses a mirror host here https://github.com/felixjones/linux-rga3-mirror for the rga3 driver

---

AI Usage

The detailed patch descriptions and comments are machine generated; I have reviewed them for correctness.

The code comments are generated by Copilot, there was a heck of a lot of scar tissue I had to crawl through when cleaning these up for PR and most of the comments I had originally left for myself, now they are a lot more verbose and more explainy as AI tends to do with comments. As a result, there's more comment lines then there are active lines of code (I did try to keep this as minimal as possible).

[loki666]

sick of LLM users...

[felixjones]

sick of LLM users...

:|

I'm at a loss for words. Great job pushing me out of the project with a swing and a miss?

[loki666]

just looking at the PR, I can see you didn't even tested it...
hint (Rockchip folder)

just looking at your PR description I can see you didn't write it (an LLM did)

this is not acceptable

[felixjones]

gamescope__overlay_the_Rockchip_package_apps.patch

I'm happy to leave this project

Wishing the best, this was a fun one to work on, and thank you.

[spycat88]

gamescope__overlay_the_Rockchip_package_apps.patch

I'm happy to leave this project

Wishing the best, this was a fun one to work on, and thank you.

Sorry to hear you feel that way. We genuinely appreciate the time and effort you and everyone else puts into this project.

That said, when submitting a pull request it's important to be open to constructive feedback, as code review is a normal part of the development process.

Looking at this PR, it doesn't appear to have been tested, as the patches have been placed in the wrong directory (projects/Rockchip). Issues like this make it difficult for us to review and merge changes, and they also create unnecessary work for the maintainers.

We're always happy to work with contributors and help improve submissions, but we do ask that PRs are reasonably validated before they're submitted. Taking the time to test your changes and ensure they're in the correct location helps everyone involved, including the wider community.