gamescope: Vulkan Compositor without VK_EXT_robustness2

[felixjones]

Summary

These patches allow Gamescope to run on the Mali-G52

Gamescope relies on VK_EXT_robustness2, which is not exposed by either Panvk nor libmali, however it's also only used for null descriptor binding so in practice it doesn't take much effort to remove that required extension.

This also uses Vulkan shader rotation as the rockchip VOP doesn't have hardware rotation without RGA (which is coming via other PRs). There is a performance penalty with Vulkan shader rotation, but it's supposed to be the most compatible. As such, expect bad performance with Gamescope until librga comes into the picture.

Feel free to ask questions - this was quite the journey.

Testing

This was tested and built exclusively around glmark2-es2-wayland, vkgears, and vkcube. I recommend installing those if you want to test this yourself.

To actually test this, release the EmulationStation/Sway session from the DRM so gamescope can own it:

systemctl stop essway.service sway.service

For panvk the following environment variables need to be set:

export PAN_I_WANT_A_BROKEN_VULKAN_DRIVER=1 MESA_VK_VERSION_OVERRIDE=1.3

libmali masks libGL.so.1 with /dev/null (days of debugging...) so that needs to be unmasked so gamescope can use it.

Here's a convenience script for running glmark2-es-wayland:

#!/bin/sh
export XDG_RUNTIME_DIR=/var/run/0-runtime-dir
mkdir -p "$XDG_RUNTIME_DIR"; chmod 700 "$XDG_RUNTIME_DIR"
export PAN_I_WANT_A_BROKEN_VULKAN_DRIVER=1 MESA_VK_VERSION_OVERRIDE=1.3
exec unshare -m --propagation private -- /bin/sh -c '
  for lib in /usr/lib/libGL.so.1 /usr/lib32/libGL.so.1; do
    real=$(readlink -f "$lib" 2>/dev/null)
    [ -n "$real" ] && umount "$real" 2>/dev/null
  done
  exec gamescope --expose-wayland -W 1920 -H 1080 -r 60 -- glmark2-es2-wayland
'

And yes --expose-wayland is required.

Additional Context

Again, performance isn't ideal, using hardware RGA for direct scanout is the real goal here

---

AI Usage

The detailed patch descriptions and comments are machine generated; I have reviewed them for correctness.

[loki666]

but gamescope is currently only needed for Steam... I don't remember steam being bundled for rk3576 ....

[felixjones]

but gamescope is currently only needed for Steam... I don't remember steam being bundled for rk3576 ....

This is all towards working on getting Steam running on rk3576

The current Arm builds of Steam require atomic extensions that the rk3576 lacks, however historic Arm builds do not have that requirement, plus Steam x86 works via FEX. Gamescope is one of the missing puzzle pieces, this PR adds that piece

[loki666]

ok but then gates these specific patches to RK3576
in that same PR (but different commit) also bring up Steam so it can be tested

[felixjones]

ok but then gates these specific patches to RK3576 in that same PR (but different commit) also bring up Steam so it can be tested

I can gate them for rk3576, however these patches also apply to other architectures and help unblock Steam for those also. My plan is to keep strictly rk3576 specifics gated, but patches that help other architectures ungated

I will rename the PR title to remove mention of rk3576

---

I will demonstrate Steam on a different branch without the intention of merging it as that is something that needs additional scrutiny separate to the components for gamescope. I wouldn't want to give users the impression that Steam will be flawless, when it will be either old-arm Steam versus software rendered x86 latest. Most of my efforts were in writing thunks to make Steam x86 an almost native experience (again something that is not rk3576 specific)

[felixjones]

Superseded by #3019