Fix Steam Remote Play black screen (Intel/NVIDIA) and inverted R/B colours (NVIDIA)

[cmspam]

Fix black screen (Intel/NVIDIA) and inverted colours (NVIDIA) when streaming via Steam Remote Play

Two issues affect PipeWire streaming (e.g. Steam Remote Play).

Black screen: Screenshot textures were created without bSampled = true, so no sampled image view was created for the RGB-to-NV12 conversion shader, resulting in a black frame. Fixed by adding bSampled = true in vulkan_acquire_screenshot_texture(). This is safe on all hardware and also fixes the same black screen on Intel (confirmed by @matte-schwartz in the PR comments).

Inverted R/B channels: Sampling a VK_FORMAT_A2R10G10B10_UNORM_PACK32 storage image on the NVIDIA proprietary driver returns R and B swapped. The RGB-to-NV12 colour matrix then operates on the wrong channels, causing the remote client to display inverted red and blue. Fixed by using DRM_FORMAT_XBGR2101010 as the intermediate capture format on NVIDIA, compensating for the driver's component ordering.

The colour fix is gated on VK_DRIVER_ID_NVIDIA_PROPRIETARY and has no effect on Nouveau, NVK, Intel, or any other vendor. A new isNvidiaProprietaryDriver() accessor is added to CVulkanDevice, populated via VkPhysicalDeviceDriverProperties chained into the existing GetPhysicalDeviceProperties2 call in createDevice().

Per review feedback, split into two commits: the sampling fix (Intel + NVIDIA) and the NVIDIA-specific colour swap.

Tested with NVIDIA RTX 4080, with Steam Remote Play in Game Mode (using Jovian) to Steam Link on an iPhone (PipeWire NV12 + NVENC AV1).

[nauzethc]

I can confirm both problems are fixed (black screen and inverted colors) but introduces double inputs from client side. I tested Steam Remote Play from Steam Deck, D-Pad and buttons are send twice to remote host (in-game interactions) while the controller connected directly to the host works correctly.

[cmspam]

I can confirm both problems are fixed (black screen and inverted colors) but introduces double inputs from client side. I tested Steam Remote Play from Steam Deck, D-Pad and buttons are send twice to remote host (in-game interactions) while the controller connected directly to the host works correctly.

Are you sure this is a new problem introduced by this patch? It doesn't touch input at all, so I think you will likely have the same issue without this patch. Could you please test it without the patch (it will have a black screen, yes, but inputs still go through) to make sure this patch introduces the issue?

I don't have a steam deck to test with, but I tested with a nintendo switch controller paired to my iPhone, logged controller inputs, and didn't see any double inputs in that case, so this may be something steam deck-specific. Do you have any other clients you can try streaming to?

[DeNo64]

No, change for me at all.

Not getting the double inputs like nauzethc. But also still seeing back frame.
Potentially something different on the nvidia-open-proprietry driver?

GPU: Nvidia 2080 Super
Driver: nvidia-open-dkms 590.48.01-4
DM: Hyprland (Wayland)
Reported Encoding: Desktop Black Frame + NVENC HEVC
Tested Clients: Steam Deck & Pixel 6 Pro (Steam Link Android)

[cmspam]

No, change for me at all.

Not getting the double inputs like nauzethc. But also still seeing back frame. Potentially something different on the nvidia-open-proprietry driver?

GPU: Nvidia 2080 Super Driver: nvidia-open-dkms 590.48.01-4 DM: Hyprland (Wayland) Reported Encoding: Desktop Black Frame + NVENC HEVC Tested Clients: Steam Deck & Pixel 6 Pro (Steam Link Android)

I am on the same driver you are. Your post mentions that you are using Hyprland.
This fix targets not using a desktop environment or a traditional compositor like Hyprland, but instead using gamescope-session (Steam's Game Mode) as the Wayland compositor.

I also get a black screen when trying to stream with KDE, but I believe this is a different issue regarding wayland security, as I also have a black screen on a different PC with AMD graphics in this case. The issue of streaming from a desktop environment (which I think you are describing) can be worked around by running steam with the -pipewire flag and allowing steam to share the desktop via pipewire.

Have you tested using gamescope-session?

[DeNo64]

Okay, using your gamescope patch from a TTY does work and I don't seem to be getting the double inputs either. Which is great.

Can this patch also be applied to the "Game Vulkan NV12" encoder, for using in a Nested Gamescope session?

The encoding performance of 'Desktop PipeWire NV12' is significantly worse. Which I assume is related to the graphical flickering I'm experiencing on the host (not visible on client) relating to some combination of my setup, nvidia driver issue, and/or another gamescope issue on nvidia (not related to this patch)

[cmspam]

Can this patch also be applied to the "Game Vulkan NV12" encoder, for using in a Nested Gamescope session?

I don't think the Game Vulkan NV12 encoder has any similar issues that need to be patched. When running a game in steam it switches from the "Black Screen" encoder to the "Game Vulkan NV12" encoder, at which point I was able to see it on my phone.

But I may be misunderstanding what you're asking. Could you be really specific?

[nauzethc]

Replying to #2094 (comment)

I used Plasma Desktop under Wayland and I can confirm that it works without getting black screen and with the colors displayed correctly. I'll test with the master version from the repository to see if the input problem occurs, but I seem to recall that it didn't happen in previous tests. It's certainly strange.

[DeNo64]

Can this patch also be applied to the "Game Vulkan NV12" encoder, for using in a Nested Gamescope session?

I don't think the Game Vulkan NV12 encoder has any similar issues that need to be patched. When running a game in steam it switches from the "Black Screen" encoder to the "Game Vulkan NV12" encoder, at which point I was able to see it on my phone.

But I may be misunderstanding what you're asking. Could you be really specific?

Are you using gamescope as a launch options for the game you're streaming from steam?
This is the issue called out in #1596. I can stream games fine over Remote Play without Gamescope. But if I want to run my game in gamescope (for better HDR, as an example) and stream via Remote Play the game will stay on 'Black Frame NV12'

[cmspam]

Replying to #2094 (comment)

As far as I can tell, anything running on Wayland, including nested gamescope sessions, can't be captured by Steam unless pipewire is utilized or Steam itself is running inside gamescope. This PR only fixes an NVIDIA-specific issue with streaming when Steam is already running inside a gamescope session — it doesn't address the broader nested session capture problem.

As far as I can tell, this is a Steam/Wayland compositor limitation rather than a gamescope-specific one, and PipeWire is likely the only supported path forward for that. I could be wrong though.
I've also removed the reference to resolving #1596 from the PR — you're right that while the NVIDIA bug was discussed there, this doesn't address the original intent of that issue. Thanks for pointing that out.

That said, there is a workaround for streaming nested sessions — run Steam itself inside gamescope:

gamescope -W 3840 -H 2160 -r 120 -f --hdr-enabled -- steam

From there I was able to successfully stream games with HDR and so on.

[DeNo64]

Okay, this is the aspect I was missing. Running Steam within the gamescope window and then launching the game is now displaying the stream, with all the correct colours, and is using PipeWire NV12. (As opposed to just a black screen - still using PipeWire NV12 on v3.16.20-1 of gamescope).
So that's good.

Testing again under steam -pipewire, using the current gamescope 3.16.20-1 as the launch option for the game, the game is displayed correctly (although with the bad Pipewire performance I'm getting).
With your patched version of gamescope in the launch options, all I get is the generic game splashscreen (with the repeated game poster) which shows while it's setting up the connection + audio from the game.

I'll keep playing around with this tomorrow and see if I can figure out what's going on.

[cmspam]

Thank you for testing it out so thoroughly.

The patch really only makes two changes-- adding bSampled = true in vulkan_acquire_screenshot_texture(), and inverting the red and blue if the hardware is nvidia.

I don't think those changes would cause any technical differences in pipewire streaming, in terms of whether it works or not, so it's interesting that you are experiencing them. Please let me know if you find out anything else, I'm happy to adjust the changes in this PR if they can be made better somehow.

Although I may not have the same game or hardware, if you can list the game you're trying it with and the exact command used to run it, I can try to recreate it.

[nauzethc]

I've run several tests today with different parameters and code versions, and I think I can conclude that the double input problem is related to my setup or GameScope configuration. For some reason I can't remember, I was including the -pipewire-dmabuf parameter, and it seems to be introducing this problem (sometimes on the host and sometimes on the client).

Now everything seems to be working correctly. Thank you so much!

[lukewilde]

there is a workaround for streaming nested sessions — run Steam itself inside gamescope

Can confirm this patch is working for me if I run steam with the following params:

gamescope -W 2560 -H 1440 -e -- steam -pipewire -tenfoot

[TheOverpassArsonist]

there is a workaround for streaming nested sessions — run Steam itself inside gamescope

Can confirm this patch is working for me if I run steam with the following params:

gamescope -W 2560 -H 1440 -e -- steam -pipewire -tenfoot

Not directly related to this issue (though I really do hope this can be released soon as I am currently running into this issue myself) but if you're using a flatpaked gamescope how would you run steam via the gamescope binary like that? For me I've always had to use 'flatpak run ##.##.##' explicitly when running anything, so I don't think that command would work on my system. This is probably just some stupid flatpak-specific quirk but I don't have a bespoke setup or anything (basically just garuda) so I imagine this is a pretty common configuration.

If I try to run the gamescope flatpak like I normally would I just get a bwrap execvp error about /bin/sh not existing, like below

[🔍] × flatpak run org.freedesktop.Platform.VulkanLayer.gamescope kate
bwrap: execvp /bin/sh: No such file or directory

PS : once this fix gets released, it might be worth integrating this into Steam's "Settings > Remote Play > Change desktop resolution to match streaming client" setting. Not sure who should be pinged about that (as far as I'm aware the Steam client itself is not open source so it'd probably have to be someone at Valve internally) but unless I'm missing something the intended behaviour of that option (once this issue is resolved) should essentially just add/modify a gamescope launch option like "gamescope -w -h -W -H -f -- %command%" to any game you launch from a steam Remote Play session. (for instance "gamescope -w 3840 -h 2160 -W 1920 -H 1080 -f -- %command%" to stream to a 4k display from a 1080p host)

[blondejefe]

Man, I'd trade a kidney for someone to fix this on AMD too. Wild that this bug has been chilling since 2019 like it pays rent. And now Valve's about to roll out a whole Steam Machine renaissance where everyone discovers Steam Remote Play—doesn't actually remote or play.

[matte-schwartz]

I think it would maybe make more sense as two commits, one for the sampled texture (which affects both Intel and NVIDIA) and then one just focused on NVIDIA, but that's more of a nit than a blocker imo. otherwise looks good to me.

[cmspam]

@matte-schwartz Did you try this patch on intel? Did it output over Remote Play with the correct colors? I hadn't had intel for testing and had just worked on nvidia, so this is the first time I was made aware intel needed it too. Nvidia requires a color swap for inverted R/B channels, but I applied that only if the driver is nvidia, so if you could let me know if intel also needs it I can update accordingly.

[matte-schwartz]

@cmspam I tested my own PR earlier on Intel which was just the sampling change and the colors appeared to be normal in both recordings and Remote Play, so that part of the issue seems to just be nvidia:

5D4079B5FD3743F3.mp4

clip.mp4

[cmspam]

@matte-schwartz Right now in this PR, the color swap is tied to nvidia-only so it should be fine for intel as-is then. Thank you very much for checking it out. I am not so sure that those with the decision-making powers are looking into merging this anytime soon or at all, unfortunately, but it's good to know that it can fix intel as well. Thanks for letting me know.

I'm happy to split it into two commits, or keep it as is, whatever Valve would like me to do to be willing to merge it.

[matte-schwartz]

I think for now just resolve any merge conflicts (I haven't checked if it applies cleanly against master still), and @misyltoad can hopefully take a look and review when she has a chance. Nice spotting on both issues.

[cmspam]

@matte-schwartz Done. Let's see how it goes :)