Skip to content

Fix color mask display when color write enable is static - #3929

Merged
baldurk merged 1 commit into
baldurk:v1.xfrom
ShabbyX:fix-color-mask-display
Nov 7, 2026
Merged

baldurk merged 1 commit into
baldurk:v1.xfrom
ShabbyX:fix-color-mask-display

Conversation

@ShabbyX

@ShabbyX ShabbyX commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

When the color write enable state (from VK_EXT_color_write_enable) is
not dynamic (i.e the extension is not used),
VulkanRenderState::colorWriteEnable was initialized from pipeline's
color mask info. However, this information should be ignored if color
write mask is dynamic (with shader object / EDS3).

Instead, if color write enable state is not dynamic, it can be
implicitly considered to be true if VkPipelineColorWriteCreateInfoEXT is
not provided, or otherwise it's taken from that.

This change fixes the UI (and HTML export) where the color mask was
output as ____ in such situations regardless of what mask was
dynamically set.

@baldurk baldurk left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I understand this fix correctly then Serialise_vkCmdSetColorWriteMaskEXT should also fill out renderstate.colorWriteEnable - I think it should set it all to true since that just means the mask gets used. The idea of this state tracking code is that when a pipeline is bound we set all of its static state into the renderstate, and any dynamic state gets skipped - the application is responsible for setting all dynamic states before or after the pipeline bind, so it effectively layers and the state comes from exactly one source depending on if it's static or dynamic.

If we don't do that then nothing sets renderstate.colorWriteEnable when either of these are dynamic, so it may become stale. I expect in practice no-one will ever make a pipeline with color write enable state as static because it's pretty meaningless and only exists to define the dynamic state, but if you had a pipeline where color write enable was dynamic and you set it some of them to false, then bound a pipeline where color write mask was dynamic, nothing would reset that color write enable state now after your change so it would incorrectly be reported as disabled.

@ShabbyX

ShabbyX commented Nov 7, 2026

Copy link
Copy Markdown
Contributor Author

Ah I was debating something along those lines with myself. I understand your point about single source of truth, that's how I implemented dynamic state in a driver myself, so no argument there.

Setting renderstate.colorWriteEnable in Serialise_vkCmdSetColorWriteMaskEXT doesn't seem right, but let me lay it out to convince myself too. So there's no such thing as "static state" for color write enable; if it's not dynamic, it's implicit. So:

  • Write mask static, write enable not dynamic: Write mask controls the output, write enable could be implicitly false if mask is 0, but TBH if write enable is unconditionally true in this case, it probably doesn't make a difference anywhere.
  • Write mask dynamic, write enable not dynamic: In this case, write enable cannot be implicitly derived from write mask (I guess it could when setting the state dynamically). Again, I think it doesn't matter if write enable is unconditionally considered true, right?
  • Write mask whatever, write enable dynamic: Write enable is just set dynamically

If my understanding of the first two cases is correct, then what I can do here is to unconditionally set write-enable to true in this same piece of code. The internal debate I had was whether it makes any difference if the array is empty or set to all true values. I thought empty would be fine (since elsewhere empty implies enabled writes), but as you pointed out it doesn't stay empty after pipeline changes.

I'll update the PR in a few hours. Do let me know if my assumption is incorrect (my assumption being that "color mask 0 + color write enabled" behaves identically to "color mask 0 + color write disabled").

@ShabbyX
ShabbyX force-pushed the fix-color-mask-display branch from b653b81 to e8d0afa Compare November 7, 2026 01:50
@ShabbyX

ShabbyX commented Nov 7, 2026

Copy link
Copy Markdown
Contributor Author

Actually found a minute to update the PR, please take a look again.

My reasoning for why this is ok is that if both state is dynamic, the app could set color-write-enable to true and color-write-mask to false and that's presumably fine. If color-write-enable is not dynamic, it's as if it's set to true and works the same way.

@ShabbyX
ShabbyX requested a review from baldurk November 7, 2026 01:52
@baldurk

baldurk commented Nov 7, 2026

Copy link
Copy Markdown
Owner

So there's no such thing as "static state" for color write enable; if it's not dynamic, it's implicit.

The thing is that this is not quite accurate, though as I say in practical terms I don't expect anyone to use the state statically because why would you bother. We do need to handle what the spec defines though.

The spec defines that there is VkPipelineColorWriteCreateInfoEXT which can be chained onto VkPipelineColorBlendStateCreateInfo to override the writeMask. If that struct is omitted (which again anyone sensible would do) it will act as if the array is attachmentCount sized and all true.

As I understand it, this is mostly just a quirk of how the spec was written so that the dynamic state can be defined in terms of "setting that state dynamically" and that's where vkCmdSetColorWriteEnableEXT kicks in. But it does allow the state to be set statically and does also interact with a dynamic write mask.

I don't think your updated code is quite correct in handling this specific case, but I think you're right that setting writeEnable in Serialise_vkCmdSetColorWriteMaskEXT won't quite be correct either if the write enable was static.

I think in order to handle this properly we need to track write enable entirely separately from what the pipeline's static writeMask was so that this static state can be applied entirely independently of what the writeMask was. Flattening it down means we can't distinguish and can't properly apply them depending on the combination of dynamic states.

@ShabbyX
ShabbyX force-pushed the fix-color-mask-display branch from e8d0afa to d89f428 Compare November 7, 2026 04:50
When the color write enable state (from VK_EXT_color_write_enable) is
not dynamic (i.e the extension is not used),
VulkanRenderState::colorWriteEnable was initialized from pipeline's
color mask info.  However, this information should be ignored if color
write mask is dynamic (with shader object / EDS3).

Instead, if color write enable state is not dynamic, it can be
implicitly considered to be true if VkPipelineColorWriteCreateInfoEXT is
not provided, or otherwise it's taken from that.

This change fixes the UI (and HTML export) where the color mask was
output as `____` in such situations regardless of what mask was
dynamically set.
@ShabbyX
ShabbyX force-pushed the fix-color-mask-display branch from d89f428 to c09774e Compare November 7, 2026 04:51
@ShabbyX

ShabbyX commented Nov 7, 2026

Copy link
Copy Markdown
Contributor Author

The spec defines that there is VkPipelineColorWriteCreateInfoEXT which can be chained onto VkPipelineColorBlendStateCreateInfo to override the writeMask

😮 Unbelievable

Thanks for pointing that out. I made an attempt at taking that struct into account by finding my way via grep, hopefully I haven't missed any spot. The dynamic vs static write mask issue is still fixed, but disclaimer, I don't have access to content that uses VK_EXT_color_write_enable (yet)

@baldurk baldurk left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this looks good now, but I should probably write a test for the different combinations of static/dynamic and one set of states leading into the other etc.

@baldurk
baldurk merged commit 2835a73 into baldurk:v1.x Nov 7, 2026
20 checks passed
@github-actions github-actions Bot locked as resolved and limited conversation to collaborators Nov 10, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants