Windows: handle generic executable map-view notifications

[FrontMage]

Summary

This is a draft/RFC PR from the investigation in #5399. While debugging the WoW NoExec failure path, I found a case where Wine can report a successful executable mapped-section view that is not a PE image mapping. FEX currently routes NotifyMapViewOfSection through image-map handling, so this kind of executable MEM_MAPPED view can miss the normal executable interval/protection tracking path.

This PR contains only the FEX-side part of the experiment. I am opening it as a draft because the full behavior depends on Wine/FEX cooperation, and I would like guidance on whether this is the expected place to handle these notifications.

What changed

• Make ARM64EC and WOW64 HandleImageMap() reject non-PE mappings instead of treating every map-view notification as an image map.
• For non-image map-view notifications that still look like valid mapped ranges, route the range through InvalidationTracker->HandleMemoryProtectionNotification(...).
• Keep image map handling unchanged for real PE images.

Why

The failing path observed during #5399 had executable MEM_MAPPED guest pages visible to Wine/Windows state, but FEX did not have a matching executable interval for the page. In the local diagnostic build this showed up as NoExec entry-block failures, for example:

E B4 NoExec instruction in entry block: 1A2163D
E B4 NoExec instruction in entry block: 1A20539

With the paired Wine-side diagnostic change, Wine reports executable non-image section views through the existing backend memory-protect callback. Example diagnostic evidence from the local run:

notify_map_view_of_section wine.mapview.exec-protect-shim addr=0000000001A60000 size=1048576 protect=0x20 alloc=0 status=0

The FEX-side change in this PR makes those generic map-view notifications update the executable/protection interval state instead of being dropped by image-only handling.

Cross-repo concern

This is the part I would like guidance on. The local fix involved both sides:

• Wine side: reusing the existing memory-protect backend callback for successful executable non-image map views.
• FEX side: accepting non-PE map-view notifications as generic protection updates.

I am not sure whether this is the intended contract for NotifyMapViewOfSection, or whether FEX would prefer a different Wine callback/ABI shape, a Wine-only change, or a different FEX-side path. Please treat this PR as a request for review of the approach rather than a final proposal.

Question

What would be the preferred upstream shape for this fix: keep this FEX-side handling, move more of the responsibility to Wine, split this into coordinated Wine and FEX PRs, or use a different notification path entirely?

[Sonicadvance1]

A good thing to discuss here is:

1. What API is the game using to map executable memory?
2. What interface does Windows use to pass that information to the emulator?
3. Why is WINE missing whatever path this is?

From FEX's point of view the only reason we care to distinguish if the mapping is a PE or not is so we can load the volatile metadata and other bits of data. If it is not a PE file, but still backed by the filesystem then we just want to know that for code caching, since we don't want to try and cache anonymous executable memory.

It seems to me that currently on FEX's side, we just need to make sure our code that reads PE file data just needs to skip a bunch of work when it isn't a PE file, but researching what Windows does by tracking API calls to FEX with a test app would obviously elucidate what is necessary.

[FrontMage]

A good thing to discuss here is:

1. What API is the game using to map executable memory?
2. What interface does Windows use to pass that information to the emulator?
3. Why is WINE missing whatever path this is?

From FEX's point of view the only reason we care to distinguish if the mapping is a PE or not is so we can load the volatile metadata and other bits of data. If it is not a PE file, but still backed by the filesystem then we just want to know that for code caching, since we don't want to try and cache anonymous executable memory.

It seems to me that currently on FEX's side, we just need to make sure our code that reads PE file data just needs to skip a bunch of work when it isn't a PE file, but researching what Windows does by tracking API calls to FEX with a test app would obviously elucidate what is necessary.

Yes, on the Wine side my local experiment already adds the missing notification for the path I observed: after a successful non-image executable section view, Wine reuses the existing backend memory-protect callback and emits the wine.mapview.exec-protect-shim diagnostic. That is the paired Wine-side change I mentioned in the PR body.

What I have not proven yet is the Windows-native oracle for the whole chain. From the Wine side the observed path is a successful mapped-section view, effectively reaching the NtMapViewOfSection layer, and the mapped range is executable MEM_MAPPED. However, I have not yet captured whether the game directly calls MapViewOfFile(Ex) / NtMapViewOfSection, or reaches it through an internal loader/helper path, nor have I captured the exact callback sequence that native Windows uses to notify the emulator for this non-PE executable view.

So I think the current FEX-side patch should still be treated as an RFC. The concrete next step is to build a small Windows test/probe that maps executable non-PE section views and compare the native Windows emulator notification behavior against the Wine path. That should answer whether FEX should handle this in NotifyMapViewOfSection, whether Wine should only send a memory-protect notification, or whether the split should be different.