Add a Windows Graphics Capture backend for content recognition - #20630
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a native Windows Graphics Capture (WGC) implementation to nvdaHelperLocalWin10.dll to enable a future switch of NVDA’s content-recognition region capture away from GDI when Screen Curtain is active (per ADR/issue #20480), while keeping call sites unchanged for now.
Changes:
- Added a new native helper (
wgcCapture.cpp/.h) that can (1) detect WGC availability and (2) capture a virtual-screen region across monitors into a caller-owned BGRA buffer with scaling. - Exported the new helper from
nvdaHelperLocalWin10.dlland added Pythonctypesbindings insource/NVDAHelper/localWin10.py. - Updated the localWin10 SCons build to compile the new source and link required Windows libraries.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| source/NVDAHelper/localWin10.py | Adds ctypes bindings for wgcCapture_isSupported and wgcCapture_captureScreenRegion. |
| nvdaHelper/localWin10/wgcCapture.h | Declares the exported WGC support + capture APIs for the localWin10 helper DLL. |
| nvdaHelper/localWin10/wgcCapture.cpp | Implements WGC monitor capture, compositing across intersecting monitors, and scaling into a BGRA buffer. |
| nvdaHelper/localWin10/sconscript | Adds wgcCapture.cpp to the build and links against required libs (d3d11, runtimeobject, user32). |
Suppressed comments (1)
nvdaHelper/localWin10/wgcCapture.cpp:472
- The requestPixels buffer is value-initialized to RGBQUAD{} (all zeros), which makes the “pixels outside all monitors are black” region transparent (alpha=0). If alpha is preserved for captured pixels, it’s better to make the background an opaque black (alpha=255) to avoid inconsistent alpha semantics in the returned BGRA8 buffer.
std::vector<RGBQUAD> requestPixels(
static_cast<std::size_t>(width) * height,
RGBQUAD {}
);
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
seanbudd
reviewed
Aug 11, 2026
seanbudd
left a comment
Member
There was a problem hiding this comment.
Thanks @cary-rowen - just minor suggestions
Merged
5 tasks
seanbudd
pushed a commit
that referenced
this pull request
Aug 18, 2026
…ain and Magnifier (#20664) Follow up #20630 Part of #20480. Fixes #19164 Summary of the issue: Content recognition currently uses GDI to capture screen pixels. Screen Curtain causes this capture to return black pixels, while NVDA's built-in Magnifier can cause recognition to receive magnified, cropped or color-transformed pixels instead of the original screen content. Description of user facing changes: On supported systems, Windows OCR can now be used while Screen Curtain or NVDA's built-in Magnifier is active. Existing GDI capture behavior is preserved when neither feature is active and when WGC is unavailable for Magnifier. Description of developer facing changes: Added a private Python wrapper for the Windows Graphics Capture backend introduced in #20630 and centralized capture selection in contentRecog.recogUi. Content recognizers using the shared recognition framework receive the same capture behavior. Negative virtual-screen origins are now accepted for monitors positioned left of or above the primary display. No public API changes are introduced. Description of development approach: GDI remains the default capture path. WGC is used when Screen Curtain is active, or when NVDA's built-in Magnifier is active and WGC is supported. A WGC failure while Magnifier is active falls back to GDI, preserving the previous usable behavior. No GDI fallback is allowed while Screen Curtain is active because it would return pixels from the obscured display. Capture failures and watchdog cancellation are passed through the existing recognition result lifecycle. Enabling Screen Curtain during automatically refreshed recognition is blocked only when WGC is unavailable.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Link to issue number:
Part of ADR #20480.
Summary of the issue:
NVDA did not have a native Windows Graphics Capture backend for the region capture path used by content recognition.
Description of user facing changes:
None yet. This PR adds backend support only.
Description of developer facing changes:
Added
wgcCaptureexports tonvdaHelperLocalWin10.dlland bound them fromsource/NVDAHelper/localWin10.py.Description of development approach:
Implemented a native helper in
nvdaHelper/localWin10/wgcCapture.cpp/.hthat checks whether WGC monitor capture is available, captures the virtual-screen region by composing the intersecting monitors, copies the pixels into an NVDA-owned buffer, and scales the result to the requested destination size.The helper stays isolated from the content recognition call sites so the next PR can switch behavior in one place without broad churn.
Testing strategy:
Built the source tree successfully and confirmed the new helper links into
nvdaHelperLocalWin10.dll. No call site is wired up yet, so there is no user-facing workflow to exercise at this stage.Known issues with pull request:
None.
Code Review Checklist: