Skip to content

Add a Windows Graphics Capture backend for content recognition - #20630

Merged
seanbudd merged 3 commits into
nvaccess:masterfrom
cary-rowen:wgc-capture-backend
Aug 13, 2026
Merged

Add a Windows Graphics Capture backend for content recognition#20630
seanbudd merged 3 commits into
nvaccess:masterfrom
cary-rowen:wgc-capture-backend

Conversation

@cary-rowen

Copy link
Copy Markdown
Contributor

Link to issue number:

Part of ADR #20480.

Summary of the issue:

NVDA did not have a native Windows Graphics Capture backend for the region capture path used by content recognition.

Description of user facing changes:

None yet. This PR adds backend support only.

Description of developer facing changes:

Added wgcCapture exports to nvdaHelperLocalWin10.dll and bound them from source/NVDAHelper/localWin10.py.

Description of development approach:

Implemented a native helper in nvdaHelper/localWin10/wgcCapture.cpp / .h that checks whether WGC monitor capture is available, captures the virtual-screen region by composing the intersecting monitors, copies the pixels into an NVDA-owned buffer, and scales the result to the requested destination size.

The helper stays isolated from the content recognition call sites so the next PR can switch behavior in one place without broad churn.

Testing strategy:

Built the source tree successfully and confirmed the new helper links into nvdaHelperLocalWin10.dll. No call site is wired up yet, so there is no user-facing workflow to exercise at this stage.

Known issues with pull request:

None.

Code Review Checklist:

  • Documentation:
    • Change log entry
    • User Documentation
    • Developer / Technical Documentation
    • Context sensitive help for GUI changes
  • Testing:
    • Unit tests
    • System (end to end) tests
    • Manual testing
  • UX of all users considered:
    • Speech
    • Braille
    • Low Vision
    • Different web browsers
    • Localization in other languages / culture than English
  • API is compatible with existing add-ons.
  • Security precautions taken.

@cary-rowen
cary-rowen marked this pull request as ready for review August 10, 2026 12:46
@cary-rowen
cary-rowen requested a review from a team as a code owner August 10, 2026 12:46
@cary-rowen
cary-rowen requested a review from seanbudd August 10, 2026 12:46
@seanbudd
seanbudd requested a lite review from Copilot August 11, 2026 01:01

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a native Windows Graphics Capture (WGC) implementation to nvdaHelperLocalWin10.dll to enable a future switch of NVDA’s content-recognition region capture away from GDI when Screen Curtain is active (per ADR/issue #20480), while keeping call sites unchanged for now.

Changes:

  • Added a new native helper (wgcCapture.cpp/.h) that can (1) detect WGC availability and (2) capture a virtual-screen region across monitors into a caller-owned BGRA buffer with scaling.
  • Exported the new helper from nvdaHelperLocalWin10.dll and added Python ctypes bindings in source/NVDAHelper/localWin10.py.
  • Updated the localWin10 SCons build to compile the new source and link required Windows libraries.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File Description
source/NVDAHelper/localWin10.py Adds ctypes bindings for wgcCapture_isSupported and wgcCapture_captureScreenRegion.
nvdaHelper/localWin10/wgcCapture.h Declares the exported WGC support + capture APIs for the localWin10 helper DLL.
nvdaHelper/localWin10/wgcCapture.cpp Implements WGC monitor capture, compositing across intersecting monitors, and scaling into a BGRA buffer.
nvdaHelper/localWin10/sconscript Adds wgcCapture.cpp to the build and links against required libs (d3d11, runtimeobject, user32).
Suppressed comments (1)

nvdaHelper/localWin10/wgcCapture.cpp:472

  • The requestPixels buffer is value-initialized to RGBQUAD{} (all zeros), which makes the “pixels outside all monitors are black” region transparent (alpha=0). If alpha is preserved for captured pixels, it’s better to make the background an opaque black (alpha=255) to avoid inconsistent alpha semantics in the returned BGRA8 buffer.
	std::vector<RGBQUAD> requestPixels(
		static_cast<std::size_t>(width) * height,
		RGBQUAD {}
	);

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread nvdaHelper/localWin10/wgcCapture.cpp
Comment thread nvdaHelper/localWin10/wgcCapture.cpp
@SaschaCowley SaschaCowley added the conceptApproved Similar 'triaged' for issues, PR accepted in theory, implementation needs review. label Aug 11, 2026

@seanbudd seanbudd left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @cary-rowen - just minor suggestions

Comment thread nvdaHelper/localWin10/wgcCapture.cpp Outdated
Comment thread nvdaHelper/localWin10/wgcCapture.cpp Outdated
Comment thread nvdaHelper/localWin10/wgcCapture.cpp Outdated
Comment thread nvdaHelper/localWin10/wgcCapture.cpp Outdated

@seanbudd seanbudd left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @cary-rowen

@seanbudd
seanbudd merged commit 429abaa into nvaccess:master Aug 13, 2026
43 of 46 checks passed
@github-actions github-actions Bot added this to the 2026.3 milestone Aug 13, 2026
seanbudd pushed a commit that referenced this pull request Aug 18, 2026
…ain and Magnifier (#20664)

Follow up #20630
Part of #20480.
Fixes #19164
Summary of the issue:

Content recognition currently uses GDI to capture screen pixels. Screen Curtain causes this capture to return black pixels, while NVDA's built-in Magnifier can cause recognition to receive magnified, cropped or color-transformed pixels instead of the original screen content.
Description of user facing changes:

On supported systems, Windows OCR can now be used while Screen Curtain or NVDA's built-in Magnifier is active. Existing GDI capture behavior is preserved when neither feature is active and when WGC is unavailable for Magnifier.
Description of developer facing changes:

Added a private Python wrapper for the Windows Graphics Capture backend introduced in #20630 and centralized capture selection in contentRecog.recogUi. Content recognizers using the shared recognition framework receive the same capture behavior. Negative virtual-screen origins are now accepted for monitors positioned left of or above the primary display. No public API changes are introduced.
Description of development approach:

GDI remains the default capture path. WGC is used when Screen Curtain is active, or when NVDA's built-in Magnifier is active and WGC is supported. A WGC failure while Magnifier is active falls back to GDI, preserving the previous usable behavior. No GDI fallback is allowed while Screen Curtain is active because it would return pixels from the obscured display. Capture failures and watchdog cancellation are passed through the existing recognition result lifecycle. Enabling Screen Curtain during automatically refreshed recognition is blocked only when WGC is unavailable.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

conceptApproved Similar 'triaged' for issues, PR accepted in theory, implementation needs review.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants