From ffd3981461f4860eb9958fb68080aec87e5f5ec9 Mon Sep 17 00:00:00 2001 From: Etienne Lescot Date: Thu, 13 Aug 2026 20:55:20 +0200 Subject: [PATCH 01/10] docs(e2e): say what injected input can never prove about the HUD Four corrections to the computer-use E2E guidance, each one found by following the existing text and hitting the wall it does not mention. The HUD click-through note had only its positive half: move the real cursor and the control becomes clickable. The negative half is the one that costs an hour. On Windows `forward` is a global WH_MOUSE_LL hook, and only a real OS mouse move drives it; CDP-injected input arrives below the OS hit-test, fires the DOM handler, and looks like it worked while never exercising click-through at all. This repo has a green Playwright test clicking HUD testids, which reads as proof that Playwright can drive the HUD -- it proves renderer wiring and nothing else. The failure #266 actually shipped, a painted and permanently inert HUD, is invisible to injected input by construction and cannot be regression-tested there, so the spec now says so next to those clicks. `request_access` was documented as "grant electron.exe" with no timing. electron.exe is not an installed app, so the resolver only finds it once the process exists and owns a window; asking earlier fails, and one unresolvable name short-circuits the whole request. Granting Openscreen instead resolves to the installed exe and reports success while leaving the dev window masked. The worktree setup step said to copy the prebuilt native binaries without saying they are frozen. Nothing rebuilds them, so a helper older than the change under test runs silently: this pass recorded a healthy 1080p60 file whose encoder-selection event had no `container` field, because the helper predated the fragmented-MP4 commit by seventeen hours. Date the binary and grep it for a string the change introduced. --- AGENTS.md | 3 +++ tests/e2e/windows-native-checklist.spec.ts | 10 ++++++++++ 2 files changed, 13 insertions(+) diff --git a/AGENTS.md b/AGENTS.md index d7b2ec8ab..fc38e0736 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -86,10 +86,12 @@ Unit/browser tests can't exercise real capture (native screen recording, a physi - Normal: `npm run dev` — Vite serves the renderer and `vite-plugin-electron` opens the Electron window. The main process logs `Global shortcut registered: CommandOrControl+Shift+O` when ready (Ctrl/Cmd+Shift+O toggles the HUD). - The app is single-instance through `app.requestSingleInstanceLock()`, which keys on the `userData` path. If a leftover Electron process still holds it, a new launch quits silently (exit 0, no window) — kill leftover `electron` processes before relaunching. The lock is held by the OS and dies with the process, so there is nothing to clean up on disk. A dev build and the installed `Openscreen` resolve different `userData` paths and can run side by side. - **From a git worktree** (no `node_modules`/native binaries): junction/symlink `node_modules` from the main checkout (deps are usually identical — check `package-lock.json`), and copy the prebuilt native capture binaries from `electron/native/bin//` (gitignored — rebuilding needs the full VS/Xcode toolchain). Then `npm run dev` works normally. +- **Those binaries are frozen at whenever someone last built them, and nothing warns you.** They are not rebuilt by `npm run dev` or `npm run build`, so a helper older than the native change you came to test will run happily and silently exercise the old code path — the recording succeeds, and the thing you wanted to see is simply absent. Before trusting any native result, date the binary against the commit and grep it for a string the change introduced (`strings -a wgc-capture.exe | grep fragmented-mp4`). If it is stale and you have no toolchain, test the CI-built artifact instead; a dev build cannot answer the question. **Granting access** - `request_access` resolves names against installed apps. A **dev build runs as `electron.exe`** (or `Electron.app`), *not* the installed `Openscreen` — grant **`electron.exe`** or the dev window stays masked in screenshots. Non-allowlisted windows are masked (solid rectangles); the screenshot note lists their process names to add. +- **Start the app before asking for it.** `electron.exe` is not an installed app, so the resolver only finds it once the process exists *and* owns a window; ask any earlier and the call fails with `doesn't match any installed or running application` — and one unresolvable name short-circuits the whole request, including the names that would have resolved. Granting `Openscreen` instead is not a workaround: it resolves to `…\programs\openscreen\openscreen.exe`, so the dev window stays masked while the grant reports success. **The HUD widget** (recording controller) @@ -97,6 +99,7 @@ Unit/browser tests can't exercise real capture (native screen recording, a physi - **On macOS 26+ content protection is auto-disabled, so the HUD *is* visible and screenshottable with no flag.** That OS never displays a content-protected window at all — not just absent from captures, but never painted, leaving a tray icon, a live renderer and nothing on screen (confirmed on macOS 26.5 / Electron 41.2.1). `applyContentProtection` therefore skips the call there and logs a warning per window; the trade-off is that the HUD can appear in recordings on that OS until the ScreenCaptureKit helper excludes our own windows via `SCContentFilter(excludingWindows:)`, which it currently passes as `[]`. `OPENSCREEN_FORCE_CONTENT_PROTECTION=1` re-enables it to re-test against a future Electron. - The HUD is what opens the editor (clapper icon, tooltip *Open Studio*), so without that flag a whole slice of the app is unreachable from automation: killing the app to redeploy a native addon leaves you unable to reopen a project. - Frameless, transparent, always-on-top, `skipTaskbar`, centered at the **bottom of the primary display** (`createHudOverlayWindow`, 600×160). It is **click-through** (`setIgnoreMouseEvents(true, { forward: true })`): moving the real cursor over an interactive control makes that region clickable and shows its tooltip, so `mouse_move` → screenshot → `left_click` works; a blind click on empty HUD area passes through to the desktop. +- **Only a real OS mouse move reaches the HUD.** On Windows that `forward` option is a global `WH_MOUSE_LL` hook, and driving it is what lifts the input-transparency over a control. CDP-injected input never does: Playwright's `.click()`, `javascript_tool`-dispatched pointer events, and anything else synthesised into the renderer arrive *below* the OS hit-test, fire the DOM handler, and look like they worked — while the click-through path was never exercised at all. `tests/e2e/windows-native-checklist.spec.ts` does click HUD testids and stays green for exactly that reason; it proves renderer wiring, not reachability. Use computer-use (`mouse_move` → `left_click`), and never conclude from a passing injected click that a user could have clicked it. - Control row (left→right): layout preset, **source** button (`Screen`/`Window` → label becomes the picked source), system-audio toggle, mic toggle, **webcam toggle** (shows the detected camera name), cursor-highlight toggle, **record**, notes, open-editor, language, minimize, close. The record button is disabled until a source is chosen (tooltip: "Please select a source to record"). **The tray icon** (bottom-right notification area) diff --git a/tests/e2e/windows-native-checklist.spec.ts b/tests/e2e/windows-native-checklist.spec.ts index 959b15f1d..fcda17a2a 100644 --- a/tests/e2e/windows-native-checklist.spec.ts +++ b/tests/e2e/windows-native-checklist.spec.ts @@ -326,6 +326,16 @@ test.describe("Windows native checklist smoke tests", () => { // input-transparent when the hook fails to install can never be clicked again, // which is what bricked the app in issue #266. Both halves matter — that nothing // asks during construction, and that the renderer still does after mount. + // + // Note what this test therefore cannot do, and what no test in this file can. + // Only a real OS cursor move drives a WH_MOUSE_LL hook; CDP-injected input + // arrives below the OS hit-test, so Playwright's own `.click()` on a HUD testid + // — above, and in the source-selector step of the checklist test — reaches the + // DOM handler whether or not click-through is installed, or even working. Those + // clicks assert renderer wiring and nothing else. The failure #266 actually shipped + // (a painted, permanently inert HUD) is invisible to injected input by construction, + // so it belongs on the manual computer-use checklist and cannot be regression-tested + // here. Do not read a green run as evidence that the HUD is clickable. test("the HUD asks for click-through instead of being born with it", async () => { const app = await launchApp(); From 43af2fdda481b03140be4627a1f166906d9bcddc Mon Sep 17 00:00:00 2001 From: Etienne Lescot Date: Thu, 13 Aug 2026 21:08:06 +0200 Subject: [PATCH 02/10] docs(e2e): give a binary-string check that works on Windows MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The advice I had just written recommended `strings -a … | grep`, and Git Bash has no `strings`: the pipeline returns nothing and every binary reads as missing the change. It produced five confident false negatives against the CI-built helper, which does contain the fix. Use `findstr /M /C:` (handles binaries, ships with Windows), and always search a control string the old binary also has, so a broken search cannot masquerade as a stale binary. --- AGENTS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/AGENTS.md b/AGENTS.md index fc38e0736..a220da756 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -86,7 +86,7 @@ Unit/browser tests can't exercise real capture (native screen recording, a physi - Normal: `npm run dev` — Vite serves the renderer and `vite-plugin-electron` opens the Electron window. The main process logs `Global shortcut registered: CommandOrControl+Shift+O` when ready (Ctrl/Cmd+Shift+O toggles the HUD). - The app is single-instance through `app.requestSingleInstanceLock()`, which keys on the `userData` path. If a leftover Electron process still holds it, a new launch quits silently (exit 0, no window) — kill leftover `electron` processes before relaunching. The lock is held by the OS and dies with the process, so there is nothing to clean up on disk. A dev build and the installed `Openscreen` resolve different `userData` paths and can run side by side. - **From a git worktree** (no `node_modules`/native binaries): junction/symlink `node_modules` from the main checkout (deps are usually identical — check `package-lock.json`), and copy the prebuilt native capture binaries from `electron/native/bin//` (gitignored — rebuilding needs the full VS/Xcode toolchain). Then `npm run dev` works normally. -- **Those binaries are frozen at whenever someone last built them, and nothing warns you.** They are not rebuilt by `npm run dev` or `npm run build`, so a helper older than the native change you came to test will run happily and silently exercise the old code path — the recording succeeds, and the thing you wanted to see is simply absent. Before trusting any native result, date the binary against the commit and grep it for a string the change introduced (`strings -a wgc-capture.exe | grep fragmented-mp4`). If it is stale and you have no toolchain, test the CI-built artifact instead; a dev build cannot answer the question. +- **Those binaries are frozen at whenever someone last built them, and nothing warns you.** They are not rebuilt by `npm run dev` or `npm run build`, so a helper older than the native change you came to test will run happily and silently exercise the old code path — the recording succeeds, and the thing you wanted to see is simply absent. Before trusting any native result, date the binary against the commit and search it for a string the change introduced: `findstr /M /C:"fragmented-mp4" wgc-capture.exe` — `findstr` handles binaries and ships with Windows, whereas Git Bash has **no `strings`**, so `strings … | grep` there returns nothing and reads as a confident *absent* for every binary you point it at. Always search a control string the old binary also has (`encoder-selection`) so a broken search cannot masquerade as a stale binary. If it is stale and you have no toolchain, test the CI-built artifact instead; a dev build cannot answer the question. **Granting access** From 0b151373a1e849f8c7033b0c68fe1824bcb34eb1 Mon Sep 17 00:00:00 2001 From: Etienne Lescot Date: Thu, 13 Aug 2026 23:16:01 +0200 Subject: [PATCH 03/10] docs(agents): name the thing that actually builds a capture helper MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three fixes from review. The rebuild claim was wrong, and wrong in the direction that causes the trap the rest of this PR documents: electron-builder and `@electron/rebuild` do Node native-module ABI work, not the standalone Swift and C++ capture helpers. Those are separate executables built by `npm run build:native:` and only copied into the package as `extraResources` — `build:win` even passes `--config.npmRebuild=false`. A reader who believed the old sentence would expect a normal build to pick up a helper change. Nothing does. The staleness check quoted a bare filename, so it only worked from inside `electron/native/bin//`. Given from the repo root now, and it names the rebuild command instead of only offering the no-toolchain escape hatch. And `testids` is not a word. --- AGENTS.md | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index a220da756..12165d631 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -14,7 +14,7 @@ OpenScreen is a free, open-source screen recorder and video editor (Electron + R - Format: `npm run format` (Biome, tabs, double quotes, 100-col) - i18n check: `npm run i18n:check` (validates the 13 locale files) -**Use npm, not bun/pnpm/yarn/Deno.** Not a style preference. The native Swift (macOS) and C++ (Windows) capture helpers are rebuilt against Electron's ABI by electron-builder + `@electron/rebuild`, which resolve the tree through `package-lock.json`. Another package manager writes a different lockfile, so that rebuild breaks. `packageManager` + `engines` in `package.json` pin the versions; CI installs with `npm ci`. +**Use npm, not bun/pnpm/yarn/Deno.** Not a style preference. Node native modules are rebuilt against Electron's ABI by electron-builder + `@electron/rebuild`, which resolve the tree through `package-lock.json`. Another package manager writes a different lockfile, so that rebuild breaks. `packageManager` + `engines` in `package.json` pin the versions; CI installs with `npm ci`. Note what this does *not* cover: the standalone Swift (macOS) and C++ (Windows) capture helpers are separate executables, built by `npm run build:native:` and only *copied* into the package as `extraResources` — `build:win` even passes `--config.npmRebuild=false`. Nothing in a normal build compiles them. ## Development principles @@ -86,7 +86,13 @@ Unit/browser tests can't exercise real capture (native screen recording, a physi - Normal: `npm run dev` — Vite serves the renderer and `vite-plugin-electron` opens the Electron window. The main process logs `Global shortcut registered: CommandOrControl+Shift+O` when ready (Ctrl/Cmd+Shift+O toggles the HUD). - The app is single-instance through `app.requestSingleInstanceLock()`, which keys on the `userData` path. If a leftover Electron process still holds it, a new launch quits silently (exit 0, no window) — kill leftover `electron` processes before relaunching. The lock is held by the OS and dies with the process, so there is nothing to clean up on disk. A dev build and the installed `Openscreen` resolve different `userData` paths and can run side by side. - **From a git worktree** (no `node_modules`/native binaries): junction/symlink `node_modules` from the main checkout (deps are usually identical — check `package-lock.json`), and copy the prebuilt native capture binaries from `electron/native/bin//` (gitignored — rebuilding needs the full VS/Xcode toolchain). Then `npm run dev` works normally. -- **Those binaries are frozen at whenever someone last built them, and nothing warns you.** They are not rebuilt by `npm run dev` or `npm run build`, so a helper older than the native change you came to test will run happily and silently exercise the old code path — the recording succeeds, and the thing you wanted to see is simply absent. Before trusting any native result, date the binary against the commit and search it for a string the change introduced: `findstr /M /C:"fragmented-mp4" wgc-capture.exe` — `findstr` handles binaries and ships with Windows, whereas Git Bash has **no `strings`**, so `strings … | grep` there returns nothing and reads as a confident *absent* for every binary you point it at. Always search a control string the old binary also has (`encoder-selection`) so a broken search cannot masquerade as a stale binary. If it is stale and you have no toolchain, test the CI-built artifact instead; a dev build cannot answer the question. +- **Those binaries are frozen at whenever someone last built them, and nothing warns you.** They are not rebuilt by `npm run dev` or `npm run build`, so a helper older than the native change you came to test will run happily and silently exercise the old code path — the recording succeeds, and the thing you wanted to see is simply absent. Before trusting any native result, date the binary against the commit and search it for a string the change introduced — from the repo root: + + ``` + findstr /M /C:"fragmented-mp4" electron\native\bin\win32-x64\wgc-capture.exe + ``` + + `findstr` handles binaries and ships with Windows, whereas Git Bash has **no `strings`**, so `strings … | grep` there returns nothing and reads as a confident *absent* for every binary you point it at. Always search a control string the old binary also has (`encoder-selection`) so a broken search cannot masquerade as a stale binary. If it is stale, rebuild it with `npm run build:native:win` (or `:mac` / `:linux`) — that is the only thing that compiles a helper. Without the toolchain, test the CI-built artifact instead; a dev build cannot answer the question. **Granting access** @@ -99,7 +105,7 @@ Unit/browser tests can't exercise real capture (native screen recording, a physi - **On macOS 26+ content protection is auto-disabled, so the HUD *is* visible and screenshottable with no flag.** That OS never displays a content-protected window at all — not just absent from captures, but never painted, leaving a tray icon, a live renderer and nothing on screen (confirmed on macOS 26.5 / Electron 41.2.1). `applyContentProtection` therefore skips the call there and logs a warning per window; the trade-off is that the HUD can appear in recordings on that OS until the ScreenCaptureKit helper excludes our own windows via `SCContentFilter(excludingWindows:)`, which it currently passes as `[]`. `OPENSCREEN_FORCE_CONTENT_PROTECTION=1` re-enables it to re-test against a future Electron. - The HUD is what opens the editor (clapper icon, tooltip *Open Studio*), so without that flag a whole slice of the app is unreachable from automation: killing the app to redeploy a native addon leaves you unable to reopen a project. - Frameless, transparent, always-on-top, `skipTaskbar`, centered at the **bottom of the primary display** (`createHudOverlayWindow`, 600×160). It is **click-through** (`setIgnoreMouseEvents(true, { forward: true })`): moving the real cursor over an interactive control makes that region clickable and shows its tooltip, so `mouse_move` → screenshot → `left_click` works; a blind click on empty HUD area passes through to the desktop. -- **Only a real OS mouse move reaches the HUD.** On Windows that `forward` option is a global `WH_MOUSE_LL` hook, and driving it is what lifts the input-transparency over a control. CDP-injected input never does: Playwright's `.click()`, `javascript_tool`-dispatched pointer events, and anything else synthesised into the renderer arrive *below* the OS hit-test, fire the DOM handler, and look like they worked — while the click-through path was never exercised at all. `tests/e2e/windows-native-checklist.spec.ts` does click HUD testids and stays green for exactly that reason; it proves renderer wiring, not reachability. Use computer-use (`mouse_move` → `left_click`), and never conclude from a passing injected click that a user could have clicked it. +- **Only a real OS mouse move reaches the HUD.** On Windows that `forward` option is a global `WH_MOUSE_LL` hook, and driving it is what lifts the input-transparency over a control. CDP-injected input never does: Playwright's `.click()`, `javascript_tool`-dispatched pointer events, and anything else synthesised into the renderer arrive *below* the OS hit-test, fire the DOM handler, and look like they worked — while the click-through path was never exercised at all. `tests/e2e/windows-native-checklist.spec.ts` does click HUD test IDs and stays green for exactly that reason; it proves renderer wiring, not reachability. Use computer-use (`mouse_move` → `left_click`), and never conclude from a passing injected click that a user could have clicked it. - Control row (left→right): layout preset, **source** button (`Screen`/`Window` → label becomes the picked source), system-audio toggle, mic toggle, **webcam toggle** (shows the detected camera name), cursor-highlight toggle, **record**, notes, open-editor, language, minimize, close. The record button is disabled until a source is chosen (tooltip: "Please select a source to record"). **The tray icon** (bottom-right notification area) From cc55d4deccd82932274cab3b8e07d6e8a2d64505 Mon Sep 17 00:00:00 2001 From: Etienne Lescot Date: Thu, 13 Aug 2026 23:24:35 +0200 Subject: [PATCH 04/10] docs(agents): the HUD click-through rule is macOS too, not Windows only MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The bullet opened with "On Windows", which reads as a scope and is one. `forward` is `@platform darwin,win32` in Electron's typings, and the renderer asks for click-through on both — `!enabled && !isLinuxHud`. Linux is the exception, and the only platform where a blind click on the HUD lands; LaunchWindow.tsx already said so thirty lines from where I wrote the opposite. That mattered: computer-use drives the macOS build too, and an agent reading "On Windows" concludes the caveat is somebody else's problem, then spends an hour on an injected click that fires the DOM handler and proves nothing. The mechanisms do differ — WH_MOUSE_LL on Windows, Electron's own forwarding on macOS — so the sentence now separates the implementation from the consequence, which is shared. Also notes that a macOS spec written like the Windows one would prove no more than it does, since there is no macOS e2e spec yet to say it in. --- AGENTS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/AGENTS.md b/AGENTS.md index 12165d631..286e2855b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -105,7 +105,7 @@ Unit/browser tests can't exercise real capture (native screen recording, a physi - **On macOS 26+ content protection is auto-disabled, so the HUD *is* visible and screenshottable with no flag.** That OS never displays a content-protected window at all — not just absent from captures, but never painted, leaving a tray icon, a live renderer and nothing on screen (confirmed on macOS 26.5 / Electron 41.2.1). `applyContentProtection` therefore skips the call there and logs a warning per window; the trade-off is that the HUD can appear in recordings on that OS until the ScreenCaptureKit helper excludes our own windows via `SCContentFilter(excludingWindows:)`, which it currently passes as `[]`. `OPENSCREEN_FORCE_CONTENT_PROTECTION=1` re-enables it to re-test against a future Electron. - The HUD is what opens the editor (clapper icon, tooltip *Open Studio*), so without that flag a whole slice of the app is unreachable from automation: killing the app to redeploy a native addon leaves you unable to reopen a project. - Frameless, transparent, always-on-top, `skipTaskbar`, centered at the **bottom of the primary display** (`createHudOverlayWindow`, 600×160). It is **click-through** (`setIgnoreMouseEvents(true, { forward: true })`): moving the real cursor over an interactive control makes that region clickable and shows its tooltip, so `mouse_move` → screenshot → `left_click` works; a blind click on empty HUD area passes through to the desktop. -- **Only a real OS mouse move reaches the HUD.** On Windows that `forward` option is a global `WH_MOUSE_LL` hook, and driving it is what lifts the input-transparency over a control. CDP-injected input never does: Playwright's `.click()`, `javascript_tool`-dispatched pointer events, and anything else synthesised into the renderer arrive *below* the OS hit-test, fire the DOM handler, and look like they worked — while the click-through path was never exercised at all. `tests/e2e/windows-native-checklist.spec.ts` does click HUD test IDs and stays green for exactly that reason; it proves renderer wiring, not reachability. Use computer-use (`mouse_move` → `left_click`), and never conclude from a passing injected click that a user could have clicked it. +- **Only a real OS mouse move reaches the HUD — on macOS as much as on Windows.** `forward` is `@platform darwin,win32` in Electron's own typings, and the renderer asks for click-through on both; **Linux is the exception** (`!enabled && !isLinuxHud` in `LaunchWindow.tsx`, where the call is a no-op), so it is the one platform where a blind click on the HUD simply lands. The implementations differ — Windows installs a global `WH_MOUSE_LL` hook, macOS forwards through its own event path — but the consequence is identical: moving the real cursor onto a control is what lifts the input-transparency. CDP-injected input never does that, on any platform: Playwright's `.click()`, `javascript_tool`-dispatched pointer events, and anything else synthesised into the renderer arrive *below* the OS hit-test, fire the DOM handler, and look like they worked — while the click-through path was never exercised at all. `tests/e2e/windows-native-checklist.spec.ts` does click HUD test IDs and stays green for exactly that reason; it proves renderer wiring, not reachability, and a macOS spec written the same way would prove no more. Use computer-use (`mouse_move` → `left_click`), and never conclude from a passing injected click that a user could have clicked it. - Control row (left→right): layout preset, **source** button (`Screen`/`Window` → label becomes the picked source), system-audio toggle, mic toggle, **webcam toggle** (shows the detected camera name), cursor-highlight toggle, **record**, notes, open-editor, language, minimize, close. The record button is disabled until a source is chosen (tooltip: "Please select a source to record"). **The tray icon** (bottom-right notification area) From 0153a21177d535753c0919245b036679442c9f23 Mon Sep 17 00:00:00 2001 From: Etienne Lescot Date: Thu, 13 Aug 2026 23:27:38 +0200 Subject: [PATCH 05/10] docs(agents): point at the testing docs, which only linked one way manual-e2e-checklist.md sends the reader to AGENTS.md for the computer-use mechanics. AGENTS.md sent nobody back: its whole "Desktop E2E testing with computer-use" section, and the testing section above it, named no file under technical-documentation/testing/ at all. An agent starting from AGENTS.md -- which its own first line calls the canonical guide -- could read every mechanic for driving the app and never learn that a 410-line capture-to-export checklist exists, with per-platform sections and a results log meant to be appended to. The repo already solved this shape for releases: the Release flow section carries "Full operational guide ... read it before touching a release". Same treatment here, for writing-tests.md and the checklist, plus native-cursor-diagnostics.md for cursor work. Pointers only, no content moved -- the checklist stays the place that says what to run, this stays the place that says how. --- AGENTS.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/AGENTS.md b/AGENTS.md index 286e2855b..00932d9e4 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -76,11 +76,14 @@ every edit is the main way an agent turns a 5-minute task into a 30-minute one, - E2E tests are in `tests/e2e/` (Playwright). Some specs are platform-specific (e.g. `windows-native-checklist.spec.ts`). - Add a test for every new behavior in the same package as the code under test. - All tests must pass before opening a PR. CI runs `npm run test` on every PR. +- **Which kind of test to write, and where: [`technical-documentation/testing/writing-tests.md`](technical-documentation/testing/writing-tests.md).** ## Desktop E2E testing with computer-use Unit/browser tests can't exercise real capture (native screen recording, a physical webcam, the tray). To verify a recording/editor feature end to end, drive the actual Electron app with the **computer-use** MCP (screenshot + click/type on the desktop). This is the required "manual smoke test on real Windows/macOS" for native changes. +This section is the *mechanics*. **What to actually run is [`technical-documentation/testing/manual-e2e-checklist.md`](technical-documentation/testing/manual-e2e-checklist.md)** — the capture-to-export pass, per-platform sections, and a results log to append to. Run it before promoting a release candidate and after any change to native capture, preview or export. For cursor work specifically, [`native-cursor-diagnostics.md`](technical-documentation/testing/native-cursor-diagnostics.md) gets you sidecars and reports without a full record-edit-export cycle. The checklist links back here for the mechanics below; the pairing only works if you know both halves exist. + **Launch the app** - Normal: `npm run dev` — Vite serves the renderer and `vite-plugin-electron` opens the Electron window. The main process logs `Global shortcut registered: CommandOrControl+Shift+O` when ready (Ctrl/Cmd+Shift+O toggles the HUD). From 5dd53ef6a99548321780ce3c85227dc0f4d7e37c Mon Sep 17 00:00:00 2001 From: Etienne Lescot Date: Thu, 13 Aug 2026 23:29:10 +0200 Subject: [PATCH 06/10] docs(testing): log the 1.9.5-rc.1 Windows pass in the results table The table has had one row since July and asks for the run to be recorded. This pass was run and not recorded, which is the same failure as not running it: the next person cannot tell what was covered. Records what the shipped artifact actually did (fragmented MP4 confirmed, 48 fragments over 47.6s), the defect found and where it was fixed, and the finding that matters most for anyone reaching for this checklist next -- a dev build cannot answer a native question, because the prebuilt worktree helper predated the change under test and ran the old path without a word. --- technical-documentation/testing/manual-e2e-checklist.md | 1 + 1 file changed, 1 insertion(+) diff --git a/technical-documentation/testing/manual-e2e-checklist.md b/technical-documentation/testing/manual-e2e-checklist.md index f1375ff73..d3a8fc99d 100644 --- a/technical-documentation/testing/manual-e2e-checklist.md +++ b/technical-documentation/testing/manual-e2e-checklist.md @@ -406,5 +406,6 @@ The agent may only call the fixed tool set in [ai-agent.md](../architecture/ai-a | Date | Build / tag | Platform | Pass/fail | Notes | |------|-------------|----------|-----------|-------| | 2026-07-31 | dev build, `claude/e2e-tests-v1-8-0-474894` (e9578f09) | macOS 26.5, M1 | Partial — 1 defect | Ran launch/HUD, media, modifier anchoring, and export. **Defect: a dangling asset blanks the preview.** Modifier anchoring across a reorder verified in preview and in the exported frames. macOS export produced 1280×720 h264 + AAC at ~2× realtime. Chat sections skipped: no AI provider configured. HUD drag not runnable under computer-use (drop point is the desktop). | +| 2026-08-13 | installed `v1.9.5-rc.1` | Windows 11 26200, 1920×1080 @ 100% | Partial — 1 defect | Ran launch/HUD, source selection, recording, stop, editor open. Fragmented MP4 confirmed on the shipped artifact: 48 `moof`+`mdat` pairs over 47.6s, `mvex` present, `mfra` on clean stop. **Defect: a recording that survives a helper kill is thrown away by the app** — killing `wgc-capture.exe` mid-recording leaves a fully decodable 41s file (2460 packets, `ffmpeg -f null -` exit 0) with no `.session.json` and no `.cursor.json`, and stop answers "The recording could not be saved". Fixed in #363, re-verified end to end. Truncation ablation at 60%: plain MP4 unreadable, fragmented plays 29s. **A dev build cannot test any of this** — the prebuilt worktree helper predated the change and silently ran the old path. Editor/export/chat sections not run. | | | | | | | | | | | | | From bf6ab18f02e80e941349fb20fbfe8e9050321a64 Mon Sep 17 00:00:00 2001 From: Etienne Lescot Date: Thu, 13 Aug 2026 23:33:48 +0200 Subject: [PATCH 07/10] docs(testing): the manual tester is an agent, so say what not to drive it with I had left this fact out of the checklist on the reasoning that a manual tester uses a real cursor by definition. That is only true of a human. "Manual" here means an agent holding the mouse, and an agent has a choice a human does not: it can drive the same real app through CDP. That choice is the failure. Injected input arrives below the OS hit-test, so on Windows and macOS -- where the HUD is input-transparent until a real cursor move lifts it -- a Playwright click fires the DOM handler and returns green while the path a user takes was never exercised. Injection is also the faster-looking option, which is what makes it worth an explicit prohibition rather than an implication. Step 1 named the tool and contrasted it with a browser shim; the shim was never the temptation. Two prerequisites promoted next to it, both of which silently void a run rather than failing it: the prebuilt helpers are frozen and a stale one exercises the old path, and the access resolver cannot see a dev build until it is running, while granting the installed name instead reports success and leaves the window masked. --- technical-documentation/testing/manual-e2e-checklist.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/technical-documentation/testing/manual-e2e-checklist.md b/technical-documentation/testing/manual-e2e-checklist.md index d3a8fc99d..678b831f8 100644 --- a/technical-documentation/testing/manual-e2e-checklist.md +++ b/technical-documentation/testing/manual-e2e-checklist.md @@ -6,10 +6,12 @@ Sections marked **v1.8.0** cover what this release changed: chat-driven editing ## How to run this -1. Drive the real Electron app with computer-use, not a browser shim. Start a dev build with `npm run dev`, or launch the packaged build under test. +1. Drive the real Electron app with computer-use — real OS mouse and keyboard events. Start a dev build with `npm run dev`, or launch the packaged build under test. + + "Manual" here usually means an agent holding the mouse, so the tempting shortcut is not a browser shim: it is driving the real app through CDP instead. **Do not.** Playwright's `.click()`, `javascript_tool`-dispatched pointer events and anything else synthesised into the renderer arrive *below* the OS hit-test. On Windows and macOS the HUD is input-transparent until a real cursor move lifts it, so an injected click fires the DOM handler and comes back green while the path a user actually takes was never exercised at all. Injection is faster, and it is the one thing that can make this entire checklist mean nothing. 2. The app is single-instance per `userData` path. If a leftover Electron/OpenScreen process still holds the lock, stop that process before relaunching; a second launch can exit successfully without opening a window. The lock is held by the OS and is released when the process dies, so there is nothing to delete on disk. -3. From a worktree, link or junction `node_modules` to the main checkout and provide the prebuilt native capture binaries for the platform before starting the dev build. -4. Grant computer-use access to the process name that actually owns the window: `electron.exe` or `Electron.app` for a dev build, and `Openscreen.exe` or `Openscreen.app` for a packaged build. Do not grant access only to the installed app name when testing a dev build. +3. From a worktree, link or junction `node_modules` to the main checkout and provide the prebuilt native capture binaries for the platform before starting the dev build. **Date those binaries against the change you came to test.** Nothing rebuilds them, so an older helper runs the old code path in silence: the recording succeeds and the thing you were checking for is simply absent. See AGENTS.md for how to check one and how to rebuild it; when you cannot, test the CI-built artifact, because a dev build cannot answer a native question. +4. Grant computer-use access to the process name that actually owns the window: `electron.exe` or `Electron.app` for a dev build, and `Openscreen.exe` or `Openscreen.app` for a packaged build. Do not grant access only to the installed app name when testing a dev build — it resolves to the *installed* executable and reports success while the dev window stays masked. This is step 4 and not step 1 for a reason: a dev build is not an installed app, so the resolver cannot find it until it is running and owns a window, and one unresolvable name voids the whole request. 5. Read [AGENTS.md](../../AGENTS.md) for the computer-use mechanics, screenshot permissions, tray interaction, and cleanup procedure. Read one check, perform it, observe the result, then continue; close each modal or popover with `Esc` before the next check. 6. The recording HUD is protected from capture by default and is invisible in screenshots. For this session only, launch with `OPENSCREEN_DISABLE_CONTENT_PROTECTION=1`; this is the environment variable checked before `setContentProtection(true)`. Unset it before making any recording whose HUD must not appear in the video. 7. A preview screenshot is downscaled. Settle every pixel-level question by exporting a frame and measuring the exported frame, not by judging fine edges, corners, shadows, or alignment from the preview screenshot. From e87533b6483688cf132d0121e88b9f78db022613 Mon Sep 17 00:00:00 2001 From: Etienne Lescot Date: Thu, 13 Aug 2026 23:39:15 +0200 Subject: [PATCH 08/10] docs(testing): give the editor sections their own reason not to inject The prohibition I just added was argued entirely from the HUD being input-transparent. That is true, and it is also HUD-only: the HUD and the countdown overlay are the only click-through windows, the editor is `transparent: false` and never calls setIgnoreMouseEvents, and an injected click there really does reach the handler a user would. Which means an agent that reads the reason, clears the HUD sections and then thinks about the ~350 editor checks can conclude, correctly from what was written, that injection is fine for the rest. That guts the document. The editor's reason is different and is in this file's own first line: it covers what unit, browser and Playwright tests cannot reach. Driving it the way those tests already drive it re-runs coverage that exists and writes "passed" beside the parts nothing checked. --- technical-documentation/testing/manual-e2e-checklist.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/technical-documentation/testing/manual-e2e-checklist.md b/technical-documentation/testing/manual-e2e-checklist.md index 678b831f8..40511ce90 100644 --- a/technical-documentation/testing/manual-e2e-checklist.md +++ b/technical-documentation/testing/manual-e2e-checklist.md @@ -8,7 +8,9 @@ Sections marked **v1.8.0** cover what this release changed: chat-driven editing 1. Drive the real Electron app with computer-use — real OS mouse and keyboard events. Start a dev build with `npm run dev`, or launch the packaged build under test. - "Manual" here usually means an agent holding the mouse, so the tempting shortcut is not a browser shim: it is driving the real app through CDP instead. **Do not.** Playwright's `.click()`, `javascript_tool`-dispatched pointer events and anything else synthesised into the renderer arrive *below* the OS hit-test. On Windows and macOS the HUD is input-transparent until a real cursor move lifts it, so an injected click fires the DOM handler and comes back green while the path a user actually takes was never exercised at all. Injection is faster, and it is the one thing that can make this entire checklist mean nothing. + "Manual" here usually means an agent holding the mouse, so the tempting shortcut is not a browser shim: it is driving the real app through CDP instead. **Do not.** Playwright's `.click()`, `javascript_tool`-dispatched pointer events and anything else synthesised into the renderer arrive *below* the OS hit-test. On Windows and macOS the HUD is input-transparent until a real cursor move lifts it, so an injected click fires the DOM handler and comes back green while the path a user actually takes was never exercised at all. + + That trap is specific to the HUD and the countdown overlay — they are the only click-through windows; the editor is an ordinary one, and an injected click there does reach the handler a user would reach. The reason not to inject in the editor either is the first line of this file: this checklist covers what unit, browser and **Playwright** tests cannot. Drive it the way those tests already drive it and you have re-run the coverage you had, then written "passed" beside the parts nothing checked. 2. The app is single-instance per `userData` path. If a leftover Electron/OpenScreen process still holds the lock, stop that process before relaunching; a second launch can exit successfully without opening a window. The lock is held by the OS and is released when the process dies, so there is nothing to delete on disk. 3. From a worktree, link or junction `node_modules` to the main checkout and provide the prebuilt native capture binaries for the platform before starting the dev build. **Date those binaries against the change you came to test.** Nothing rebuilds them, so an older helper runs the old code path in silence: the recording succeeds and the thing you were checking for is simply absent. See AGENTS.md for how to check one and how to rebuild it; when you cannot, test the CI-built artifact, because a dev build cannot answer a native question. 4. Grant computer-use access to the process name that actually owns the window: `electron.exe` or `Electron.app` for a dev build, and `Openscreen.exe` or `Openscreen.app` for a packaged build. Do not grant access only to the installed app name when testing a dev build — it resolves to the *installed* executable and reports success while the dev window stays masked. This is step 4 and not step 1 for a reason: a dev build is not an installed app, so the resolver cannot find it until it is running and owns a window, and one unresolvable name voids the whole request. From bd7cef891edee200be1c1931ad0fd990a6b7d820 Mon Sep 17 00:00:00 2001 From: Etienne Lescot Date: Thu, 13 Aug 2026 23:46:08 +0200 Subject: [PATCH 09/10] docs(agents): orient an agent toward computer-use before it needs to know MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three ordering and framing problems, all of which only bite someone reading this front to back and acting as they go. The content-protection flag was documented at line 109 and the launch step is at line 89. Environment variables are set at launch. By the time you reach the explanation you have already started the app without it, screenshotted, found no HUD, and started looking for a bug. It is a module-scope constant read once as the main process loads (`electron/windows.ts:20`), so there is no recovery short of relaunching — which is exactly why it belongs in the launch step, with the log line that confirms it took. The section opened by framing computer-use as the answer to real capture — screen recording, webcam, tray. That is what forces it, not what it covers: the checklist it points at runs the editor, timeline, regions, transcript, export, settings and persistence the same way. An agent reading the old sentence would use computer-use for the capture checks and reach for something faster afterwards. And "Testing instructions" listed Vitest and Playwright and stopped, with no path to the computer-use section below it. Whoever reads only that section concludes Playwright is where e2e ends. It now says what Playwright structurally cannot reach, and links onward. --- AGENTS.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/AGENTS.md b/AGENTS.md index 00932d9e4..e0e10eb20 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -74,19 +74,21 @@ every edit is the main way an agent turns a 5-minute task into a 30-minute one, every Windows and macOS machine — `electron/recording/webm-seek-index.test.ts` is the worked example. - E2E tests are in `tests/e2e/` (Playwright). Some specs are platform-specific (e.g. `windows-native-checklist.spec.ts`). +- **Playwright is not the end of the e2e story.** It drives the app through CDP, which cannot reach real capture, a real webcam, the tray, or the click-through HUD. Everything those miss is covered by a manual pass driven with computer-use — see [Desktop E2E testing with computer-use](#desktop-e2e-testing-with-computer-use) below, which is required for native changes and before promoting a release candidate. - Add a test for every new behavior in the same package as the code under test. - All tests must pass before opening a PR. CI runs `npm run test` on every PR. - **Which kind of test to write, and where: [`technical-documentation/testing/writing-tests.md`](technical-documentation/testing/writing-tests.md).** ## Desktop E2E testing with computer-use -Unit/browser tests can't exercise real capture (native screen recording, a physical webcam, the tray). To verify a recording/editor feature end to end, drive the actual Electron app with the **computer-use** MCP (screenshot + click/type on the desktop). This is the required "manual smoke test on real Windows/macOS" for native changes. +**Computer-use is how the manual end-to-end pass is driven — all of it, not only the native parts.** Real capture is what forces it (native screen recording, a physical webcam, the tray: no unit or browser test reaches those), but once the app is up you drive everything the same way — editor, timeline, regions, transcript, export, settings, persistence. Screenshot and click/type on the desktop, through the **computer-use** MCP, against the actual Electron app. This is the required "manual smoke test on real Windows/macOS" for native changes, and the only mode in which the checklist below means anything. This section is the *mechanics*. **What to actually run is [`technical-documentation/testing/manual-e2e-checklist.md`](technical-documentation/testing/manual-e2e-checklist.md)** — the capture-to-export pass, per-platform sections, and a results log to append to. Run it before promoting a release candidate and after any change to native capture, preview or export. For cursor work specifically, [`native-cursor-diagnostics.md`](technical-documentation/testing/native-cursor-diagnostics.md) gets you sidecars and reports without a full record-edit-export cycle. The checklist links back here for the mechanics below; the pairing only works if you know both halves exist. **Launch the app** - Normal: `npm run dev` — Vite serves the renderer and `vite-plugin-electron` opens the Electron window. The main process logs `Global shortcut registered: CommandOrControl+Shift+O` when ready (Ctrl/Cmd+Shift+O toggles the HUD). +- **Set `OPENSCREEN_DISABLE_CONTENT_PROTECTION=1` in the environment you launch from, or the HUD is invisible in every screenshot you take.** It is a module-scope constant (`electron/windows.ts:20`), read once as the main process loads, so it cannot be turned on afterwards — you relaunch or you work blind. The main process prints `[content-protection] OFF for the HUD window` when it took effect; if that line is missing, stop and relaunch rather than hunting a HUD you will never see. What it does and when to unset it: the HUD notes below. - The app is single-instance through `app.requestSingleInstanceLock()`, which keys on the `userData` path. If a leftover Electron process still holds it, a new launch quits silently (exit 0, no window) — kill leftover `electron` processes before relaunching. The lock is held by the OS and dies with the process, so there is nothing to clean up on disk. A dev build and the installed `Openscreen` resolve different `userData` paths and can run side by side. - **From a git worktree** (no `node_modules`/native binaries): junction/symlink `node_modules` from the main checkout (deps are usually identical — check `package-lock.json`), and copy the prebuilt native capture binaries from `electron/native/bin//` (gitignored — rebuilding needs the full VS/Xcode toolchain). Then `npm run dev` works normally. - **Those binaries are frozen at whenever someone last built them, and nothing warns you.** They are not rebuilt by `npm run dev` or `npm run build`, so a helper older than the native change you came to test will run happily and silently exercise the old code path — the recording succeeds, and the thing you wanted to see is simply absent. Before trusting any native result, date the binary against the commit and search it for a string the change introduced — from the repo root: From 65ce28620466912e949b6863e97983b235bbe5f9 Mon Sep 17 00:00:00 2001 From: Etienne Lescot Date: Thu, 13 Aug 2026 23:52:24 +0200 Subject: [PATCH 10/10] docs(agents): show the control search, not just require it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The prose demanded a control string and the example showed only the positive search, which makes the control read as optional advice. It is the load-bearing half: without it a broken search is indistinguishable from a stale binary, and that is not hypothetical — `strings … | grep` in Git Bash produced five confident false negatives earlier in this PR, including against the helper that does contain the change. Both commands now appear, with the repository-root path, and the fence is tagged. Outcomes measured against the two helpers the section is about rather than reasoned about: stale gives no match then HIT, current gives HIT twice. --- AGENTS.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index e0e10eb20..e966eeeb3 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -93,11 +93,14 @@ This section is the *mechanics*. **What to actually run is [`technical-documenta - **From a git worktree** (no `node_modules`/native binaries): junction/symlink `node_modules` from the main checkout (deps are usually identical — check `package-lock.json`), and copy the prebuilt native capture binaries from `electron/native/bin//` (gitignored — rebuilding needs the full VS/Xcode toolchain). Then `npm run dev` works normally. - **Those binaries are frozen at whenever someone last built them, and nothing warns you.** They are not rebuilt by `npm run dev` or `npm run build`, so a helper older than the native change you came to test will run happily and silently exercise the old code path — the recording succeeds, and the thing you wanted to see is simply absent. Before trusting any native result, date the binary against the commit and search it for a string the change introduced — from the repo root: - ``` + ```powershell + # the string the change introduced — absent from a stale helper findstr /M /C:"fragmented-mp4" electron\native\bin\win32-x64\wgc-capture.exe + # the control — present in every helper, stale or not + findstr /M /C:"encoder-selection" electron\native\bin\win32-x64\wgc-capture.exe ``` - `findstr` handles binaries and ships with Windows, whereas Git Bash has **no `strings`**, so `strings … | grep` there returns nothing and reads as a confident *absent* for every binary you point it at. Always search a control string the old binary also has (`encoder-selection`) so a broken search cannot masquerade as a stale binary. If it is stale, rebuild it with `npm run build:native:win` (or `:mac` / `:linux`) — that is the only thing that compiles a helper. Without the toolchain, test the CI-built artifact instead; a dev build cannot answer the question. + Run **both**. Only the second tells "the binary is stale" apart from "my search is broken", and that distinction is not hypothetical: `findstr` handles binaries and ships with Windows, but Git Bash has **no `strings`**, so `strings … | grep` there returns nothing and reads as a confident *absent* for every binary you point it at. Measured against the two helpers this section is about — stale: no match, then HIT; current: HIT, HIT. A control that does not hit means you learned nothing about the binary. If it is stale, rebuild it with `npm run build:native:win` (or `:mac` / `:linux`) — that is the only thing that compiles a helper. Without the toolchain, test the CI-built artifact instead; a dev build cannot answer the question. **Granting access**