Make possible to copy from text in screenshot mode

[wesleymatosdev]

Apple screenshot tool already provides that. I can see myself using Cap way more if this was a thing

ps: I'd love to hear maintainers' input on it and work this feature myself

[linear]

CAP-681

[tembo]

Thanks for the feature request! This would be a nice addition to complement the existing screenshot workflow.

Looking at the codebase, there's currently no OCR/text recognition functionality implemented. The screenshot infrastructure is well-established in crates/recording/src/screenshot.rs for capture and apps/desktop/src-tauri/src/screenshot_editor.rs for the editor backend. The frontend lives in apps/desktop/src/routes/screenshot-editor/.

For implementation, you'd likely want to:

1. Add an OCR dependency - On macOS, you could use Apple's Vision framework via objc2-vision or similar bindings since it already powers Live Text. For Windows, consider Tesseract (tesseract-rs or leptonica-sys) or Windows.Media.Ocr runtime APIs.

2. Integrate with the screenshot pipeline - The screenshot capture returns image data as RGBA bytes (screenshot.rs:capture_screenshot()). You'd process this through the OCR engine before or during the editor phase.

3. Add a UI layer for text selection - The screenshot editor already has annotation layers (AnnotationLayer.tsx) and tool selection. A text recognition layer could follow similar patterns, overlaying selectable text regions on the image.

4. Hook into existing clipboard - There's already write_clipboard_string and related clipboard functions in apps/desktop/src-tauri/src/lib.rs (around line 4715) that handle text copying. The extracted text would use these existing utilities.

Since Cap already has Whisper (whisper-rs) for audio transcription in videos, you might look at apps/desktop/src-tauri/src/captions.rs for patterns on how ML-based processing is integrated into the Tauri app.

The main architectural decision would be whether to run OCR:

• At capture time - Detect text immediately when screenshot is taken
• On-demand - Add a "Copy Text" tool in the screenshot editor that triggers OCR

Happy to see you're interested in contributing this!

[salmonumbrella]

@richiemcilroy I saw you added OCR in 0.5.3. This CleanShot feature that lets you select any area on your screen and copy the text would be killer. Couple of open-source implementations:

• Snapzy: https://github.com/duongductrong/Snapzy/blob/fac1a628abd4577c6230999f5549cceabc26b4d4/Snapzy/Features/Capture/CaptureViewModel.swift#L2048-L2184
• MacShot: https://github.com/sw33tLie/macshot/blob/8ce1a37d8b5189ef9b7ca8588075cd69df0d2765/macshot/UI/Overlay/OverlayWindowController.swift#L616-L632

[richiemcilroy]

Hey @salmonumbrella you can do this in Cap I'm pretty sure. You take a screenshot, and then you can copy the text by highlighting and copying from the screenshot

[salmonumbrella]

@richiemcilroy OCR works, but is a lot of clicks! on cleanshot cmd-shift-2 to highlight a part of the desktop, and text is OCR'd and copied to clipboard w/ push notif. i wish Cap had a shortcut for 'OCR anything'

[salmonumbrella]

@richiemcilroy sorry for the double tag. this feature would make me uninstall CleanShot and use Cap exclusively for screenshots. already did the same with Loom 6 months ago