feat: add audio capture support (issue #4) - #71
Merged
Merged
Conversation
Extends ScreenSight with loopback (system-audio) capture — lets agents hear what the user is playing, gated by the master switch and a separate opt-in env var (SCREENSIGHT_ENABLE_AUDIO=1) for privacy. - New capture/audio.py: record_system_audio() via soundcard WASAPI/Pulse loopback → 16-bit PCM WAV with stdlib wave; deps lazily imported so the screenshot path is unaffected when [audio] extras aren't installed. - New MCP tool screen_capture_audio(duration, question) — 9th tool, returns a FastMCP Audio content block + text context. - New CLI command: screensight capture-audio [--duration N]. - config.py: AUDIO_PATH, duration/sample-rate constants, audio_enabled(). - core.py: capture_audio() + CaptureAudioOutcome, double-gated (master switch + SCREENSIGHT_ENABLE_AUDIO), duration clamped to 1–30s. - state.py: turn_off() deletes audio.wav alongside frame.jpg. - pyproject.toml: optional [audio] extra (soundcard>=0.4.3, numpy>=1.24). - Tests: 6 new tests in test_audio.py; test_mcp_server updated 8→9 tools with schema and content-block tests for screen_capture_audio. - Docs: cli.md, mcp-tools.md, configuration.md updated with per-OS loopback notes (Windows WASAPI ✅, Linux PulseAudio ✅, macOS/WSL⚠️ ). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HGJmVMdVAbyTnUV2KYWfF7
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
screen_capture_audioMCP tool (9th tool) returns a WAV audio content block.screensight capture-audio [--duration N]CLI command.SCREENSIGHT_ENABLE_AUDIO=1env var must both be on; audio is off by default.pip install 'screensight[audio]'):soundcard+numpy; the core screenshot path has no new dependencies.audio.wavis cleaned up onscreensight off, matchingframe.jpghygiene.Changes
src/screensight/capture/audio.pyAudioResult,record_system_audio()via soundcard loopback → 16-bit PCM WAVsrc/screensight/config.pyAUDIO_PATH, duration/rate constants,audio_enabled()src/screensight/core.pyCaptureAudioOutcome,capture_audio()with double gate + duration clampsrc/screensight/mcp_server.pyscreen_capture_audiotool (9th); FastMCPAudioblock with base64 fallbacksrc/screensight/__main__.pycapture-audiosubcommandsrc/screensight/state.pyturn_off()deletesaudio.wavpyproject.toml[audio]extratests/test_audio.pytests/test_mcp_server.pyscreen_capture_audiodocs/+CHANGELOG.mdPlatform support
Test plan
pytest— 58 tests pass (6 new intest_audio.py, 2 updated intest_mcp_server.py)ruff check+ruff format --check— cleanscreensight offdeletesaudio.wavCloses #4