Skip to content

feat: add MediaBackend-v2-aware test harness - #168

Draft
JarbasAl wants to merge 1 commit into
devfrom
feat/media-backend-v2-harness
Draft

JarbasAl wants to merge 1 commit into
devfrom
feat/media-backend-v2-harness

Conversation

@JarbasAl

@JarbasAl JarbasAl commented Sep 4, 2026

Copy link
Copy Markdown
Member

🤖 Auto-generated by Claude Fable 5 (claude-fable-5) via Claude Code — NOT human-reviewed. Verify before acting.

ovos-media's feat/media-backend-v2-port branch replaces the v1 MediaBackend contract with a report()/bind_event_reporter() one: plugins report physical PlaybackEvents (TRACK_START, PAUSED, RESUMED, STOPPED, END_OF_MEDIA, ERROR) instead of emitting bus messages themselves, load_track returns a bool instead of a None, and stop() becomes a concrete template method that delegates to a plugin's _stop(). The daemon (ovos-media's BaseMediaService) is now the sole translator from those events to ovos.common_play.* wire messages, via _handle_backend_event — and it has dropped handle_media_state_change entirely. ovoscope's media harnesses predate this port and needed updating to keep exercising the real daemon shape.

This adds MediaBackendHarness (ovoscope/media_backend.py), a harness for a plugin author testing their own MediaBackend v2 class in isolation: it binds a capturing spy reporter onto a backend instance and asserts the event sequence and per-event data it reports, without any daemon involved. The v2 template it targets (ovos_plugin_manager.templates.media) only exists on the unreleased OPM branch, so both the module import and the ovoscope/__init__.py re-export are guarded: import ovoscope is always safe against a released ovos-plugin-manager, and MediaBackendHarness() raises a clear ImportError only when actually constructed without the v2 template installed. Its own test module carries the equivalent guard at collection time, since its plugin stand-in subclasses the v2 template directly.

It also adds MockOCPBackendV2 (ovoscope/media.py), the v2 counterpart to the existing MockOCPBackend (kept unmodified, still importable, for plugin repos that haven't ported yet), and makes it OCPPlayerHarness's default injected backend. OCPPlayerHarness.__enter__ now binds each injected backend's event reporter, and guards the v1-only set_track_start_callback call on whether the service it reads track_start from actually has that attribute — not on the backend, which a v1 backend always has regardless of which ovos-media build is installed; guarding on the backend alone still crashed with AttributeError against a v2 build. In the default (no backend_factory) mode, audio_service is a MagicMock with no real _handle_backend_event behind it, so the backend's reporter is bound to a small shim reproducing the daemon's END_OF_MEDIA/ERROR wire translation — the only two events that mode's own control methods can actually drive — with its docstring naming the two daemon guards (currency, staleness) it deliberately skips, so a test needing those knows to go through the real daemon (backend_factory) instead. MockOCPBackendV2.load_track now clears current_uri on a failed load, so a failed load doesn't leave a stale, previous track's uri behind for a following simulate_invalid_stream()/report_track_end() to report against.

Every v2-only test class and cell is now gated on whether the v2 template actually imported, not merely on ovos-media being installed, and the backend_factory stand-in several pre-existing cells share (_RecordingBackend) now has a v1 variant selected the same way — including fixing that variant to emit LOADED_MEDIA on the topic a real v1 BaseMediaService actually subscribes to (ovos.common_play.media.state), not the per-namespace one MockOCPBackend emits, which no real BaseMediaService instance has ever listened on. The full suite now collects and runs clean against both a released ovos-plugin-manager (v2 cells skip) and the unreleased branches (v2 cells run) — collection used to abort entirely against released deps because a test module imported the v2 template unguarded at collection time.

Fail-before (source reverted with the test kept, run foreground, then restored): the set_track_start_callback guard turned 16 of 42 test_media.py tests failing with AttributeError across every OCPPlayerHarness-based class; the default-mode event-reporter shim's two new cells failed with no wire message emitted at all when reverted to binding straight into the MagicMock's _handle_backend_event; test_failed_load_clears_current_uri failed with the current_uri clear removed.

Full suite against released PyPI deps on python 3.10, foreground: 655 passed, 55 skipped, 0 failed. The 55 skips are exactly the v2-only cells gated off without the v2 template.

No workflow installs from git. coverage-pages.yml and release-workflow.yml's build_tests job gain the media, listener and bench extras, which they lacked, and that is the whole CI change. The released ovos-plugin-manager is 2.12.1a1 (latest alpha; its templates/media.py is still the v1 one, no PlaybackEvent, no bind_event_reporter) and released ovos-media is 2.2.5a1, so neither carries the v2 backend template. The [media] floor therefore stays at ovos-media>=2.0.0a9 and the v2 cells skip on the template guard they already carry; they start running once ovos-plugin-manager publishes the v2 template and the floor can be raised to it.

That also settles the build_tests (3.10) failure on TestOCPHarnessWithoutHandlePlay::test_enter_and_play_succeed_when_handle_play_is_absent: it was specific to the git-branch ovos-media those pre_install_pip lines installed, not to dev. Against released deps that cell passes, as does the rest of the suite.

@coderabbitai

coderabbitai Bot commented Sep 4, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Fresh off the press! I've got some check results for you. 🗞️

I've aggregated the results of the automated checks for this PR below.

🔍 Lint

A quick update on the progress of your PR checks. 📈

ruff: issues found — see job log

📊 Coverage

Checking the structural integrity of our tests. 🏗️

⚠️ 61.5% total coverage

Files below 80% coverage (16 files)
File Coverage Missing lines
ovoscope/media_backend.py 0.0% 53
ovoscope/simple_listener.py 0.0% 63
ovoscope/version.py 0.0% 5
ovoscope/classic_listener.py 18.2% 117
ovoscope/intent_cases.py 22.3% 153
ovoscope/media.py 44.7% 167
ovoscope/pytest_plugin.py 47.0% 210
ovoscope/cli.py 47.8% 132
ovoscope/ocp.py 47.9% 61
ovoscope/tts_intelligibility.py 49.0% 102
ovoscope/e2e.py 56.2% 67
ovoscope/listener.py 57.1% 127
ovoscope/voice_loop.py 59.0% 119
ovoscope/__init__.py 63.2% 424
ovoscope/audio.py 64.4% 126
ovoscope/media_provider.py 67.6% 23

Full report: download the coverage-report artifact.

⚖️ License Check

I've verified the license compliance for your changes. ✅

✅ No license violations found.

Policy: Apache 2.0 (universal donor). StrongCopyleft / NetworkCopyleft / WeakCopyleft / Other / Error categories fail. MPL allowed.

📋 Repo Health

Scanning for any signs of 'comment' bad breath. 🌬️

✅ All required files present.

Latest Version: 1.8.1a1

ovoscope/version.py — Version file
README.md — README
LICENSE — License file
pyproject.toml — pyproject.toml
⚠️ setup.py — setup.py
CHANGELOG.md — Changelog
ovoscope/version.py has valid version block markers

🔒 Security (pip-audit)

Looking for any Trojan horses in the dependencies. 🐎

✅ No known vulnerabilities found (129 packages scanned).

🏷️ Release Preview

I've performed a final polish on the release notes. 🧼

Current: 1.8.1a1Next: 1.9.0a1

Signal Value
Label feature
PR title feat: add MediaBackend-v2-aware test harness
Bump minor

✅ PR title follows conventional commit format.


🚀 Release Channel Compatibility

Predicted next version: 1.9.0a1

Channel Status Note Current Constraint
Stable Not in channel -
Testing Too new (must be <1.0.0) ovoscope>=0.7.2,<1.0.0
Alpha Compatible ovoscope>=1.6.15a1

🔨 Build Tests

The compiler has spoken! Here is the verdict. 📜

✅ All versions pass

Python Build Install Tests
3.10
3.11
3.12
3.13
3.14

Powered by OVOS scripts and a bit of magic. ✨

@JarbasAl
JarbasAl force-pushed the feat/media-backend-v2-harness branch from cf5ef23 to 39f2275 Compare September 4, 2026 15:59
@github-actions github-actions Bot added feature and removed feature labels Sep 4, 2026
@JarbasAl
JarbasAl force-pushed the feat/media-backend-v2-harness branch from 39f2275 to eb6b48b Compare September 4, 2026 16:31
@github-actions github-actions Bot added feature and removed feature labels Sep 4, 2026
ovos-media's feat/media-backend-v2-port branch replaces the v1 MediaBackend
contract (bus-emitting backends, handle_media_state_change) with a
report()/bind_event_reporter() contract, and drops handle_media_state_change
entirely in favour of BaseMediaService._handle_backend_event. ovoscope's
media harnesses predate that port and needed updating to keep testing the
real daemon shape rather than a v1 stand-in.

Adds MediaBackendHarness (ovoscope/media_backend.py), a plugin-author-facing
harness that binds a capturing spy reporter onto a single MediaBackend v2
plugin instance and asserts the PlaybackEvent sequence/data it reports,
without a daemon in front of it. The v2 template it targets
(ovos_plugin_manager.templates.media) only exists on the unreleased OPM
branch, so both the module import and the ovoscope/__init__.py re-export
are guarded: importing the module (and `import ovoscope`) is always safe,
and MediaBackendHarness() raises a clear ImportError only when actually
constructed against a released ovos-plugin-manager. test_media_backend_harness.py
carries the same guard at module scope (pytestmark skipif), since its
_FakePlugin stand-in subclasses the v2 template directly.

Adds MockOCPBackendV2 (ovoscope/media.py), the v2 counterpart to the existing
v1 MockOCPBackend (kept, unmodified, for unported plugins), and makes it the
OCPPlayerHarness default. Fixes OCPPlayerHarness.__enter__ to bind each
injected backend's event reporter, and to guard the v1-only
set_track_start_callback call on whether the *service* it reads
track_start from actually has that attribute (not on the backend, which a
v1 backend always has regardless of which ovos-media build is installed) —
guarding on the backend alone still crashed with AttributeError on a v2
ovos-media build. In default (no backend_factory) mode, audio_service is a
MagicMock with no real _handle_backend_event behind it, so the reporter is
bound to a small shim that reproduces the real daemon's END_OF_MEDIA/ERROR
wire translation for simulate_track_end()/simulate_invalid_stream() (the
only two events that mode's control-method API can actually drive); its
docstring names the two daemon guards (currency, staleness) it deliberately
doesn't reproduce, since a cell exercising those needs the real daemon
instead. Also drops the now-nonexistent handle_media_state_change bus
registration and a since-stale comment referencing it.

MockOCPBackendV2.load_track now clears current_uri on a failed load, so a
failed load doesn't leave a stale, PREVIOUS track's uri in place for a
following simulate_invalid_stream()/report_track_end() to report against.

Test coverage keeps working against a released ovos-plugin-manager (no v2
template): test_media.py's v2-only classes/cells are individually gated on
whether the v2 template actually imported (not merely on ovos-media being
installed), and _RecordingBackend (the backend_factory stand-in used by
several pre-existing cells) now has a v1 variant selected the same way, so
those cells stay green against a released, still-v1-shaped ovos-media
instead of silently trying to speak v2 to it. The v1 variant emits
LOADED_MEDIA on ovos.common_play.media.state, the topic a real v1
BaseMediaService actually subscribes handle_media_state_change to (not the
per-namespace ovos.{namespace}.service.media.state MockOCPBackend emits,
which no real BaseMediaService instance ever listens on).

Fail-before (temporarily reverted the source change, kept the test, ran
foreground, reverted the revert):
- set_track_start_callback hasattr guard: 16 of 42 test_media.py tests
  failed (AttributeError) across every OCPPlayerHarness-based class.
- Default-mode event-reporter shim: the two new
  TestOCPHarnessMediaBackendV2DefaultMode cells failed (no wire message
  emitted at all) with the shim reverted to binding straight into the
  MagicMock's _handle_backend_event.
- MockOCPBackendV2.load_track current_uri clearing:
  test_failed_load_clears_current_uri failed with the clear removed.

Full ovoscope/test/unittests suite against released PyPI deps, python 3.10,
foreground: 655 passed, 55 skipped, 0 failed. The 55 skips are the v2-only
cells gated off without the v2 template.

CI: coverage-pages.yml and release-workflow.yml's build_tests job gain the
media, listener and bench extras, which they lacked. No workflow installs
from git: released ovos-plugin-manager (2.12.1a1, latest alpha) still ships
the v1 templates/media.py with no PlaybackEvent and no bind_event_reporter,
and released ovos-media is 2.2.5a1, so the [media] floor stays where it is
and the v2-only cells skip on the guard they already carry. They start
running once ovos-plugin-manager publishes the v2 template.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@JarbasAl
JarbasAl force-pushed the feat/media-backend-v2-harness branch from eb6b48b to 15d0dba Compare September 8, 2026 11:09
@github-actions github-actions Bot added feature and removed feature labels Sep 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant