Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 15 additions & 12 deletions .github/workflows/pytest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,19 +3,22 @@ name: Unit Testing
on: [pull_request]

jobs:
unittest:
runs-on: ubuntu-latest
unittest:
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v2
with:
fetch-depth: 1
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 1

- name: Install uv
uses: astral-sh/setup-uv@v5
- name: Install Rust
uses: dtolnay/rust-toolchain@stable

- name: Set up Python
run: uv sync
- name: Install uv
uses: astral-sh/setup-uv@v5

- name: Run Unit Tests
run: uv run pytest -n auto
- name: Set up Python
run: uv sync

- name: Run Unit Tests
run: uv run pytest -n auto
29 changes: 29 additions & 0 deletions .github/workflows/rust.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
name: Rust CI

on: [pull_request]

jobs:
rust-ci:
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v4
with:
fetch-depth: 1

- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
with:
components: rustfmt, clippy

- name: Rust cache
uses: Swatinem/rust-cache@v2

- name: Check formatting
run: cargo fmt --all -- --check

- name: Clippy
run: cargo clippy --all-targets --all-features -- -D warnings

- name: Tests
run: cargo test --all-features
6 changes: 5 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -15,4 +15,8 @@ assets/
.vscode/
.hypothesis/
prof/
.coverage
.coverage

# Rust build artifacts
target/
Cargo.lock
22 changes: 22 additions & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -19,3 +19,25 @@ repos:
- id: ruff
args: [--fix]
- id: ruff-format

- repo: local
hooks:
- id: cargo-fmt
name: cargo fmt
entry: cargo fmt --all
language: system
types: [rust]
pass_filenames: false
- id: cargo-clippy
name: cargo clippy
entry: cargo clippy --all-targets --all-features -- -D warnings
language: system
types: [rust]
pass_filenames: false
- id: cargo-test
name: cargo test
entry: cargo test --all-features
language: system
types: [rust]
pass_filenames: false
stages: [pre-push]
60 changes: 53 additions & 7 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,20 +13,29 @@ Future metrics (e.g., audio coverage, state-graph coverage) will follow the same

```
.
├── Cargo.toml # Rust manifest (maturin builds the extension)
├── pyproject.toml # Project metadata, maturin build backend
├── src/
│ ├── main.py # CLI entry point (Typer)
│ ├── lib.rs # PyO3 module entry point (Rust)
│ ├── bktree.rs # BK-tree<u64> with POPCNT Hamming distance
│ ├── unionfind.rs # Flat Vec-based union-find
│ ├── monitor.rs # CoverageTracker (BK-tree + UnionFind combined)
│ └── gamecov/
│ ├── __init__.py # Public API re-exports
│ ├── _gamecov_core.pyi # Type stub for Rust extension
│ ├── cov_base.py # Abstract protocols: CoverageItem, Coverage, CoverageMonitor
│ ├── frame.py # Frame dataclass (PIL Image wrapper with average-hash)
│ ├── dedup.py # Deduplication algorithms (pHash, SSIM [deprecated])
│ ├── frame_cov.py # FrameCoverage, FrameMonitor, BKFrameMonitor, BK-tree, UnionFind
│ ├── frame_cov.py # FrameCoverage, FrameMonitor, BKFrameMonitor, RustBKFrameMonitor, BK-tree, UnionFind
│ ├── loader.py # MP4 loading: bulk, lazy (generator), last-n
│ ├── writer.py # MP4 writing: imageio and OpenCV backends
│ ├── stitch.py # Panorama stitching of unique frames
│ ├── generator.py # Hypothesis strategies for property-based testing
│ ├── env.py # Runtime config (RADIUS env var)
│ └── py.typed # PEP 561 marker
├── rust-tests/
│ └── prop_tests.rs # Rust proptest property-based tests
├── tests/
│ ├── test_generators.py # Frame/FrameList generation strategies
│ ├── test_dedup.py # Dedup monotonicity properties
Expand All @@ -35,19 +44,34 @@ Future metrics (e.g., audio coverage, state-graph coverage) will follow the same
│ ├── test_load_write_assets.py# Differential tests across loaders on real videos
│ ├── test_monotone.py # Coverage monotonicity (FrameMonitor & BKFrameMonitor)
│ ├── test_BK_frame_monitor.py # Differential: FrameMonitor vs BKFrameMonitor
│ ├── test_rust_frame_monitor.py # Differential & monotonicity: BKFrameMonitor vs RustBKFrameMonitor
│ └── test_monotone_smb.py # Real-world monotonicity on SMB dataset
├── benchmarks/
│ ├── conftest.py # Session-scoped fixtures (pre-generated FrameCoverage)
│ └── test_bench_monitor.py # Python vs Rust monitor throughput benchmarks
├── assets/
│ ├── videos/ # Small sample MP4s for integration tests
│ └── smb/ # Super Smash Bros recordings for stress tests
├── docs/
│ └── design.md # Architecture and design documentation
├── pyproject.toml # Project metadata, dependencies, tool configs
├── .pre-commit-config.yaml # Pre-commit hooks
├── .github/workflows/ # CI: pytest, mypy, ruff, pylint
├── rustfmt.toml # Rust formatting config
├── .pre-commit-config.yaml # Pre-commit hooks (Python + Rust)
├── .github/workflows/ # CI: pytest, mypy, ruff, pylint, rust (fmt/clippy/test)
├── AGENTS.md # This file
└── README.md # Human-facing documentation
```

### Rust extension (gamecov-core)

The Rust extension is built as part of the package via maturin. The compiled
module is installed as `gamecov._gamecov_core` and provides high-performance
replacements for the BK-tree, union-find, and coverage tracker.

Build the package (includes Rust compilation): `uv sync` or `pip install .`
Run Rust tests independently: `cargo test`
Check Rust formatting: `cargo fmt --all -- --check`
Run Rust linting: `cargo clippy --all-targets --all-features -- -D warnings`

## Design

See [docs/design.md](docs/design.md) for the coverage framework architecture, frame coverage pipeline, BK-tree optimization, and loading strategies.
Expand All @@ -59,7 +83,7 @@ See [docs/design.md](docs/design.md) for the coverage framework architecture, fr
| `cov_base.py` | `CoverageItem`, `Coverage[T]`, `CoverageMonitor[T]` protocols/ABC |
| `frame.py` | `Frame` dataclass (PIL Image + average-hash) |
| `dedup.py` | `is_dup()`, `dedup_unique_frames()`, `dedup_unique_hashes()`, `ssim_dedup()` [deprecated] |
| `frame_cov.py` | `FrameCoverage`, `FrameMonitor`, `BKFrameMonitor`, `get_frame_cov()`, `_UnionFind`, `_BKTree` |
| `frame_cov.py` | `FrameCoverage`, `FrameMonitor`, `BKFrameMonitor`, `RustBKFrameMonitor`, `get_frame_cov()`, `_UnionFind`, `_BKTree` |
| `loader.py` | `load_mp4()`, `load_mp4_lazy()`, `load_mp4_last_n()` |
| `writer.py` | `write_mp4()`, `write_mp4_cv2()` |
| `stitch.py` | `stitch_images()` (panorama via AffineStitcher) |
Expand All @@ -80,7 +104,8 @@ See [docs/design.md](docs/design.md) for the coverage framework architecture, fr
| `opencv-python` | Color conversion, video writing, image processing |
| `scikit-image` | SSIM metric (deprecated path) |
| `stitching` | Panorama stitching via OpenCV features |
| `numpy`, `numba` | Numerical arrays, optional JIT acceleration |
| `numpy` | Numerical arrays |
| `gamecov._gamecov_core` | Built-in Rust extension: BK-tree, union-find, coverage tracker (PyO3/maturin) |
| `returns` | Functional `Result` type for error handling |
| `deprecated` | `@deprecated` decorator |
| `typer-slim` | CLI framework |
Expand All @@ -93,6 +118,7 @@ See [docs/design.md](docs/design.md) for the coverage framework architecture, fr
| `mypy` | Static type checking (strict mode, returns plugin) |
| `ruff` | Linting and formatting |
| `pre-commit` | Pre-commit hook runner |
| `pytest-benchmark` | Performance benchmarking (Python vs Rust) |
| `pytest-xdist` | Parallel test execution (`-n auto`) |
| `pytest-cov` | Coverage reporting |
| `pytest-profiling` | Performance profiling |
Expand All @@ -110,6 +136,7 @@ uv run pytest -n auto
- **Integration** (real assets): `test_load_n.py`, `test_load_write_assets.py`
- **Monotonicity**: `test_monotone.py` (random data), `test_monotone_smb.py` (real SMB recordings)
- **Differential**: `test_BK_frame_monitor.py` (FrameMonitor vs BKFrameMonitor produce identical results)
- **Rust backend**: `test_rust_frame_monitor.py` (differential Python vs Rust, order-independence, monotonicity)

Some tests require assets in `assets/videos/` or `assets/smb/` and will skip if missing.

Expand All @@ -118,6 +145,24 @@ Some tests require assets in `assets/videos/` or `assets/smb/` and will skip if
- `RADIUS` — Hamming distance threshold (default `5`).
- `N_MAX` — Maximum number of recordings to process in monotonicity tests (default `100`).

## Benchmarks

```bash
# Run benchmarks (disabled by default during normal test runs)
uv run pytest benchmarks/ --benchmark-enable

# Group output by backend for side-by-side comparison
uv run pytest benchmarks/ --benchmark-enable --benchmark-group-by=param:backend

# Save results for later comparison
uv run pytest benchmarks/ --benchmark-enable --benchmark-save=baseline
uv run pytest benchmarks/ --benchmark-enable --benchmark-compare=baseline
```

Benchmarks live in `benchmarks/` and are excluded from the normal test suite.
They compare `BKFrameMonitor` (Python) vs `RustBKFrameMonitor` (Rust) throughput
at the monitor level (`add_cov`/`is_seen` operations).

## Development

- Before start working, refresh your knowledge from contents in `.agents` first.
Expand All @@ -129,7 +174,8 @@ Some tests require assets in `assets/videos/` or `assets/smb/` and will skip if
Local variables' types are optional as long as the types can be easily inferred.
- Use f-strings for string interpolation.
- Use `TypedDict`, `Literal`, `Protocol`, and `TypeVar` from `typing` module when appropriate.
- Always run `mypy`, and `ruff` to ensure code quality after updating code in `src/`.
- Always run `mypy` and `ruff` to ensure code quality after updating Python code in `src/`.
- Always run `cargo fmt`, `cargo clippy -- -D warnings`, and `cargo test` after updating Rust code in `src/`.
- Never commit changes or create PRs. Suggest commit messages to the human developer for review after your changes to the codebase.
- Always use `typer` to handle CLI commands.

Expand Down
23 changes: 23 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
[package]
name = "gamecov-core"
version = "0.2.0"
edition = "2021"
description = "Rust-accelerated core for gamecov frame coverage monitoring"

[lib]
name = "gamecov_core"
crate-type = ["cdylib", "rlib"]

[dependencies]
pyo3 = { version = "0.23", features = ["extension-module"] }

[dev-dependencies]
proptest = "1"

[[test]]
name = "prop_tests"
path = "rust-tests/prop_tests.rs"

[profile.release]
lto = "fat"
codegen-units = 1
69 changes: 66 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ Two frames are considered duplicates if the Hamming distance between their perce

## Installation

Requires Python >= 3.11.
Requires Python >= 3.11 and a [Rust toolchain](https://rustup.rs/) (for building from source).

### As a package

Expand All @@ -41,6 +41,8 @@ Or with pip:
pip install git+https://github.com/SecurityLab-UCD/gamecov.git
```

This builds both the Python package and the embedded Rust extension (`gamecov._gamecov_core`) in a single step.

### For development

Clone the repo and sync dependencies:
Expand Down Expand Up @@ -79,6 +81,24 @@ print(f"Total unique frames: {len(monitor.item_seen)}")
print(f"Unique paths: {len(monitor.path_seen)}")
```

### Rust-accelerated monitor

`RustBKFrameMonitor` provides the same interface as `BKFrameMonitor`,
backed by an embedded Rust extension for significantly higher throughput:

```python
from gamecov import FrameCoverage, RustBKFrameMonitor

monitor = RustBKFrameMonitor() # same API as BKFrameMonitor

for recording in recordings:
cov = FrameCoverage(recording)
if not monitor.is_seen(cov):
monitor.add_cov(cov)

print(f"Coverage components: {monitor.coverage_count}")
```

### CLI

```bash
Expand All @@ -90,8 +110,12 @@ uv run python src/main.py --input-mp4-path path/to/video.mp4 --confidence-thresh

### Prerequisites

- Python >= 3.11
- [Rust toolchain](https://rustup.rs/) (stable)
- [uv](https://docs.astral.sh/uv/)

```bash
# Install dependencies
# Install dependencies (builds the Rust extension automatically)
uv sync

# Install pre-commit hooks
Expand All @@ -101,11 +125,24 @@ uv run pre-commit install
### Running Tests

```bash
# Run all tests in parallel
# Run all Python tests in parallel
uv run pytest -n auto

# Run with coverage
uv run pytest -n auto --cov=gamecov

# Run Rust unit and property tests
cargo test
```

### Benchmarks

```bash
# Compare Python vs Rust monitor throughput
uv run pytest benchmarks/ --benchmark-enable

# Side-by-side grouped by backend
uv run pytest benchmarks/ --benchmark-enable --benchmark-group-by=param:backend
```

### Code Quality
Expand All @@ -121,3 +158,29 @@ uv run ruff check src/
### CI

GitHub Actions runs four checks on every PR: `pytest`, `mypy`, `ruff`, and `pylint`.
The CI workflow installs the Rust toolchain before building.

## Performance: Rust vs Python Backend

The embedded Rust extension (`RustBKFrameMonitor`) provides significant speedups
over the pure-Python `BKFrameMonitor` for the core `add_cov`/`is_seen` monitor operations.
The advantage grows with workload size as the BK-tree and union-find structures scale.

Benchmark results (mean time per iteration, lower is better):

| Recordings | Python (ms) | Rust (ms) | Speedup |
| ---------- | ----------- | --------- | ------- |
| 10 | 4.04 | 2.31 | 1.75x |
| 50 | 42.95 | 15.00 | 2.86x |
| 200 | 424.36 | 111.03 | 3.82x |
| 500 | 2,349.74 | 549.40 | 4.28x |

The Rust backend achieves **1.8x -- 4.3x** speedup,
with larger gains at higher workloads where BK-tree traversal and union-find operations dominate.
Each recording contains randomly generated `FrameCoverage` objects with perceptual hashes.

Reproduce these results with:

```bash
uv run pytest benchmarks/ --benchmark-enable --benchmark-group-by=param:backend
```
Empty file added benchmarks/__init__.py
Empty file.
Loading