Skip to content

Plugin refactor + classifier support + idempotency - #10

Merged
joefutrelle merged 9 commits into
mainfrom
plugin-refactor
Jul 21, 2026
Merged

Plugin refactor + classifier support + idempotency#10
joefutrelle merged 9 commits into
mainfrom
plugin-refactor

Conversation

@joefutrelle

Copy link
Copy Markdown
Collaborator

Plugin refactor + classifier support + idempotency

Refactors the plugin system to use dependency injection so the core never imports plugins at load time. Plugins are now generic and parameterized; instrument-specific presets (IFCB features, IFCB CNN classification) move under improv.plugins.ifcb and pin a kind/index_table onto a generic plugin. Adds machine classification support end-to-end and makes provenance/index writes idempotent.

Highlights

  • Plugin DI — plugins register at runtime instead of being imported by the core; store/indexes.py removed, store/dedup.py added.
  • Classification — new classification plugin, router, and schema (class-scores schema change); taxonomy registration/lookup plus stateless and decoded read paths; 002_classifier_taxonomy migration.
  • Idempotency — append-only writes deduplicated at read time; provenance records keyed by (image_id, kind, source, data_hash) using RFC 8785 (JCS) canonical hashing (new rfc8785 dep). Backend-agnostic (VAST DB / DuckDB+Parquet).
  • Small-batch POSTPOST /images/provenance/batch lets low-volume producers write provenance over REST without depending on the columnar store (one instrument per batch).
  • ClientImprovClient gains taxonomy registration.
  • Tests — added coverage for classification, provenance, client, plugins, store dedup (~590 net test lines).
  • Docs — README updated (plugin model, classifier API, idempotency, client contract on event-time timestamp).

Review note: client contract — timestamp must be event time, not wall-clock-at-send, or retries won't dedup.

@joefutrelle joefutrelle self-assigned this Jul 8, 2026
Comment thread README.md
- **High-volume writes** (image metadata, provenance, index tables, image bytes) go directly to the columnar store and object store

This avoids coupling ingest scripts to the database while keeping high-throughput writes off the HTTP path.
This avoids coupling ingest scripts to the database while keeping high-throughput writes off the HTTP path. Low-volume producers that prefer not to depend on the columnar store directly can post small provenance batches over REST (`POST /images/provenance/batch`, one instrument per batch).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if there's a way we could further describe the difference between low-volume and high-volume producers. Perhaps some examples of what kind of data would be best for the REST API, and what kind of data would be best for the columnar store?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this could be elucidated further. The primary use case would be for data coming in small batches in near-real time.

Comment thread src/improv/plugins/sample_context.py
Comment thread src/improv/plugins/geolocation.py
Comment thread tests/test_api/conftest.py
Comment thread src/improv/client.py
@joefutrelle
joefutrelle merged commit e0c7298 into main Jul 21, 2026
4 checks passed
@joefutrelle
joefutrelle deleted the plugin-refactor branch July 21, 2026 19:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants