feat(plugins): out-of-tree model extensions for the sa3 family - #331
Open
ryanontheinside wants to merge 2 commits into
Open
ryanontheinside wants to merge 2 commits into
ryanontheinside wants to merge 2 commits into
Conversation
Adds a versioned plugin system so a model extension can be developed and shipped outside this repository, plus the generic SA3 engine improvements it depends on. Nothing here is specific to any concrete extension: the core treats an extension's configuration, controls and metrics as opaque and never branches on its type. With no extension selected, every path is byte-for-byte the stock one. Generic SA3 engine work, independently useful: - SA3Context.diffusion_objective, and sampler selection driven by it. The previous code assumed every SA3 checkpoint was post-trained, which silently mis-samples a plain rectified-flow base. - --sa3-base-checkpoint, so a non-catalog checkpoint can be evaluated without installing it into the managed models tree. - Context cache keys cover checkpoint CONTENT identity (size, mtime), not just path. A file its producer rewrites in place had a stable path and different weights, and the cache served whichever loaded first. - Bounds-checked argument parsing; a trailing flag raised IndexError. - acestep/engine/sa3_internals.py: a Tier-2 contract giving the vendored stable_audio_3 coupling one address, with a structural canary test that fails in public CI when the vendored layout moves. The plugin system (acestep/plugins/): - Entry-point discovery only. An explicitly selected plugin that is missing or incompatible fails startup; an unselected broken plugin is isolated. Silently dropping a requested model extension would produce plausible, wrong output. - Selection is startup-only and operator-controlled. That is a safety property (a client can never name a module, checkpoint or config path) and a mechanical one: the knob manifest ships once on the ready frame. - Extension controls are plain KnobSpecs composed in SA3Backend.knob_specs and validated by the same coercion as core knobs. This matters because coerce_knob_values passes unknown names through unclamped by design, so a control that reached the client but not the spec map would be accepted from the wire unvalidated, with no error. Tests pin both directions. - Controls must be namespaced plugin_<plugin_id>_<control>. The namespace is the whole anti-shadowing defense, in preference to a blocklist of core names that would go stale and could not enumerate the dynamically named knobs at all. - Lifecycle: veto before load, fingerprint into the cache key, install after load, close at teardown. A partial install unwinds to fully stock rather than serving a half-hooked model. - Extension knobs are float/int only, because the browser panel binds enum/bool by name and would render anything else disabled. Docs in docs/PLUGINS.md.
… source decorate_conditioning received only the encoded source latent. That is enough for an extension whose condition IS the latent, and insufficient for one whose condition is derived from the source audio: recovering the audio from inside an extension costs a full decode per swap and cannot return what the encoder discarded. The hook now receives a SourceView carrying the latent, the pre-encode waveform, and its sample rate. Every field is optional, because both absences are real: a session created without a source has no latent, and a source re-anchored from a latent DEMON did not encode has no waveform. PLUGIN_API_VERSION goes to 2. This is deliberately breaking rather than additive - a version-1 runtime's hook would bind the view where it expected a latent, and DEMON would hand it conditioning that looks fine and is wrong. The version check refuses it at load instead. SA3Backend retains the (sample_rate, waveform) pair the create path already built, and handle_swap_source replaces it together with the anchor: the two halves of the view must always describe the same audio, or an extension re-decorates the new anchor against the old waveform. Nothing on the denoising path reads the retained waveform. Docs get a "Conditioning on the session source" section, including the rule that a condition must be sized from the session's own geometry rather than a constant.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Lets an installed Python package extend a loaded model, out of tree, without
DEMON knowing anything about what the package does.
The core never branches on an extension's concrete type. It sees a spec, a
config schema, namespaced knobs and a lifecycle object. Selection is
startup-only and operator-controlled: a browser client can never name a
plugin, a module path, a checkpoint or a config file.
Import tiers
acestep.pluginsPLUGIN_API_VERSION.acestep.engine.sa3_internalsAPI_VERSIONand a structural canary test.acestep.*Tier 2 exists because an extension that adds modules participating in the
transformer's computation cannot be written against an abstract API. That
coupling is real, so it gets one public, tested address rather than being
spread across a package. A vendored model bump then breaks one file with a
clear message instead of failing silently somewhere else.
Plugin API 2
decorate_conditioningreceives aSourceView(latent, waveform, samplerate) rather than a bare latent. This is a deliberately breaking change: an
API-1 runtime would bind the view where it expected a latent, so the version
gate has to reject it rather than let it run.
The waveform rides along because DEMON already holds it. An extension whose
condition derives from audio rather than from a latent would otherwise have
to decode the latent back to audio, spending real time per swap to recover
something that was in memory a moment earlier.
Notes for review
plans contain only the stock graph, so running them would silently ignore
the extension. The windowed VAE decoder keeps TensorRT.
decorate_conditioningreturns anew bundle.
Tests
tests/unit: 655 passed, 4 skipped.tests/unit/test_lora_facade_session.pyhas 4 failures that arepre-existing on main and untouched by this branch (the test file's blob
is byte-identical to main's). They are fixture rot:
session.pyreadspending_registerand the test never sets it.