feat(architectures): serve model policy from the backend, and read it in webv2 - #146
Draft
Pfannkuchensack wants to merge 12 commits into
Draft
Conversation
- Added `scheduler_applies_to_graph` to ERNIE-Image architecture definition to ensure proper UI behavior. - Introduced a new fixture for architecture capabilities to maintain consistency between backend and frontend. - Enhanced generation policies to correctly prioritize guidance settings based on architecture labels. - Added tests to verify that the scheduler application flag aligns with the node's capabilities.
…ven-model-policies
… in webv2 `BASE_GENERATION` in webv2 held 14 architectures with grid, canvas, steps, CFG, scheduler, negative-prompt policy and six UI flags -- the same facts `architectures/defs/` declares. Adding a model type meant editing the frontend too, even when nothing about it was special. It is deleted; the policy now comes from `GET /api/v2/models/capabilities`. ## Reading the table `core/` cannot reach `data/` under `feature-core-purity`, so the table is pushed in through a registry, the shape `configureHttpAuth` already uses. `SupportedGenerateBase` is no longer `keyof typeof BASE_GENERATION` but a declared list beside `GRAPH_BUILDERS`: which architectures we can build a graph for is the one question only the frontend can answer, and it must not depend on load state -- `isSupportedGenerateModel` has a dozen callers outside the widget. Loading is gated on two levels. The widget shows a loading state, but the load-bearing gate is in `createGenerateWidgetSyncRuntime`: it reconciles at construction and *persists* the result into the project, so without the table it would write fallback defaults to disk -- an off-grid canvas for CogView 4 among them. `resolveGenerateWidgetValues` returns null until the table is there. Defaults change for six architectures, deliberately. FLUX moves to 28 steps at guidance 3.5 with real variant rows (Schnell 4, Fill 50 at guidance 30) -- the single row could not express that and carried Schnell's step count for every variant. sd-3 40/4.5, cogview4 50/3.5, anima 35/4.5, z-image 9 steps, SD 2.x 768px. Control kinds, reference-image limits and regional-guidance support are read from the table too; their data was verified identical to the hardcoded rules first, so those are behaviour-preserving. The limits in `controlValidation.ts` and the FLUX-Fill exception stay -- the endpoint has no column for them, and a comment says so. ## Which VAEs an architecture accepts The same fact was written down three times -- once per loader as `ui_model_base`, once in webv2's VAE picker, once more in its related-models linker -- and all three disagreed about Anima. It is not derivable: eight architectures share a 16-channel space at 8x compression, but SD 3.5 and CogView 4 belong to neither the Wan/Qwen family nor FLUX's. Nor does the VAE's own base settle it. The same physical file is registered as `anima` or `qwen-image` depending on which family it was installed for -- byte-identical, 194 tensors -- and a `wan` VAE is either the shared 16-channel Wan 2.1 file or TI2V-5B's 48-channel Wan2.2-VAE, which fits nothing else. So the new facet declares base *and*, where the base is ambiguous, the channel count `VAE_Checkpoint_Wan_Config` has been recording all along without anyone reading it. Two loader declarations were wrong and are corrected. `anima_model_loader` declared nothing, so its picker offered every VAE ever installed; it accepts the Wan-family file under any of its three bases plus a FLUX one, since `anima_l2i` branches on `isinstance(vae, (AutoencoderKLWan, FluxAutoEncoder))` with a separate decode path for each. `flux2_klein_model_loader` offered FLUX VAEs, claiming Klein "uses the same VAE as FLUX (16-channel)" -- it does not: generating with one raises `AutoEncoder.decode() got an unexpected keyword argument 'return_dict'`, the legacy 16-channel class against a 32-channel latent. ## Loading a Qwen-Image VAE Two layouts of that autoencoder are in circulation. Repo exports carry diffusers keys (`decoder.conv_in.weight`); community redistributions carry the original layout (`decoder.conv1.weight`) and need converting. Only the first was handled, and `AutoencoderKLQwenImage` registers no single-file conversion, so a redistributed file failed `load_state_dict(strict=True)` with all 194 keys missing. `strict=True` was right; the assumption in front of it was not. The byte-identical checkpoint installed under `anima` loaded fine all along through `AutoencoderKLWan.from_single_file`, which converts -- so whether a VAE worked came down to which base it happened to be probed as, and `VAE_Checkpoint_Wan_Config` deliberately hands 16-channel files to the Qwen-Image config unless the filename says "wan". The loader now looks at the state dict: diffusers keys take the direct path with `strict=True` unchanged, anything else is converted as the Wan-family file it is. No installation needs adjusting -- identification, model records and bases are untouched. ## Bugs that fall out of removing the duplicates - `bboxGrid.ts` kept its own copy of the grid column and had drifted: krea-2, wan and ideogram-4 got 8px steps while their denoise nodes enforce multiples of 16, so the canvas offered sizes that failed at enqueue. - `RegionalGuidanceSettings` asked "is this the FLUX family?" where the graph builder asks `regional_negative`. On krea-2 it rendered a negative prompt and an Auto-Negative switch that `addRegionalGuidance` then discarded. - ERNIE-Image's `FeaturesFacet` omitted `scheduler_applies_to_graph`, defaulting it to False, while `ernie_image_denoise` builds its sampler from the `scheduler` field. Thirteen of fourteen agreed; nothing compared them. - `getGenerationDefaults` folded `cfg_scale ?? guidance`, so a guidance-distilled model's "CFG off" marker won over the guidance it actually samples with. FLUX dev generated at guidance 1.0 instead of 3.5. ## Keeping it honest A committed fixture of the response body is pinned against `architecture_capabilities()` by a backend test, feeds the frontend's unit tests, and is what `scripts/mock-backend.mjs` serves -- so the hand-written wire type, the tests and the mock cannot drift from the real route. Invariant tests bind each facet to the node that enforces it: `dimension_grid` to `multiple_of`, `scheduler_applies_to_graph` to whether the denoise node takes a scheduler, and each `VaeFacet` to its loader's `ui_model_base`. Tests seed the registry through `architectureCapabilities.testing.ts` rather than a global setup file, so a test that wants to observe the unloaded behaviour still can. Verified end to end against real models: all seven installed architectures generate, previews included, and a `qwen-image`-registered VAE that failed before now works.
…erver-driven-model-policies
Four CI jobs failed, from two causes. `oxfmt --check` rejected 28 files. They were linted and typechecked but never formatted: on a Windows checkout `core.autocrlf` gives every file CRLF while oxfmt writes LF, so `format:check` fails for all 1898 files locally and is useless as a signal. Formatted the ones this branch touches. `openapi.json` and `schema.ts` were stale. Correcting `ui_model_base` on the Anima, Qwen-Image and FLUX.2 Klein loaders changes the invocation schema, and the artifacts were last regenerated before that. Two more surfaced once formatting stopped short-circuiting the lint job: - The test helper was named `useArchitectureCapabilitiesFixture`, so `react-hooks/rules-of-hooks` read it as a hook called at module top level. It is not one; `seedArchitectureCapabilities` says what it does anyway. - Dead imports left behind by deleting `BASE_GENERATION` and moving the ordered-bases pin out.
Four CI jobs failed, from two causes. `oxfmt --check` rejected 28 files. They were linted and typechecked but never formatted: on a Windows checkout `core.autocrlf` gives every file CRLF while oxfmt writes LF, so `format:check` fails for all 1898 files locally and is useless as a signal. Formatted the ones this branch touches. `openapi.json` and `schema.ts` were stale. Correcting `ui_model_base` on the Anima, Qwen-Image and FLUX.2 Klein loaders changes the invocation schema, and the artifacts were last regenerated before that. Two more surfaced once formatting stopped short-circuiting the lint job: - The test helper was named `useArchitectureCapabilitiesFixture`, so `react-hooks/rules-of-hooks` read it as a hook called at module top level. It is not one; `seedArchitectureCapabilities` says what it does anyway. - Dead imports left behind by deleting `BASE_GENERATION` and moving the ordered-bases pin out.
…hub.com/invoke-ai/InvokeAI-7 into feat/webv2-server-driven-model-policies
`compileUpscaleGraph` passes Upscale's own `kdpm_2` through `coerceSchedulerForGraph`, which reads architecture policy. Without the table the fallback config applies, and its `schedulerAppliesToGraph: false` makes the function return the fallback scheduler rather than pass the requested one through -- so the compiled graph carried `euler_a`. Missed because the webv2 suite was run in subsets while iterating. It completes in under two minutes; there was never a reason not to run all of it.
…test `deriveImageRecallCapabilities` reports whether an image's metadata can be recalled, and clip skip depends on the architecture's `clipSkipMax`. That comes from the capability table now, so without it the fallback applies and the capability reads false. Missed because `vitest.config.mts` excludes `*.browser.test.*` -- they run under `vitest.browser.config.mts`, which had not been run at all. Both suites pass now: 807 browser tests and 6435 node tests.
Reading architecture policy from the backend adds four modules to the editor and launchpad bundles: +52 bytes owned JavaScript on launchpad, +281 on editor, and one additional script request per editor route. Only the structural numbers are updated. `--update-baseline` also re-records timings, and those come from whichever machine ran it -- the fresh numbers were ~10% above the committed ones, which would have loosened a gated tolerance and let real regressions of that size through. The 56 timing values are kept as committed.
`--update-baseline` writes `architecture-baseline.json` itself and not in oxfmt's style, so `format:check` rejected it. Values are unchanged -- the only difference is how one array is wrapped.
`oxfmt` and `prettier` read the file on disk, so on a Windows checkout `core.autocrlf=true` makes `format:check` fail for all 1898 files -- including every file the contributor never touched. The check is therefore unusable locally, and formatting errors are only found by CI. That happened four times in a row on one PR. `* text=auto` already normalizes these files to LF in the repository, so this changes the working copy only: `git add --renormalize invokeai/frontend` reports nothing, and no file content differs after re-checkout. `docker/**` has carried the same rule for the same reason. Verified after re-checkout: `pnpm run format:check` reports "All matched files use the correct format", and `prettier --check` passes for the legacy frontend.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adding a
BaseModelTypeshould not mean editing the frontend. It still did:BASE_GENERATIONin webv2 held 14 architectures with grid, canvas, steps, CFG, scheduler, negative-prompt policy and six UI flags — the same factsinvokeai/backend/architectures/defs/already declares. This deletes that table and reads the policy fromGET /api/v2/models/capabilitiesinstead.Three things had to happen for that to be safe, and each turned up a bug of its own.
Reading the table.
core/cannot reachdata/underfeature-core-purity, so the rows are pushed into a registry — the shapeconfigureHttpAuthalready uses.SupportedGenerateBasebecomes a declared list besideGRAPH_BUILDERSrather thankeyof typeof BASE_GENERATION: which architectures we can build a graph for is the one question only the frontend can answer, and it must not depend on load state, sinceisSupportedGenerateModelhas a dozen callers outside the widget.Loading is gated twice. The widget shows a loading state, but the load-bearing gate is in
createGenerateWidgetSyncRuntime, which reconciles at construction and persists its result into the project — without the table it would have written fallback defaults to disk, including an off-grid canvas for CogView 4.Which VAEs an architecture accepts. The same fact was written down three times — once per loader as
ui_model_base, once in webv2's VAE picker, once more in its related-models linker — and all three disagreed about Anima. It is not derivable: eight architectures share a 16-channel space at 8× compression, but SD 3.5 and CogView 4 belong to neither the Wan/Qwen family nor FLUX's. Nor does the VAE's own base settle it — the same physical file is registered asanimaorqwen-imagedepending on which family it was installed for (byte-identical, 194 tensors), and awanVAE is either the shared 16-channel Wan 2.1 file or TI2V-5B's 48-channel one. So the new facet declares base and, where the base is ambiguous, the channel countVAE_Checkpoint_Wan_Confighas been recording all along without anyone reading it.Loading a Qwen-Image VAE. Two layouts of that autoencoder circulate; only the diffusers one was handled, with
strict=True, so a community redistribution failed with all 194 keys missing — while the byte-identical file installed underanimaworked, because that path converts. Whether a VAE worked came down to which base it happened to be probed as.Defaults change for six architectures
Deliberate, and the reason this is a
featand not arefactor. webv2 is not shipped yet, so no existing user is affected.The variant rows are the point: one row per architecture could not express that FLUX Schnell wants 4 steps where dev wants 28, so it carried Schnell's number for every variant.
Bugs fixed on the way
bboxGrid.tskept a second copy of the grid column and had drifted — krea-2, wan and ideogram-4 got 8px steps while their denoise nodes enforce multiples of 16, so the canvas offered sizes that failed at enqueue.RegionalGuidanceSettingsasked "is this the FLUX family?" where the graph builder asksregional_negative. On krea-2 it rendered a negative prompt and an Auto-Negative switch thataddRegionalGuidancesilently discarded.flux2_klein_model_loaderoffered FLUX VAEs, claiming Klein "uses the same VAE as FLUX (16-channel)". It does not — generating with one raisesAutoEncoder.decode() got an unexpected keyword argument 'return_dict'.anima_model_loaderdeclared noui_model_baseat all, so its picker offered every VAE ever installed.scheduler_applies_to_graph, defaulting it toFalse, whileernie_image_denoisebuilds its sampler from theschedulerfield. Thirteen of fourteen agreed; nothing compared them.getGenerationDefaultsfoldedcfg_scale ?? guidance, so a guidance-distilled model's "CFG off" marker won over the guidance it actually samples with — FLUX dev generated at 1.0, not 3.5.How it stays honest
A committed fixture of the response body is pinned against
architecture_capabilities()by a backend test, feeds the frontend's unit tests, and is whatscripts/mock-backend.mjsserves — so the hand-written wire type, the tests and the mock cannot drift from the real route. webv2 has no generated OpenAPI types, so without this the TS type would be an unguarded assertion.Invariant tests bind each facet to the thing that enforces it:
dimension_gridto the node'smultiple_of,scheduler_applies_to_graphto whether the denoise node takes a scheduler, eachVaeFacetto its loader'sui_model_base, andnegative_prompt: nevertocfg_scale == 1.0. Four of the six bugs above were found by writing those, not by review.Related Issues / Discussions
Stacked on #133 (the architecture facet registry), which introduces the endpoint this consumes.
QA Instructions
Nothing about an existing install needs adjusting — no migration, no re-identification, no model records touched.
Generation. Open Generate and confirm the panel renders after a brief load. Pick FLUX Schnell and FLUX dev in turn: they should prefill 4 and 28 steps respectively, which the old single-row table could not do. Check SD 2.x prefills 768×768.
The grid fix. On the canvas with a Krea-2, Wan or Ideogram-4 model, the bbox should snap to 16px. Before this, 8px was offered and an off-grid size failed at enqueue — try 1032px wide on
mainto see the old behaviour.Regional guidance. Add a regional guidance layer with a Krea-2 model. The negative prompt field and Auto-Negative switch should be gone; they were rendered and then discarded by the graph builder.
VAE pickers. With an Anima model, the VAE list should offer Wan/Qwen/Anima and FLUX VAEs and nothing else — it previously offered every VAE installed. With FLUX.2 Klein, FLUX VAEs should no longer appear; selecting one on
maincrashes during decode.The Qwen VAE fix. If you have a community Qwen-Image VAE registered under base
qwen-image(thedecoder.conv1.*layout rather thandecoder.conv_in.*), generating with it failed before with "194 missing keys" and should now work.The gate. Point the frontend at a backend without the
/capabilitiesroute, or make it 500. The Generate panel must show an error with a retry, not render fields prefilled with generic defaults.Verified end to end on this branch: all seven installed architectures generate with previews;
qwen-imagewith both VAE registrations; FLUX.2 Klein with the FLUX.2 VAE (and confirmed failing with the FLUX one).Merge Plan
Merge #133 first. This branch is stacked on it and consumes the endpoint it adds.
The diff currently shows ~96 files, but only 60 are this branch's — the rest is the generate-widget redesign merged in from
mainand will disappear from the diff once the base branch catches up.Nothing touches a DB schema, a redux slice or a migration.
openapi.jsonandschema.tsare regenerated; the only delta is one added enum value inArchitectureModality.modes, which is additive for any client.Checklist
What's Newcopy (if doing a release after this PR)🤖 Generated with Claude Code