feat: multi-GPU parallel session execution

[lstein]

Summary

This PR adds multi-GPU parallel generation: on a machine with more than one GPU, InvokeAI runs several generation sessions concurrently — one per GPU — instead of draining the queue one job at a time. Jobs are distributed fairly across users so a single user's large batch can't monopolize every GPU while others wait.

It's controlled by a new generation_devices config setting (defaults to auto = use every available CUDA GPU). Setting it to a single device, or leaving CUDA out of the picture, preserves the previous serial behavior exactly. The choice of GPUs can also be controlled via a new section of the Settings dialogue (restart required to take effect).

Demo (turn on the sound!)

invoke-mgpu.mp4

How it works — the change is built around five small backend seams plus a frontend update, rather than per-node edits:

• Device context (invokeai/backend/util/devices.py): a thread-local set/get/clear_session_device on TorchDevice; choose_torch_device() consults it first. This is the lynchpin — the ~79 existing call sites resolve to the worker's GPU with no per-node changes.
• Per-device model caches (model_manager_default, model_load_default): one ModelCache per device, resolved by the current thread's device, with fan-out for clear/drop/shutdown. Model construction is serialized against VRAM moves to prevent meta-device corruption.
• Atomic dequeue (session_queue_sqlite.dequeue): a lock makes select+claim atomic so concurrent workers never grab the same queue item.
• Worker pool (session_processor_default): one _SessionWorker per device, each pinning torch.cuda.set_device + the session device, with its own runner and cancel event; cancellation is routed per item. Profiling is disabled when more than one worker is active.
• Concurrency hardening: added a Lock to ObjectSerializerForwardCache and made DiskImageFileStorage thread-safe for parallel sessions.

Frontend: during parallel generation the progress display stacks one progress bar per active session (each disappears as its session finishes), and the image viewer tiles per-session progress previews when ≥2 sessions are active.

Related Issues / Discussions

• Builds conceptually alongside [feat] Round robin job scheduling in multiuser mode #9086 (round-robin per-user dequeue). This PR does not depend on [feat] Round robin job scheduling in multiuser mode #9086 being merged — the atomic claim works on the current FIFO dequeue, and [feat] Round robin job scheduling in multiuser mode #9086's round-robin CTE can slot in cleanly when it lands.

QA Instructions

On a multi-GPU machine:

• With default config (generation_devices: auto), enqueue a batch larger than the GPU count and confirm multiple sessions run simultaneously (one per GPU), with stacked progress bars and tiled previews in the viewer.
• Set generation_devices: [cuda:0] and confirm generation runs serially, exactly as before this PR.
• Set generation_devices: [cuda:0, cuda:2] and confirm only those devices are used.
• Cancel an in-flight item and confirm only that session stops.
• On a single-GPU / CPU / MPS machine, confirm auto resolves to the one best device and behavior is unchanged.

New automated tests cover device routing (test_model_load_device_routing.py), dequeue concurrency (test_session_queue_dequeue_concurrency.py), and device resolution (test_devices.py).

Merge Plan

Standard merge. No DB schema or redux migrations. Touches the session processor and model cache, so worth a careful look from those areas' owners.

Checklist

• The PR has a short but descriptive title, suitable for a changelog
• Tests added / updated (if applicable)
• ❗Changes to a redux slice have a corresponding migration — N/A, no slice changes
• Documentation added / updated (if applicable)
• Updated What's New copy (if doing a release after this PR)

[JPPhoto]

I can take a look at this but I don't have the ability to test it. Recommend assigning to a second person as well.

[lstein]

I have converted back to a draft temporarily while I work out excessive RAM consumption issues.

[JPPhoto]

Some comments with the current code (not posted as a review):

• invokeai/invokeai/app/services/session_queue/session_queue_sqlite.py:478 and :651

  Bulk cancellation only cancels one active item in multi-GPU mode. The new processor starts one worker per device and each worker calls dequeue() independently, so there can be multiple in_progress rows at the same time. But cancel_by_batch_ids(), cancel_by_destination(), and cancel_by_queue_id() all exclude status = 'in_progress' from the bulk update, then call get_current() once. get_current() uses LIMIT 1 at line 287, so only one active item is moved to canceled; other active items in the same queue/batch/destination continue running.

  Trigger: two GPUs are running two items from the same batch or destination, then the user cancels that batch/destination or queue.

  Consequence: cancellation appears to succeed but at least one generation keeps consuming GPU and can still produce output after the user requested cancellation.

  To expose this issue, add a test that dequeues two items into in_progress with the same batch/destination, calls each bulk cancel API, and asserts all matching in_progress items become canceled and emit cancel events.

• invokeai/invokeai/app/services/session_queue/session_queue_sqlite.py:234

  The documented per-user fairness is not implemented. The PR docs say a single user's large batch cannot monopolize every GPU, but dequeue() still orders only by priority DESC, item_id ASC. With two workers, if user A enqueues a large batch and user B enqueues after it, every worker keeps claiming A's lower item_id rows until A's older batch is drained. There is no user grouping, round-robin CTE, or last-served-user state in this PR.

  Trigger: multiuser mode, multiple GPUs, user A enqueues many items before user B enqueues one item at the same priority.

  Consequence: user B can wait behind user A's entire older batch, contradicting the user-facing behavior described in docs/src/content/docs/configuration/invokeai-yaml.mdx:119.

  To expose this issue, add a queue test that enqueues many pending items for user A, then one for user B, simulates repeated concurrent dequeue() calls, and asserts the second user is selected before A monopolizes every worker cycle.

[lstein]

@JPPhoto thanks for the careful review — both points were spot on.

Bulk cancellation in multi-GPU mode — fixed. cancel_by_batch_ids, cancel_by_destination, and cancel_by_queue_id now cancel every matching in_progress item (via a shared _cancel_in_progress_matching helper) instead of only the single get_current() item, so each running worker receives its QueueItemStatusChangedEvent and stops. While in there I found and fixed two siblings:

• delete_by_destination had the same one-item limitation (it canceled one, then deleted all matching rows — leaving the other workers running against deleted rows).
• A genuine cancel-loss race in the worker loop: cancel_event.clear() ran after dequeue() + gc.collect(), so a cancel arriving in that window got wiped. The clear now happens before dequeue, plus a fresh post-claim status re-check so a cancel that races the claim still skips the run.

All three are covered by new regression tests (two in_progress items dequeued onto separate devices, asserting all matching items are canceled and emit cancel events; user-scoping preserved). They fail on the old code and pass on the fix.

Per-user fairness — correct, this PR intentionally does not implement it; dequeue() is still priority DESC, item_id ASC. The round-robin scheduler is coming in #9086, which I've checked for compatibility with this branch: the two are orthogonal (round-robin picks which pending item; multi-GPU atomically claims it under a lock), and they actually compose well — an in_progress item counts as "recently served" via the existing started_at trigger, so round-robin naturally spreads the concurrent GPU slots across distinct users. The plan is to land #9086 first and rebase this on top (minor migration renumber + a small dequeue() merge), so fairness arrives through that PR rather than being duplicated here.

[JPPhoto]

There are some more issues that need resolving:

• invokeai/invokeai/backend/model_manager/load/model_cache/shared_cpu_weights.py:64

  Stale shared CPU weights can survive a model setting change and be adopted by a rebuilt per-device cache. SharedCpuWeightsStore.acquire() returns the existing canonical state dict for the same cache key. drop_model() only marks locked entries stale at invokeai/invokeai/backend/model_manager/load/model_cache/model_cache.py:1068, so their shared-store reference remains live until unlock. If another device cache reloads the same model key before that unlock, the wrapper re-points the newly built module at the old canonical tensors at invokeai/invokeai/backend/model_manager/load/model_cache/cached_model/cached_model_with_partial_load.py:61 and cached_model_only_full_load.py:52. This defeats the invalidation added for load-affecting settings in invokeai/invokeai/app/api/routers/model_manager.py:445.

  Trigger: one GPU is running a model while an admin toggles a load-affecting setting such as fp8_storage; another GPU then loads that model before the first GPU unlocks.

  Consequence: the "rebuilt" cache entry can silently use the pre-change canonical weights and keep them alive under the same key.

  To expose this issue, add a test that loads the same key into two caches sharing a SharedCpuWeightsStore, locks one entry, calls drop_model() on both caches, then puts a changed model under the same key into the unlocked cache and asserts it does not adopt the old canonical tensors.

• invokeai/invokeai/app/api/routers/app_info.py:151

  The runtime settings API validates only the device string pattern, not the same constraints enforced at startup. UpdateAppGenerationSettingsRequest.validate_generation_devices() accepts values like ["cuda:99"] if they match the regex, and update_runtime_config() persists them at lines 219 and 226. Startup later calls TorchDevice.get_generation_devices() from the model manager path, where unavailable or out-of-range CUDA devices raise at invokeai/invokeai/backend/util/devices.py:200. Empty lists are also accepted by the request model but rejected by InvokeAIAppConfig at invokeai/invokeai/app/services/config/config_default.py:272, producing an unhandled server error instead of a 422.

  Trigger: an admin or API client PATCHes /api/v1/app/runtime_config with generation_devices: ["cuda:99"] or [].

  Consequence: the server can persist a config that fails on restart, or return a 500 for input the request schema accepted.

  To expose this issue, add a test that PATCHes out-of-range CUDA and empty-list values and asserts the API rejects them with 422 without mutating or writing config.

• invokeai/invokeai/app/services/session_queue/session_queue_sqlite.py:235

  The PR and docs still promise per-user fairness, but dequeue remains strict priority DESC, item_id ASC. The author comment says fairness is intentionally deferred to PR 9086, but this PR's docs still say a single user's large batch cannot monopolize every GPU at invokeai/docs/src/content/docs/configuration/invokeai-yaml.mdx:119 and generated settings repeat that at invokeai/docs/src/generated/settings.json:496.

  Trigger: user A enqueues a large batch before user B at the same priority on a multi-GPU system.

  Consequence: all workers keep claiming user A's older rows until they drain, so user B can still be starved despite the user-facing claim.

  To expose this issue, add a queue test that enqueues many items for user A, then one for user B, calls dequeue() repeatedly for multiple worker slots, and asserts user B is selected before A monopolizes all slots. Otherwise hold off on the fairness claim from docs until PR 9086 is merged.

• invokeai/docs/src/content/docs/configuration/invokeai-yaml.mdx:132

  The docs advertise generation_devices: [] as a valid serial fallback, but the config validator now rejects empty lists at invokeai/invokeai/app/services/config/config_default.py:272 and tests assert rejection in tests/app/services/config/test_config_generation_devices.py:31. The same docs also say weights are duplicated in RAM per GPU at line 144, while this PR now adds shared CPU weights to avoid that.

  Trigger: Set generation_devices: [].

  Consequence: InvokeAI rejects the config instead of starting serially as documented.

  Expected docs fix: update docs/src/content/docs/configuration/invokeai-yaml.mdx and regenerated settings copy to match the accepted values and the new shared-RAM behavior.

[lstein]

Lots of changes in commit 2d3802a . Previously each GPU had its own RAM cache, which meant that the same model could be loaded and stay resident twice, doubling the amount of RAM needed. These changes:

1. De-duplicate models such that the same model is only resident once.
2. Handles the case of the model being modified by a LoRA in one GPU session and not in the other. The RAM copy holds the canonical unmodified model and the LoRA modified version only exists transiently in VRAM of the model that needs it. Same logic applies to reference images and controlnets.
3. I was able to test on a machine with the interesting configuration of two 48 GB VRAM GPUs and 96 GB RAM. This exposed a bunch of places where model loading was being handled inefficiently and causing RAM spikes. A variety of checks have been implemented to avoid double-loading, OOMs and thrashing.