image generation endpoints need a cancel operation api call

[HarvieKrumpet]

Issuing a sdapi call for long generations, is easy enough to cancel on the client side. but the generation server will continue to process until it actually has to respond with the image. At which point it notices that it has no connection. But this could be minutes later. and in the meantime, no more images can be started until it figures this out. The api needs to poll in a background task at least that the tcp connection is still valid. Or a separate cancel api call to abort any current operations.

[LostRuins]

A valid concern. Are you generating images on CPU? How long does it take normally?

Adding an abort function should be relatively easy - but having it trigger automatically on disconnect not so straightforward since SD is a sync api.

[HarvieKrumpet]

they are fully gpu generated, but hi-res. so very hard to experiment without a cancel. not the greatest gpu in any case a rtx 3060.
snappy response until I exceed 384x384.

A separate put/post for cancel will be totally fine. But I noticed the api seems to be queuing up api commands waiting for each one to complete. At least I can't infer simultaneously with generation and visa versa,

If SD can handle a Task.yield() between steps, could listen for the cancel and break gracefully. you could trap sd when it attempts to output a log for each step.

[LostRuins]

Another concern is that I would need to require each request a unique ID (same as the text gen side) when generating in order for that request to be cancellable. This is to prevent ongoing requests from being maliciously cancelled repeatedly by a third party when used in a shared-server environment.

[LostRuins]

I've checked the per-step denoising loop and it's not really easy to interrupt cleanly at the moment - there are also resources needed to be released, and the loop is iterated in multiple places depending on each sampler - there is no clean main loop where a yield can be added. So I agree this is good to have but probably will take some effort to be added.

[NintendoManiac64]

Is this by chance the problem I'm demonstrating in the following video? I was going to create a new issue but then realized I forgot to check and see if there was an existing one.

(jump to the 7:00 mark to skip the set up and just see the problem in action)

video.mp4

[wbruna]

That's the issue, yes.

I have a PR upstream for the backend part (leejet/stable-diffusion.cpp#1124) that I could add to Koboldcpp; not exactly a yield, but it provides a cancel function that can be called asynchronously. And I have to keep it up to date with upstream changes anyway, so merging it wouldn't be additional trouble.

The main issue, as @LostRuins explained, would be how to call it in a safe way. The sdcpp server support I've implemented just checks for a dropped client connection; if the stable-ui work in the same way, and if the Python support allows the necessary kind of low level access (web libraries usually make an effort to hide that kind of detail), maybe a similar functionality could be implemented on our web server.

[LostRuins]

detecting a dropped connection might be possible - i use it for the openai sse streaming. Behind the scenes it'll probably do something like set the remaining steps to 0 so the diffuser ends after the current step (that way we get proper cleanup)

[wbruna]

detecting a dropped connection might be possible - i use it for the openai sse streaming. Behind the scenes it'll probably do something like set the remaining steps to 0 so the diffuser ends after the current step (that way we get proper cleanup)

My cancel function does something like that: it sets a 'cancelled' flag on the sd context. The normal generation path checks it at the beginning of the next model call, and aborts it through the same path as a failed generation.