Feat: Optimize operations for Bonsai on mobile devices - #2
Conversation
|
I've done an initial code cleanup.
The other backends are either niche, not useful for our application or too complicated to compile |
…enVINO, RPC, VirtGPU, WebGPU, zDNN, ZenDNN) Keep only the 6 targeted backends: CPU, CUDA, HIP, SYCL, Vulkan, Metal. - Delete all backend source directories from ggml/src/ - Delete unused backend headers from ggml/include/ - Remove ggml_add_backend() and register_backend() calls from CMakeLists.txt and ggml-backend-reg.cpp - Remove dead cmake options and ggml-config.cmake.in dependencies - Clean up root CMakeLists.txt (SD_OPENCL, SD_MUSA) and misc docs refs - Verified: CUDA and CPU inference both work correctly (cat test 512x512)
- AVX2 nrc=4: fused 4-row x 1-col kernel for GEMV (by=0),
4 x nrc=2 decomposition for GEMM (by>0)
- Generic fallback nrc=4 for non-AVX2
- Dispatcher: removed ne11/ir1_end checks to allow nrc>1 for GEMV
- tmp[64] for up to 4 output columns
- by_val=0 when fewer columns than nrc to keep kernels safe
- Copy loop bounded by num_cols to avoid out-of-bounds writes
|
Alright, here are the initial performance improvements on the x86 architecture using an i5-13600K: I went from 46 seconds per step to 36 seconds (512*512). I'm going to switch to the ARM architecture. |
|
For those who want to try their hand at inference on mobile devices, I've put together a short guide; you'll need at least 4 GB of RAM. For now, the mobile version hasn't been optimized yet. |
…add gitattributes config
…al logic and command blocks
… enable attention-free optimization for mobile execution scripts
This reverts commit 0e00352.
|
After spending hours trying to get the code to work on Termux with Vulkan, I found myself unable to make it work. I think I went about it the wrong way, so the best thing to do might be to close this PR and start completely from scratch. It’s a failure, but hey, it’s no problem :). |
|
I'm going to start a new project. It's going to take a while, but I think this is the best approach: https://github.com/Juste-Leo2/bonsai.cpp |
The primary goal of this PR is to optimize internal operations to improve performance when running Bonsai on mobile devices.
Below is a quick guide on how to perform a standard inference on CUDA.
Required Models (Downloads)
Please download the following weights before testing:
Build Instructions
To test this branch, clone the repository and build the project with CUDA enabled:
Testing / Inference
Once built and the models are downloaded (make sure they are in your working directory), run the following command to test standard inference:
./build/bin/sd-cli \ --cfg-scale 1 \ --width 512 \ --height 512 \ --steps 4 \ --seed 42 \ -p "a cat" \ -o outputs/cat_test.png \ --diffusion-model bonsai_image_4b-q1_0.gguf \ --vae flux2-vae.safetensors \ --llm Qwen3-4B-UD-Q4_K_XL.gguf