Skip to content

Feat: Optimize operations for Bonsai on mobile devices - #2

Closed
Juste-Leo2 wants to merge 21 commits into
masterfrom
bonsai_dev
Closed

Feat: Optimize operations for Bonsai on mobile devices#2
Juste-Leo2 wants to merge 21 commits into
masterfrom
bonsai_dev

Conversation

@Juste-Leo2

@Juste-Leo2 Juste-Leo2 commented May 31, 2026

Copy link
Copy Markdown
Owner

The primary goal of this PR is to optimize internal operations to improve performance when running Bonsai on mobile devices.

Note : I'm giving myself some time to implement this at a leisurely pace. Don't expect much from this PR just yet; I'll be experimenting with it.

Below is a quick guide on how to perform a standard inference on CUDA.

Required Models (Downloads)
Please download the following weights before testing:

Build Instructions
To test this branch, clone the repository and build the project with CUDA enabled:

git clone https://github.com/Juste-Leo2/stable-diffusion.cpp
cd stable-diffusion.cpp
git checkout bonsai_dev

# Initialize and update submodules
git submodule init
git submodule update

# Build the project
mkdir build && cd build
cmake .. -DSD_CUDA=ON
cd ..
cmake --build build -j$(nproc)

Testing / Inference
Once built and the models are downloaded (make sure they are in your working directory), run the following command to test standard inference:

./build/bin/sd-cli \
  --cfg-scale 1 \
  --width 512 \
  --height 512 \
  --steps 4 \
  --seed 42 \
  -p "a cat" \
  -o outputs/cat_test.png \
  --diffusion-model bonsai_image_4b-q1_0.gguf \
  --vae flux2-vae.safetensors \
  --llm Qwen3-4B-UD-Q4_K_XL.gguf

gemini-code-assist[bot]

This comment was marked as outdated.

@Juste-Leo2

Juste-Leo2 commented Jun 1, 2026

Copy link
Copy Markdown
Owner Author

I've done an initial code cleanup.
Now I'm going to focus on streamlining the backends to keep the ones most useful for our needs and make the code as universal as possible.
So I've decided to keep:

  • CPU (universal)
  • Vulkan (universal)
  • CUDA (NVIDIA GPU)
  • Metal (Apple ecosystem)
  • HIP (AMD GPU)
  • SYCL (Intel GPU)

The other backends are either niche, not useful for our application or too complicated to compile

…enVINO, RPC, VirtGPU, WebGPU, zDNN, ZenDNN)

Keep only the 6 targeted backends: CPU, CUDA, HIP, SYCL, Vulkan, Metal.

- Delete all backend source directories from ggml/src/
- Delete unused backend headers from ggml/include/
- Remove ggml_add_backend() and register_backend() calls from CMakeLists.txt and ggml-backend-reg.cpp
- Remove dead cmake options and ggml-config.cmake.in dependencies
- Clean up root CMakeLists.txt (SD_OPENCL, SD_MUSA) and misc docs refs
- Verified: CUDA and CPU inference both work correctly (cat test 512x512)
- AVX2 nrc=4: fused 4-row x 1-col kernel for GEMV (by=0),
          4 x nrc=2 decomposition for GEMM (by>0)
- Generic fallback nrc=4 for non-AVX2
- Dispatcher: removed ne11/ir1_end checks to allow nrc>1 for GEMV
- tmp[64] for up to 4 output columns
- by_val=0 when fewer columns than nrc to keep kernels safe
- Copy loop bounded by num_cols to avoid out-of-bounds writes
@Juste-Leo2

Juste-Leo2 commented Jun 2, 2026

Copy link
Copy Markdown
Owner Author

Alright, here are the initial performance improvements on the x86 architecture using an i5-13600K: I went from 46 seconds per step to 36 seconds (512*512). I'm going to switch to the ARM architecture.

@Juste-Leo2

Copy link
Copy Markdown
Owner Author

For those who want to try their hand at inference on mobile devices, I've put together a short guide; you'll need at least 4 GB of RAM. For now, the mobile version hasn't been optimized yet.

@Juste-Leo2

Copy link
Copy Markdown
Owner Author

After spending hours trying to get the code to work on Termux with Vulkan, I found myself unable to make it work. I think I went about it the wrong way, so the best thing to do might be to close this PR and start completely from scratch.

It’s a failure, but hey, it’s no problem :).

@Juste-Leo2 Juste-Leo2 closed this Jun 3, 2026
@Juste-Leo2

Copy link
Copy Markdown
Owner Author

I'm going to start a new project. It's going to take a while, but I think this is the best approach: https://github.com/Juste-Leo2/bonsai.cpp

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant