diff --git a/content/learning-paths/mobile-graphics-and-gaming/gemma4-kleidiai-sme2/_index.md b/content/learning-paths/mobile-graphics-and-gaming/gemma4-kleidiai-sme2/_index.md index 58f670584b..baaae9aba2 100755 --- a/content/learning-paths/mobile-graphics-and-gaming/gemma4-kleidiai-sme2/_index.md +++ b/content/learning-paths/mobile-graphics-and-gaming/gemma4-kleidiai-sme2/_index.md @@ -1,5 +1,5 @@ --- -title: Benchmark Gemma LiteRT-LM prefill performance with KleidiAI and SME2 on macOS +title: Benchmark Gemma 4 LiteRT-LM prefill performance with KleidiAI and SME2 on macOS minutes_to_complete: 45 @@ -7,17 +7,18 @@ draft: true cascade: draft: true -who_is_this_for: This is an advanced topic for software developers and performance engineers who want a reproducible Gemma prefill benchmark workflow using LiteRT-LM, KleidiAI, and XNNPACK on macOS. +who_is_this_for: This is an advanced topic for software developers and performance engineers who want a reproducible Gemma 4 prefill benchmark workflow using LiteRT-LM, KleidiAI, and XNNPACK on macOS. learning_objectives: - - Create a pinned workspace with LiteRT-LM, KleidiAI, and XNNPACK - - Install Bazelisk and required macOS prerequisites for LiteRT-LM builds - - Prepare a LiteRT-LM compatible `.litertlm` Gemma model from Hugging Face - - Run LiteRT-LM benchmark commands and measure prefill throughput on a reproducible setup + - Create a workspace with pinned LiteRT-LM and KleidiAI versions + - Create upstream-optimized and historical-baseline XNNPACK worktrees + - Download a LiteRT-LM-compatible Gemma 4 model from Hugging Face + - Compare baseline and upstream SME2 benchmark results with a cache-neutral workflow prerequisites: - - A SME2 device (macOS M4 on Apple Silicon) + - An SME2 device (this tutorial uses macOS on Apple M4) - Git, Homebrew, and Xcode Command Line Tools + - At least 25 GB of free disk space for model files and local builds author: Annie Tallund generate_summary_faq: true @@ -61,19 +62,11 @@ further_reading: type: website - resource: title: XNNPACK repository - link: https://github.com/google/xnnpack + link: https://github.com/google/XNNPACK type: website - resource: - title: google/gemma-3n-E4B-it-litert-lm on Hugging Face - link: https://huggingface.co/google/gemma-3n-E4B-it-litert-lm - type: website - - resource: - title: google/gemma-3-4b-it on Hugging Face - link: https://huggingface.co/google/gemma-3-4b-it - type: website - - resource: - title: litert-community/Gemma3-4B-IT on Hugging Face - link: https://huggingface.co/litert-community/Gemma3-4B-IT + title: Gemma 4 E2B for LiteRT-LM on Hugging Face + link: https://huggingface.co/litert-community/gemma-4-E2B-it-litert-lm type: website ### FIXED, DO NOT MODIFY diff --git a/content/learning-paths/mobile-graphics-and-gaming/gemma4-kleidiai-sme2/download_model.md b/content/learning-paths/mobile-graphics-and-gaming/gemma4-kleidiai-sme2/download_model.md index dfb041af0b..c42bde00f4 100644 --- a/content/learning-paths/mobile-graphics-and-gaming/gemma4-kleidiai-sme2/download_model.md +++ b/content/learning-paths/mobile-graphics-and-gaming/gemma4-kleidiai-sme2/download_model.md @@ -1,18 +1,14 @@ --- -title: Install prerequisites and download the model +title: Install prerequisites and download Gemma 4 weight: 4 ### FIXED, DO NOT MODIFY layout: learningpathall --- -## Overview - -In this section, you install the macOS dependencies required to build LiteRT-LM and prepare a Gemma model in LiteRT-LM `.litertlm` format. - ## Install prerequisites on macOS -Install Xcode Command Line Tools (if needed): +Install Xcode Command Line Tools if they are not already available: ```bash xcode-select --install @@ -27,65 +23,55 @@ brew install bazelisk Install the Hugging Face Hub CLI: ```bash -python3 -m pip install -U "huggingface_hub[cli]" +python3 -m pip install -U huggingface_hub ``` -Pin Bazel version `7.6.1` from the LiteRT-LM root: +Confirm that LiteRT-LM pins Bazel `7.6.1`: ```bash cd $HOME/gemma4-prefill-bench/LiteRT-LM -echo "7.6.1" > .bazelversion +cat .bazelversion bazelisk version ``` -## Create model directory +## Download Gemma 4 from Hugging Face + +Create a shared model directory in the workspace: ```bash -mkdir -p $HOME/gemma4-prefill-bench/LiteRT-LM/models +mkdir -p $HOME/gemma4-prefill-bench/models +cd $HOME/gemma4-prefill-bench ``` -{{% notice Note %}} -The benchmark commands in this Learning Path assume your model directory is under `LiteRT-LM/models/`. -{{% /notice %}} - -## Prepare a LiteRT-LM-compatible model +Download the CPU-compatible Gemma 4 E2B LiteRT-LM artifact: -LiteRT-LM benchmark commands in this Learning Path use `litert_lm_advanced_main`, which expects a LiteRT-LM model artifact (`.litertlm`) for `--model_path`. +```bash +hf download litert-community/gemma-4-E2B-it-litert-lm \ + gemma-4-E2B-it.litertlm \ + --local-dir models/gemma-4-E2B-it-litert-lm +``` -The `google/gemma-3-4b-it` repository on Hugging Face follows the Transformers layout (multiple `safetensors` files), so it is not directly consumable by `litert_lm_advanced_main`. +The model repository is public. If Hugging Face asks for credentials, run +`hf auth login` and repeat the download. -Use a prebuilt LiteRT-LM model artifact instead: +Verify the downloaded file: ```bash -cd $HOME/gemma4-prefill-bench/LiteRT-LM/models -hf auth login -hf download google/gemma-3n-E4B-it-litert-lm \ - --include "gemma-3n-E4B-it-int4.litertlm" \ - --local-dir ./gemma-3n-E4B-it-litert-lm +shasum -a 256 \ + models/gemma-4-E2B-it-litert-lm/gemma-4-E2B-it.litertlm +ls -lh models/gemma-4-E2B-it-litert-lm/gemma-4-E2B-it.litertlm +``` -cp ./gemma-3n-E4B-it-litert-lm/gemma-3n-E4B-it-int4.litertlm \ - ./gemma-3n-E4B-it-int4.litertlm +The expected SHA-256 checksum is: -ls -lh ./gemma-3n-E4B-it-int4.litertlm +```output +181938105e0eefd105961417e8da75903eacda102c4fce9ce90f50b97139a63c models/gemma-4-E2B-it-litert-lm/gemma-4-E2B-it.litertlm ``` {{% notice Note %}} -The model repository is gated. Make sure your Hugging Face account has accepted the model terms before downloading. +The file is about 2.6 GB. LiteRT-LM requires the `.litertlm` artifact; a +Transformers repository containing only `safetensors` files is not a direct +replacement. {{% /notice %}} -Expected layout: - -```text -LiteRT-LM/models/ -├── gemma-3n-E4B-it-int4.litertlm -└── gemma-3n-E4B-it-litert-lm/ - └── gemma-3n-E4B-it-int4.litertlm -``` - -## What you've accomplished and what's next - -In this section: -- You installed macOS prerequisites and pinned Bazel for LiteRT-LM -- You created the local model directory and prepared a LiteRT-LM-compatible `.litertlm` Gemma model - -In the next section, you will build and run the benchmark workflow. +In the next section, you will build each XNNPACK variant and run the benchmark. diff --git a/content/learning-paths/mobile-graphics-and-gaming/gemma4-kleidiai-sme2/introduction.md b/content/learning-paths/mobile-graphics-and-gaming/gemma4-kleidiai-sme2/introduction.md index b8617cd494..1450ab77e8 100644 --- a/content/learning-paths/mobile-graphics-and-gaming/gemma4-kleidiai-sme2/introduction.md +++ b/content/learning-paths/mobile-graphics-and-gaming/gemma4-kleidiai-sme2/introduction.md @@ -8,36 +8,48 @@ layout: learningpathall ## Why this benchmark matters -This Learning Path gives you a reproducible process to evaluate Gemma prefill performance improvements when using SME2-optimized paths across LiteRT-LM, KleidiAI, and XNNPACK. +This Learning Path gives you a reproducible process to evaluate Gemma 4 CPU +performance with the upstream XNNPACK SME2 Int4 and Int2 paths used through +LiteRT-LM and KleidiAI. -The flow in this guide is based on a pinned set of repository commits and a local benchmark command (`litert_lm_advanced_main --benchmark`) so your numbers are comparable across runs. +The workflow records the repository revisions and uses the +`litert_lm_advanced_main --benchmark` command so that you can compare results +across XNNPACK variants. ## What you will do You will complete the workflow in this order: -1. Create a local workspace and clone LiteRT-LM, KleidiAI, and XNNPACK at tested commits. -2. Install macOS prerequisites and Bazelisk, then pin Bazel to version `7.6.1`. -3. Prepare a LiteRT-LM-compatible `.litertlm` model in `LiteRT-LM/models`. -4. Build LiteRT-LM and run benchmark and sample prompts. - -In the next section, you will set up the workspace with pinned repository commits. +1. Create a workspace and clone LiteRT-LM `v0.16.1`, KleidiAI `v1.30.0`, and + upstream XNNPACK with the SME2 Int4 and Int2 support. +2. Create a historical XNNPACK baseline worktree from before the SME2 support. +3. Download the Gemma 4 E2B `.litertlm` model from Hugging Face. +4. Build and benchmark the baseline and upstream-optimized variants with + identical settings. +5. Compare steady-state prefill and decode throughput. ## Find out if your device supports SME2 -Confirm your machine exposes SME/SME2 to user space: +Confirm the architecture and macOS SME feature flags: ```bash uname -m -sysctl -a | grep -Ei 'sme2|sme' +sysctl hw.optional.arm.FEAT_SME +sysctl hw.optional.arm.FEAT_SME2 +``` + +The expected output on a supported Apple M4 system is: + +```output +arm64 +hw.optional.arm.FEAT_SME: 1 +hw.optional.arm.FEAT_SME2: 1 ``` -Expected result: -- `uname -m` should report `arm64` -- `sysctl` output should include an SME/SME2 capability entry +If either feature reports `0`, XNNPACK cannot dispatch the SME2 kernels and the +performance comparison is not valid. -If you do not see SME/SME2 in the `sysctl` output, this benchmark can still run, but XNNPACK/KleidiAI will dispatch non-SME2 kernels and your prefill throughput will likely be lower. +For a deeper validation, see [Test your SME2 development +environment](./learning-paths/cross-platform/multiplying-matrices-with-sme2/2-check-your-environment). -{{% notice Note %}} -For a deeper validation (compiler + runtime streaming mode checks), see the cross-platform Learning Path section [Test your SME2 development environment](./learning-paths/cross-platform/multiplying-matrices-with-sme2/2-check-your-environment). -{{% /notice %}} +In the next section, you will set up the benchmark workspace. diff --git a/content/learning-paths/mobile-graphics-and-gaming/gemma4-kleidiai-sme2/run_llm.md b/content/learning-paths/mobile-graphics-and-gaming/gemma4-kleidiai-sme2/run_llm.md index 71eee0f472..82f2bff626 100644 --- a/content/learning-paths/mobile-graphics-and-gaming/gemma4-kleidiai-sme2/run_llm.md +++ b/content/learning-paths/mobile-graphics-and-gaming/gemma4-kleidiai-sme2/run_llm.md @@ -1,66 +1,131 @@ --- -title: Build and run benchmarks +title: Build and compare benchmarks weight: 5 ### FIXED, DO NOT MODIFY layout: learningpathall --- +## Build and benchmark one XNNPACK variant -## Build LiteRT-LM with local XNNPACK and KleidiAI +Create directories for the shared Bazel output and benchmark results: -Build from the LiteRT-LM repository: +```bash +cd $HOME/gemma4-prefill-bench +mkdir -p bazel-output-base results +``` + +Define a shell function that builds one XNNPACK tree and immediately runs the +same three-iteration benchmark: ```bash -cd $HOME/gemma4-prefill-bench/LiteRT-LM -bazel build \ +set -o pipefail + +run_variant() { + local variant="$1" + local xnnpack_dir="$2" + local root="$HOME/gemma4-prefill-bench" + local output_base="$root/bazel-output-base/ab" + local model="$root/models/gemma-4-E2B-it-litert-lm/gemma-4-E2B-it.litertlm" + local sdk_version + local binary + + sdk_version="$(xcrun --sdk macosx --show-sdk-version)" + + cd "$root/LiteRT-LM" + bazelisk --output_base="$output_base" build \ --config=macos_arm64 \ - --macos_sdk_version="$(xcrun --sdk macosx --show-sdk-version)" \ - --override_repository=XNNPACK=../xnnpack \ - --override_repository=KleidiAI=../kleidiai \ + --macos_sdk_version="$sdk_version" \ + --override_repository=XNNPACK="$xnnpack_dir" \ + --override_repository=KleidiAI="$root/kleidiai" \ //runtime/engine:litert_lm_advanced_main + binary="$(bazelisk --output_base="$output_base" info bazel-bin)/runtime/engine/litert_lm_advanced_main" + + "$binary" \ + --backend=cpu \ + --model_path="$model" \ + --benchmark \ + --benchmark_prefill_tokens=512 \ + --benchmark_decode_tokens=128 \ + --num_cpu_threads=4 \ + --disable_cache=true \ + --async=false \ + --num_iterations=3 \ + --report_peak_memory_footprint \ + --metric_proto_file_path="$root/results/${variant}.pb" \ + 2>&1 | tee "$root/results/${variant}.log" +} ``` -## Run benchmark +The first build compiles the full LiteRT-LM dependency graph. Reusing one +Bazel output base means that changing the XNNPACK override rebuilds only the +affected actions for later variants. + +Run the historical baseline and the upstream-optimized variant: ```bash -bazel-bin/runtime/engine/litert_lm_advanced_main \ - --backend=cpu \ - --model_path=models/gemma-3n-E4B-it-int4.litertlm \ - --benchmark \ - --benchmark_prefill_tokens=512 \ - --benchmark_decode_tokens=128 \ - --num_cpu_threads=4 \ - --disable_cache=false +run_variant baseline "$HOME/gemma4-prefill-bench/xnnpack-baseline" +run_variant optimized "$HOME/gemma4-prefill-bench/xnnpack" ``` -Example output pattern: +{{% notice Note %}} +Use `--disable_cache=true` for the A/B comparison. The tested Gemma 4 artifact +contains XNNPACK weight-cache fingerprints that are recognized by the upstream +optimized tree but not by the historical baseline. Enabling caches therefore +makes initialization time and memory use non-comparable. +{{% /notice %}} + +## Read the benchmark output -```text +Each iteration prints output similar to: + +```output -------------------------------------------------- - Time to first token: 0.51 s + Time to first token: 0.43 s -------------------------------------------------- Prefill Turns (Total 1 turns): - Prefill Turn 1: Processed 512 tokens in 477.85ms duration. - Prefill Speed: 1071.47 tokens/sec. + Prefill Turn 1: Processed 512 tokens in 399.049292ms duration. + Prefill Speed: 1283.05 tokens/sec. -------------------------------------------------- Decode Turns (Total 1 turns): - Decode Turn 1: Processed 128 tokens in 3.917262s duration. - Decode Speed: 32.68 tokens/sec. + Decode Turn 1: Processed 128 tokens in 4.037520542s duration. + Decode Speed: 31.70 tokens/sec. -------------------------------------------------- ``` -Run quick output sanity checks: +Treat iteration 1 as a warm-up and compare the mean of iterations 2 and 3. + +## Tested results on Apple M4 Pro + +The following results were measured on a 12-core Apple M4 Pro with 24 GB of +RAM, macOS SDK 26.5, four CPU threads, and caches disabled. Frequency and +thermal state were not fixed, so use the values as a functional comparison. + +| XNNPACK variant | Prefill tokens/s | Change | Decode tokens/s | Change | +| --- | ---: | ---: | ---: | ---: | +| Historical baseline | 930.57 | - | 54.78 | - | +| Upstream SME2 Int4 and Int2 | 1263.22 | +35.7% | 29.76 | -45.7% | + +On this system, the upstream SME2 paths provide a clear prefill improvement but +regress decode for the four-thread workload. Results can differ by model +signature, thread count, SoC, operating system, and thermal state. Measure +prefill and decode separately for your target workload. + +## Run a prompt sanity check + +After building a variant, use its generated binary for a short prompt: ```bash -bazel-bin/runtime/engine/litert_lm_advanced_main \ - --backend=cpu \ - --model_path=models/gemma-3n-E4B-it-int4.litertlm \ - --input_prompt="What is the capital of France?" +binary="$(bazelisk --output_base="$HOME/gemma4-prefill-bench/bazel-output-base/ab" info bazel-bin)/runtime/engine/litert_lm_advanced_main" -bazel-bin/runtime/engine/litert_lm_advanced_main \ +"$binary" \ --backend=cpu \ - --model_path=models/gemma-3n-E4B-it-int4.litertlm \ - --input_prompt="What is the most difficult winter Olympic sport?" + --model_path="$HOME/gemma4-prefill-bench/models/gemma-4-E2B-it-litert-lm/gemma-4-E2B-it.litertlm" \ + --input_prompt="What is the capital of France?" \ + --max_output_tokens=16 \ + --num_cpu_threads=4 \ + --async=false ``` + +The tested model answers that the capital of France is Paris. diff --git a/content/learning-paths/mobile-graphics-and-gaming/gemma4-kleidiai-sme2/setup_workspace.md b/content/learning-paths/mobile-graphics-and-gaming/gemma4-kleidiai-sme2/setup_workspace.md index 8221122ea8..82cff9c896 100644 --- a/content/learning-paths/mobile-graphics-and-gaming/gemma4-kleidiai-sme2/setup_workspace.md +++ b/content/learning-paths/mobile-graphics-and-gaming/gemma4-kleidiai-sme2/setup_workspace.md @@ -1,44 +1,79 @@ --- -title: Set up workspace and pinned repositories +title: Set up the benchmark workspace weight: 3 ### FIXED, DO NOT MODIFY layout: learningpathall --- -## Overview +## Create the workspace -In this section, you prepare a local workspace and clone the exact LiteRT-LM, KleidiAI, and XNNPACK commits used for this benchmark flow. - -Create a working directory and move into it: +Create a working directory outside the Learning Paths repository: ```bash mkdir -p $HOME/gemma4-prefill-bench cd $HOME/gemma4-prefill-bench ``` -## Clone tested commits +## Clone the tested LiteRT-LM and KleidiAI versions -Clone the repositories and check out pinned commits: +Clone LiteRT-LM and check out the tested `v0.16.1` commit: ```bash git clone https://github.com/google-ai-edge/LiteRT-LM.git LiteRT-LM -git -C LiteRT-LM checkout 41d6b964c21f225af9dad7088231b024c369adc1 +git -C LiteRT-LM checkout 924e79c91542761242244e4f1651851f822e4cbb +``` + +Clone KleidiAI and check out `v1.30.0`, which the upstream XNNPACK SME2 paths +require: +```bash git clone https://github.com/ARM-software/kleidiai.git kleidiai -git -C kleidiai checkout 72a6a50c1dba714ff27a908e8dd54be3628794b0 +git -C kleidiai checkout 74b1a12d3620c89dae4766de640e064952000f4d +``` + +## Create the XNNPACK variants + +Clone upstream XNNPACK. Its default branch contains the merged SME2 Int4 and +Int2 support and is the optimized tree: + +```bash +git clone https://github.com/google/XNNPACK.git xnnpack +git -C xnnpack log -1 --oneline +``` + +Record the commit printed by the command so you can identify the exact upstream +revision used for your results. + +Create a historical baseline worktree from the common XNNPACK revision before +the SME2 Int4 and Int2 support: -git clone https://github.com/google/xnnpack.git xnnpack -git -C xnnpack checkout deb87c027acd2cca4e59fd1f227e523a1640d24c +```bash +git -C xnnpack worktree add --detach ../xnnpack-baseline \ + eb452a766d8f1075b5491f22309e8f09bd31d828 ``` -Expected layout: +Verify the repository revisions: + +```bash +git -C LiteRT-LM rev-parse HEAD +git -C kleidiai rev-parse HEAD +git -C xnnpack rev-parse HEAD +git -C xnnpack-baseline rev-parse HEAD +``` + +The baseline command should print +`eb452a766d8f1075b5491f22309e8f09bd31d828`. + +The workspace layout is now: -```text +```output gemma4-prefill-bench/ ├── kleidiai/ ├── LiteRT-LM/ -└── xnnpack/ +├── xnnpack/ +└── xnnpack-baseline/ ``` -In the next section, you will install prerequisites and download the model. +In the next section, you will install the prerequisites and download the +Gemma 4 model.