Add final Gemma4 TODOs for hand-over

content/learning-paths/mobile-graphics-and-gaming/gemma4-kleidiai-sme2/_index.md

@@ -1,23 +1,24 @@
 ---
-title: Benchmark Gemma LiteRT-LM prefill performance with KleidiAI and SME2 on macOS
+title: Benchmark Gemma 4 LiteRT-LM prefill performance with KleidiAI and SME2 on macOS
 
 minutes_to_complete: 45
 
 draft: true
 cascade:
     draft: true
 
-who_is_this_for: This is an advanced topic for software developers and performance engineers who want a reproducible Gemma prefill benchmark workflow using LiteRT-LM, KleidiAI, and XNNPACK on macOS.
+who_is_this_for: This is an advanced topic for software developers and performance engineers who want a reproducible Gemma 4 prefill benchmark workflow using LiteRT-LM, KleidiAI, and XNNPACK on macOS.
 
 learning_objectives:
-    - Create a pinned workspace with LiteRT-LM, KleidiAI, and XNNPACK
-    - Install Bazelisk and required macOS prerequisites for LiteRT-LM builds
-    - Prepare a LiteRT-LM compatible `.litertlm` Gemma model from Hugging Face
-    - Run LiteRT-LM benchmark commands and measure prefill throughput on a reproducible setup
+    - Create a workspace with pinned LiteRT-LM and KleidiAI versions
+    - Create upstream-optimized and historical-baseline XNNPACK worktrees
+    - Download a LiteRT-LM-compatible Gemma 4 model from Hugging Face
+    - Compare baseline and upstream SME2 benchmark results with a cache-neutral workflow
 
 prerequisites:
-    - A SME2 device (macOS M4 on Apple Silicon)
+    - An SME2 device (this tutorial uses macOS on Apple M4)
     - Git, Homebrew, and Xcode Command Line Tools
+    - At least 25 GB of free disk space for model files and local builds
 author: Annie Tallund
 
 generate_summary_faq: true
@@ -61,19 +62,11 @@ further_reading:
         type: website
     - resource:
         title: XNNPACK repository
-        link: https://github.com/google/xnnpack
+        link: https://github.com/google/XNNPACK
         type: website
     - resource:
-        title: google/gemma-3n-E4B-it-litert-lm on Hugging Face
-        link: https://huggingface.co/google/gemma-3n-E4B-it-litert-lm
-        type: website
-    - resource:
-        title: google/gemma-3-4b-it on Hugging Face
-        link: https://huggingface.co/google/gemma-3-4b-it
-        type: website
-    - resource:
-        title: litert-community/Gemma3-4B-IT on Hugging Face
-        link: https://huggingface.co/litert-community/Gemma3-4B-IT
+        title: Gemma 4 E2B for LiteRT-LM on Hugging Face
+        link: https://huggingface.co/litert-community/gemma-4-E2B-it-litert-lm
         type: website
 
 ### FIXED, DO NOT MODIFY

content/learning-paths/mobile-graphics-and-gaming/gemma4-kleidiai-sme2/download_model.md

@@ -1,18 +1,14 @@
 ---
-title: Install prerequisites and download the model
+title: Install prerequisites and download Gemma 4
 weight: 4
 
 ### FIXED, DO NOT MODIFY
 layout: learningpathall
 ---
 
-## Overview
-
-In this section, you install the macOS dependencies required to build LiteRT-LM and prepare a Gemma model in LiteRT-LM `.litertlm` format.
-
 ## Install prerequisites on macOS
 
-Install Xcode Command Line Tools (if needed):
+Install Xcode Command Line Tools if they are not already available:
 
 ```bash
 xcode-select --install
@@ -27,65 +23,55 @@ brew install bazelisk
 Install the Hugging Face Hub CLI:
 
 ```bash
-python3 -m pip install -U "huggingface_hub[cli]"
+python3 -m pip install -U huggingface_hub
 ```
 
-Pin Bazel version `7.6.1` from the LiteRT-LM root:
+Confirm that LiteRT-LM pins Bazel `7.6.1`:
 
 ```bash
 cd $HOME/gemma4-prefill-bench/LiteRT-LM
-echo "7.6.1" > .bazelversion
+cat .bazelversion
 bazelisk version
 ```
 
-## Create model directory
+## Download Gemma 4 from Hugging Face
+
+Create a shared model directory in the workspace:
 
 ```bash
-mkdir -p $HOME/gemma4-prefill-bench/LiteRT-LM/models
+mkdir -p $HOME/gemma4-prefill-bench/models
+cd $HOME/gemma4-prefill-bench
 ```
 
-{{% notice Note %}}
-The benchmark commands in this Learning Path assume your model directory is under `LiteRT-LM/models/`.
-{{% /notice %}}
-
-## Prepare a LiteRT-LM-compatible model
+Download the CPU-compatible Gemma 4 E2B LiteRT-LM artifact:
 
-LiteRT-LM benchmark commands in this Learning Path use `litert_lm_advanced_main`, which expects a LiteRT-LM model artifact (`.litertlm`) for `--model_path`.
+```bash
+hf download litert-community/gemma-4-E2B-it-litert-lm \
+  gemma-4-E2B-it.litertlm \
+  --local-dir models/gemma-4-E2B-it-litert-lm
+```
 
-The `google/gemma-3-4b-it` repository on Hugging Face follows the Transformers layout (multiple `safetensors` files), so it is not directly consumable by `litert_lm_advanced_main`.
+The model repository is public. If Hugging Face asks for credentials, run
+`hf auth login` and repeat the download.
 
-Use a prebuilt LiteRT-LM model artifact instead:
+Verify the downloaded file:
 
 ```bash
-cd $HOME/gemma4-prefill-bench/LiteRT-LM/models
-hf auth login
-hf download google/gemma-3n-E4B-it-litert-lm \
-  --include "gemma-3n-E4B-it-int4.litertlm" \
-  --local-dir ./gemma-3n-E4B-it-litert-lm
+shasum -a 256 \
+  models/gemma-4-E2B-it-litert-lm/gemma-4-E2B-it.litertlm
+ls -lh models/gemma-4-E2B-it-litert-lm/gemma-4-E2B-it.litertlm
+```
 
-cp ./gemma-3n-E4B-it-litert-lm/gemma-3n-E4B-it-int4.litertlm \
-   ./gemma-3n-E4B-it-int4.litertlm
+The expected SHA-256 checksum is:
 
-ls -lh ./gemma-3n-E4B-it-int4.litertlm
+```output
+181938105e0eefd105961417e8da75903eacda102c4fce9ce90f50b97139a63c  models/gemma-4-E2B-it-litert-lm/gemma-4-E2B-it.litertlm
 ```
 
 {{% notice Note %}}
-The model repository is gated. Make sure your Hugging Face account has accepted the model terms before downloading.
+The file is about 2.6 GB. LiteRT-LM requires the `.litertlm` artifact; a
+Transformers repository containing only `safetensors` files is not a direct
+replacement.
 {{% /notice %}}
 
-Expected layout:
-
-```text
-LiteRT-LM/models/
-├── gemma-3n-E4B-it-int4.litertlm
-└── gemma-3n-E4B-it-litert-lm/
-    └── gemma-3n-E4B-it-int4.litertlm
-```
-
-## What you've accomplished and what's next
-
-In this section:
-- You installed macOS prerequisites and pinned Bazel for LiteRT-LM
-- You created the local model directory and prepared a LiteRT-LM-compatible `.litertlm` Gemma model
-
-In the next section, you will build and run the benchmark workflow.
+In the next section, you will build each XNNPACK variant and run the benchmark.

content/learning-paths/mobile-graphics-and-gaming/gemma4-kleidiai-sme2/introduction.md

@@ -8,36 +8,48 @@ layout: learningpathall
 
 ## Why this benchmark matters
 
-This Learning Path gives you a reproducible process to evaluate Gemma prefill performance improvements when using SME2-optimized paths across LiteRT-LM, KleidiAI, and XNNPACK.
+This Learning Path gives you a reproducible process to evaluate Gemma 4 CPU
+performance with the upstream XNNPACK SME2 Int4 and Int2 paths used through
+LiteRT-LM and KleidiAI.
 
-The flow in this guide is based on a pinned set of repository commits and a local benchmark command (`litert_lm_advanced_main --benchmark`) so your numbers are comparable across runs.
+The workflow records the repository revisions and uses the
+`litert_lm_advanced_main --benchmark` command so that you can compare results
+across XNNPACK variants.
 
 ## What you will do
 
 You will complete the workflow in this order:
 
-1. Create a local workspace and clone LiteRT-LM, KleidiAI, and XNNPACK at tested commits.
-2. Install macOS prerequisites and Bazelisk, then pin Bazel to version `7.6.1`.
-3. Prepare a LiteRT-LM-compatible `.litertlm` model in `LiteRT-LM/models`.
-4. Build LiteRT-LM and run benchmark and sample prompts.
-
-In the next section, you will set up the workspace with pinned repository commits.
+1. Create a workspace and clone LiteRT-LM `v0.16.1`, KleidiAI `v1.30.0`, and
+   upstream XNNPACK with the SME2 Int4 and Int2 support.
+2. Create a historical XNNPACK baseline worktree from before the SME2 support.
+3. Download the Gemma 4 E2B `.litertlm` model from Hugging Face.
+4. Build and benchmark the baseline and upstream-optimized variants with
+   identical settings.
+5. Compare steady-state prefill and decode throughput.
 
 ## Find out if your device supports SME2
 
-Confirm your machine exposes SME/SME2 to user space:
+Confirm the architecture and macOS SME feature flags:
 
 ```bash
 uname -m
-sysctl -a | grep -Ei 'sme2|sme'
+sysctl hw.optional.arm.FEAT_SME
+sysctl hw.optional.arm.FEAT_SME2
+```
+
+The expected output on a supported Apple M4 system is:
+
+```output
+arm64
+hw.optional.arm.FEAT_SME: 1
+hw.optional.arm.FEAT_SME2: 1
 ```
 
-Expected result:
-- `uname -m` should report `arm64`
-- `sysctl` output should include an SME/SME2 capability entry
+If either feature reports `0`, XNNPACK cannot dispatch the SME2 kernels and the
+performance comparison is not valid.
 
-If you do not see SME/SME2 in the `sysctl` output, this benchmark can still run, but XNNPACK/KleidiAI will dispatch non-SME2 kernels and your prefill throughput will likely be lower.
+For a deeper validation, see [Test your SME2 development
+environment](./learning-paths/cross-platform/multiplying-matrices-with-sme2/2-check-your-environment).
 
-{{% notice Note %}}
-For a deeper validation (compiler + runtime streaming mode checks), see the cross-platform Learning Path section [Test your SME2 development environment](./learning-paths/cross-platform/multiplying-matrices-with-sme2/2-check-your-environment).
-{{% /notice %}}
+In the next section, you will set up the benchmark workspace.

content/learning-paths/mobile-graphics-and-gaming/gemma4-kleidiai-sme2/run_llm.md

@@ -1,66 +1,131 @@
 ---
-title: Build and run benchmarks
+title: Build and compare benchmarks
 weight: 5
 
 ### FIXED, DO NOT MODIFY
 layout: learningpathall
 ---
 
+## Build and benchmark one XNNPACK variant
 
-## Build LiteRT-LM with local XNNPACK and KleidiAI
+Create directories for the shared Bazel output and benchmark results:
 
-Build from the LiteRT-LM repository:
+```bash
+cd $HOME/gemma4-prefill-bench
+mkdir -p bazel-output-base results
+```
+
+Define a shell function that builds one XNNPACK tree and immediately runs the
+same three-iteration benchmark:
 
 ```bash
-cd $HOME/gemma4-prefill-bench/LiteRT-LM
-bazel build \
+set -o pipefail
+
+run_variant() {
+  local variant="$1"
+  local xnnpack_dir="$2"
+  local root="$HOME/gemma4-prefill-bench"
+  local output_base="$root/bazel-output-base/ab"
+  local model="$root/models/gemma-4-E2B-it-litert-lm/gemma-4-E2B-it.litertlm"
+  local sdk_version
+  local binary
+
+  sdk_version="$(xcrun --sdk macosx --show-sdk-version)"
+
+  cd "$root/LiteRT-LM"
+  bazelisk --output_base="$output_base" build \
     --config=macos_arm64 \
-    --macos_sdk_version="$(xcrun --sdk macosx --show-sdk-version)" \
-    --override_repository=XNNPACK=../xnnpack \
-    --override_repository=KleidiAI=../kleidiai \
+    --macos_sdk_version="$sdk_version" \
+    --override_repository=XNNPACK="$xnnpack_dir" \
+    --override_repository=KleidiAI="$root/kleidiai" \
     //runtime/engine:litert_lm_advanced_main
 
+  binary="$(bazelisk --output_base="$output_base" info bazel-bin)/runtime/engine/litert_lm_advanced_main"
+
+  "$binary" \
+    --backend=cpu \
+    --model_path="$model" \
+    --benchmark \
+    --benchmark_prefill_tokens=512 \
+    --benchmark_decode_tokens=128 \
+    --num_cpu_threads=4 \
+    --disable_cache=true \
+    --async=false \
+    --num_iterations=3 \
+    --report_peak_memory_footprint \
+    --metric_proto_file_path="$root/results/${variant}.pb" \
+    2>&1 | tee "$root/results/${variant}.log"
+}
 ```
 
-## Run benchmark
+The first build compiles the full LiteRT-LM dependency graph. Reusing one
+Bazel output base means that changing the XNNPACK override rebuilds only the
+affected actions for later variants.
+
+Run the historical baseline and the upstream-optimized variant:
 
 ```bash
-bazel-bin/runtime/engine/litert_lm_advanced_main \
-  --backend=cpu \
-  --model_path=models/gemma-3n-E4B-it-int4.litertlm \
-  --benchmark \
-  --benchmark_prefill_tokens=512 \
-  --benchmark_decode_tokens=128 \
-  --num_cpu_threads=4 \
-  --disable_cache=false
+run_variant baseline "$HOME/gemma4-prefill-bench/xnnpack-baseline"
+run_variant optimized "$HOME/gemma4-prefill-bench/xnnpack"
 ```
 
-Example output pattern:
+{{% notice Note %}}
+Use `--disable_cache=true` for the A/B comparison. The tested Gemma 4 artifact
+contains XNNPACK weight-cache fingerprints that are recognized by the upstream
+optimized tree but not by the historical baseline. Enabling caches therefore
+makes initialization time and memory use non-comparable.
+{{% /notice %}}
+
+## Read the benchmark output
 
-```text
+Each iteration prints output similar to:
+
+```output
 --------------------------------------------------
-  Time to first token: 0.51 s
+  Time to first token: 0.43 s
 --------------------------------------------------
   Prefill Turns (Total 1 turns):
-    Prefill Turn 1: Processed 512 tokens in 477.85ms duration.
-      Prefill Speed: 1071.47 tokens/sec.
+    Prefill Turn 1: Processed 512 tokens in 399.049292ms duration.
+      Prefill Speed: 1283.05 tokens/sec.
 --------------------------------------------------
   Decode Turns (Total 1 turns):
-    Decode Turn 1: Processed 128 tokens in 3.917262s duration.
-      Decode Speed: 32.68 tokens/sec.
+    Decode Turn 1: Processed 128 tokens in 4.037520542s duration.
+      Decode Speed: 31.70 tokens/sec.
 --------------------------------------------------
 ```
 
-Run quick output sanity checks:
+Treat iteration 1 as a warm-up and compare the mean of iterations 2 and 3.
+
+## Tested results on Apple M4 Pro
+
+The following results were measured on a 12-core Apple M4 Pro with 24 GB of
+RAM, macOS SDK 26.5, four CPU threads, and caches disabled. Frequency and
+thermal state were not fixed, so use the values as a functional comparison.
+
+| XNNPACK variant | Prefill tokens/s | Change | Decode tokens/s | Change |
+| --- | ---: | ---: | ---: | ---: |
+| Historical baseline | 930.57 | - | 54.78 | - |
+| Upstream SME2 Int4 and Int2 | 1263.22 | +35.7% | 29.76 | -45.7% |
+
+On this system, the upstream SME2 paths provide a clear prefill improvement but
+regress decode for the four-thread workload. Results can differ by model
+signature, thread count, SoC, operating system, and thermal state. Measure
+prefill and decode separately for your target workload.
+
+## Run a prompt sanity check
+
+After building a variant, use its generated binary for a short prompt:
 
 ```bash
-bazel-bin/runtime/engine/litert_lm_advanced_main \
-  --backend=cpu \
-  --model_path=models/gemma-3n-E4B-it-int4.litertlm \
-  --input_prompt="What is the capital of France?"
+binary="$(bazelisk --output_base="$HOME/gemma4-prefill-bench/bazel-output-base/ab" info bazel-bin)/runtime/engine/litert_lm_advanced_main"
 
-bazel-bin/runtime/engine/litert_lm_advanced_main \
+"$binary" \
   --backend=cpu \
-  --model_path=models/gemma-3n-E4B-it-int4.litertlm \
-  --input_prompt="What is the most difficult winter Olympic sport?"
+  --model_path="$HOME/gemma4-prefill-bench/models/gemma-4-E2B-it-litert-lm/gemma-4-E2B-it.litertlm" \
+  --input_prompt="What is the capital of France?" \
+  --max_output_tokens=16 \
+  --num_cpu_threads=4 \
+  --async=false
 ```
+
+The tested model answers that the capital of France is Paris.