Skip to content

Board-verify the FunctionGemma KV-cache decode loop on the SL2610 #12

Description

@michalharakal

The KV-cache 2-graph decode (GemmaKvDecoder, GEMMA_KV=1) is CPU-verified (transformers PR #245) but the
board loop is an UNVERIFIED draft. On the first SL2610 run, confirm each silent-corruption trap:

  • Per-block K-vs-V output orderGemmaKvDecoder.kFirstInOutput=false ((V,K) per return-SSA); flip to true if the first run yields garbage tokens.
  • --output=@file formatIreeRuntime.invokeFiles assumes RAW bytes; strip the .npy header on read if the board writes NumPy.
  • gemma_with_past input arg order — trace-derived; confirm against the compiled vmfb.
  • Dynamic-concat — the x?x256 sentinel-relax leaves shape inference to iree-compile; confirm the vmfb accepts it (else fixed-pad+mask fallback).
  • --task_topology_group_count accepted by the board iree-run-module (gated to revert).
  • vmfbs built with the Torq-fork iree-compile (g165e12a).

Done: the loop reproduces the oracle [262146,236769,3255,718,498,1373,262152,106], PERF-LOGBOOK.md shows the O(seq²)→O(seq) collapse vs ~6000 ms/token, then make GEMMA_KV=1 the default in Pipeline.kt.

Runbook: sl2610-function-calling/docs/GEMMA-KV-BOARD-LOOP.md. Tracker: docs/GEMMA-KV-INT8.md.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions