You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
BitNetTwoStageDecode (stage-1 planes-0–3 scan + exact candidate rescoring with a provable top-k-equality bound) and the sampling API sampleFromCandidates/ScoredToken (llm-core/.../SamplingUtils.kt:154-190) shipped with PR #353 — but no generation loop calls them. OptimizedLLMRuntime exposes no pre-lm_head hidden state, and the CLI's BitNet branch decodes single-stage full-vocab. The two-stage path is reachable only from tests and library callers.
This is the residual #337 flagged as "a separate, small design": the hook was never added.
Proposed change
Runtime: expose the last-position hidden state before the lm_head projection (hook or return-shape addition in OptimizedLLMRuntime), so a family can substitute candidate scoring for the full-vocab matmul. Keep it format-driven — the hook is generic; only the BitNet path uses it today.
Problem
BitNetTwoStageDecode(stage-1 planes-0–3 scan + exact candidate rescoring with a provable top-k-equality bound) and the sampling APIsampleFromCandidates/ScoredToken(llm-core/.../SamplingUtils.kt:154-190) shipped with PR #353 — but no generation loop calls them.OptimizedLLMRuntimeexposes no pre-lm_head hidden state, and the CLI's BitNet branch decodes single-stage full-vocab. The two-stage path is reachable only from tests and library callers.This is the residual #337 flagged as "a separate, small design": the hook was never added.
Proposed change
OptimizedLLMRuntime), so a family can substitute candidate scoring for the full-vocab matmul. Keep it format-driven — the hook is generic; only the BitNet path uses it today.BITNET_PLANES(with [bitnet] Tied lm_head never engages the planes kernel on 2B4T — FP32-widened head, ~0.45 tok/s #357 landed this covers 2B4T's tied head). Single-stage remains the default everywhere else.Verification
Refs: #335, #337 (residual), #357 (tied head), SKaiNET-developers/SKaiNET#1150.