You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
hyperloom custom workload: the long-context score's unit names the context but not the decoded-token count, so rows whose denominators differ by 2x read as the same measurement #1337
throughput_unit for a long-context result names the context the server prefilled and nothing else:
"tok/s (decode at 18762-token ctx)"
The score is a decode rate, so its value is set by two things: that context, and the number of tokens the decode actually produced — which is the score's own denominator, and which is not the number the caller asked for, because the model can stop at EOS first.
Measured on this box, 2026-11-07
Two arms, same model, same ctx, same levers, same default draft cap, both drafting (drafting.serve.status: drafted), both gate PASS:
arm
HL_LONGTOK
decoded (longctx.decode_tokens)
score
unit string
arm_74_longctx_default_post1238.json
128
128
13.0592
tok/s (decode at 18762-token ctx)
arm_74_longctx_default_tok512.json
512
278
14.1784
tok/s (decode at 18762-token ctx)
HL_LONGTOK=512 produced 278 tokens — the server stopped at EOS after 278 (stop processing: n_tokens = 19039, truncated = 0), so this is not harness truncation. The bench_summary.longctx.decode_tokens field does record 278 honestly, and levers.HL_LONGTOK records 512, so the artifact is not lying.
The unit is the part that travels with the number. Both rows carry a byte-identical unit string while their denominators differ by more than 2×, so output_throughput values cut from them are not comparable and nothing in the unit says so. This lane nearly made exactly that comparison (14.18 vs 13.40 as if the decode length were constant).
The unit was introduced precisely because a long-context score's meaning is not obvious — it names the context for that reason. The decode count changes the rate by at least 6 % across the two rows above, so leaving it out is the same omission one level down.
The record must name the input that produced the number. scripts.sha256 names the code, build/model name the binary and weights, box.npu_* names the neighbour, drafting.serve.status names the decode's mode — and the score's denominator should name itself too.
Reproducer
Device-free, from the two artifacts above: json.load(...)["throughput_unit"] are equal; ["bench_summary"]["longctx"]["decode_tokens"] are 128 and 278.
Activity
bench: the long-context score pins its own denominator, and says when…