Skip to content

hyperloom custom workload: the long-context score's unit names the context but not the decoded-token count, so rows whose denominators differ by 2x read as the same measurement #1337

Closed

Description

@bong-water-water-bong

throughput_unit for a long-context result names the context the server prefilled and nothing else:

"tok/s (decode at 18762-token ctx)"

The score is a decode rate, so its value is set by two things: that context, and the number of tokens the decode actually produced — which is the score's own denominator, and which is not the number the caller asked for, because the model can stop at EOS first.

Measured on this box, 2026-11-07

Two arms, same model, same ctx, same levers, same default draft cap, both drafting (drafting.serve.status: drafted), both gate PASS:

arm HL_LONGTOK decoded (longctx.decode_tokens) score unit string
arm_74_longctx_default_post1238.json 128 128 13.0592 tok/s (decode at 18762-token ctx)
arm_74_longctx_default_tok512.json 512 278 14.1784 tok/s (decode at 18762-token ctx)

HL_LONGTOK=512 produced 278 tokens — the server stopped at EOS after 278 (stop processing: n_tokens = 19039, truncated = 0), so this is not harness truncation. The bench_summary.longctx.decode_tokens field does record 278 honestly, and levers.HL_LONGTOK records 512, so the artifact is not lying.

The unit is the part that travels with the number. Both rows carry a byte-identical unit string while their denominators differ by more than 2×, so output_throughput values cut from them are not comparable and nothing in the unit says so. This lane nearly made exactly that comparison (14.18 vs 13.40 as if the decode length were constant).

The unit was introduced precisely because a long-context score's meaning is not obvious — it names the context for that reason. The decode count changes the rate by at least 6 % across the two rows above, so leaving it out is the same omission one level down.

Why this is the same class as #1201/#1202 and #1237

The record must name the input that produced the number. scripts.sha256 names the code, build/model name the binary and weights, box.npu_* names the neighbour, drafting.serve.status names the decode's mode — and the score's denominator should name itself too.

Reproducer

Device-free, from the two artifacts above: json.load(...)["throughput_unit"] are equal; ["bench_summary"]["longctx"]["decode_tokens"] are 128 and 278.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions