I measured the cost of one forward pass at small batch sizes on two MI50s with Qwen3.8-27B Q8_0, build b10811, since that is what the speculative verify step pays. llama-bench, -p N -n 0 -r 5, -ub 2048 -b 2048 -fa on, flash attention on, f16 KV. Milliseconds per batch.
| tokens |
tensor split, repack (default) |
tensor split, --no-repack |
single card, repack |
layer split, repack |
| 1 |
31.0 |
32.4 |
47.6 |
47.5 |
| 2 |
36.2 |
37.4 |
57.0 |
56.0 |
| 4 |
42.3 |
51.4 |
63.7 |
61.0 |
| 8 |
68.7 |
84.9 |
94.5 |
90.5 |
| 9 |
106.9 |
112.1 |
152.9 |
147.9 |
| 13 |
113.7 |
119.3 |
156.2 |
150.4 |
| 16 |
118.7 |
124.7 |
158.6 |
152.3 |
| 32 |
146.1 |
188.8 |
168.5 |
162.2 |
| 33 |
190.9 |
207.2 |
270.8 |
263.1 |
| 64 |
250.6 |
322.7 |
290.1 |
282.4 |
| 128 |
452.9 |
568.5 |
425.3 |
425.5 |
Three things I read from this:
- The repack path is faster than the upstream path at every size on these cards, so the repack is the right default here.
- The step from 8 to 9 tokens is the narrow mat-vec handing over to the 32-wide tile: 38 ms on two cards, 58 ms on one, and then flat until 32. On a single card 9 tokens cost the same as 32. A speculative verify of 9 to 16 tokens pays for 32.
- The narrow path itself grows about 5.4 ms per extra token on two cards and 6.7 ms on one. For this model that is roughly 8 to 10 TOPS of int8 work per card, so it is compute-limited well before the tile takes over. At 4 tokens (the verify step for a 3-token MTP draft, which is our production setting) a pass costs 42 ms against 31 ms for a single token.
The practical effect on MTP: with a 3-token draft, code decodes at about 70 tokens per second here; a 12-token draft accepts 6.3 tokens per round but the round costs 131 ms instead of 53, so it lands at 48. The adaptive-depth controller in upstream PR 27210 cannot help on this hardware for that reason.
Question: is the 2 to 8 token mat-vec expected to be compute-limited at this level on gfx906, or is there headroom you know of in the dot-product loop? If the 4-token pass could get close to the 1-token pass, fixed depth 3 would gain about 20 percent on code here. Raw jsonl for all four arms available if useful.
I measured the cost of one forward pass at small batch sizes on two MI50s with Qwen3.8-27B Q8_0, build b10811, since that is what the speculative verify step pays. llama-bench,
-p N -n 0 -r 5,-ub 2048 -b 2048 -fa on, flash attention on, f16 KV. Milliseconds per batch.--no-repackThree things I read from this:
The practical effect on MTP: with a 3-token draft, code decodes at about 70 tokens per second here; a 12-token draft accepts 6.3 tokens per round but the round costs 131 ms instead of 53, so it lands at 48. The adaptive-depth controller in upstream PR 27210 cannot help on this hardware for that reason.
Question: is the 2 to 8 token mat-vec expected to be compute-limited at this level on gfx906, or is there headroom you know of in the dot-product loop? If the 4-token pass could get close to the 1-token pass, fixed depth 3 would gain about 20 percent on code here. Raw jsonl for all four arms available if useful.