Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -566,6 +566,11 @@ if(Python3_Interpreter_FOUND)
# WIDER tile grid than the model's dimensions (the FastFlowLM repack pads intermediate_size to 512 columns, IM
# 8960 -> 9216) instead of reporting it as an unreadable reshape. Pure python + numpy, no model, no device.
add_test(NAME q4nx_diff_grid COMMAND ${Python3_EXECUTABLE} ${CMAKE_SOURCE_DIR}/tests/q4nx_diff_grid_test.py)
# The pooled per-depth live-vs-offline verdict (tools/mtp_depth_pool.py): the brief's "within 2 pp at every
# depth" is judged on 8 prompts x 64 tokens POOLED, which is two or three npu-hold tickets of <= 3 min, and
# pooling those logs by eye produced two wrong numbers in this lane. Fixture tickets only -- no model, no
# device; the recorded 8x64 window is the tool's known-answer check and is run by hand with --selftest.
add_test(NAME mtp_depth_pool COMMAND ${Python3_EXECUTABLE} ${CMAKE_SOURCE_DIR}/tests/mtp_depth_pool_test.py)
# the t2 chunk kind (docs/npu.md "The t2 chunk kind"): synthetic only, because the real-model
# leg needs 14 GB of Bonsai-2 images. Run tests/q4nx_t2_test.py without --no-model for the gate.
add_test(NAME q4nx_t2 COMMAND ${Python3_EXECUTABLE} ${CMAKE_SOURCE_DIR}/tests/q4nx_t2_test.py --no-model)
Expand Down
45 changes: 45 additions & 0 deletions tests/fixtures/mtp_clause55/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
<!--
Copyright 2026 bong-water-water-bong
SPDX-License-Identifier: Apache-2.0

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
# The three tickets that produced the recorded 8x64 window

These are the only in-tree inputs that let `tools/mtp_depth_pool.py --selftest` check its **known answer**: the
per-depth live-vs-offline table the brief is judged on (8 prompts x 64 tokens, k=4, tau=0,
`+0.00 / +0.55 / +0.09 / +1.08 pp` at depths 1-4, 75.14 / 63.24 / 73.26 / 84.13 %). Without them the check needs the
original ticket logs, which live in one home directory on one box — so CI could break the tool without noticing.

**Provenance, so these are not a restatement of the answer.** The three `.txt` files are the lines the parser reads,
extracted verbatim from the `npu-hold` tickets of 2026-11-05 (the same tickets whose numbers the lane recorded):

```sh
pat="forward ready:|prompt [0-9]+:|depth [0-9]+: |match p[0-9]+:|== live vs offline at the SAME|== ticket "
grep -E "$pat" ticket-clause55-p0-20261105-115131.log > p0.txt # 15 lines, prompt 0
grep -E "$pat" ticket-clause55-p1-4-20261105-115551.log > p1-4.txt # 35 lines, prompts 1-4
grep -E "$pat" ticket-clause55-p5-7-20261105-120713.log > p5-7.txt # 28 lines, prompts 5-7
```

**Why `.txt` and not `.log`**: `.gitignore:19` is `*.log`, so the first cut of this directory committed the README and
silently left all three tickets behind — the test then passed on the machine that had them lying around and failed in CI
with `tests/fixtures/mtp_clause55 is incomplete`. A fixture that can be dropped by a rename or a re-clone is worse than
one with a different suffix, so the files carry `.txt` and `tests/mtp_depth_pool_test.py` asserts with `git ls-files`
that all three are **tracked**, which fails locally instead of only in CI.

Everything the tool ignores is dropped — the engine chatter, the ELF loads, the per-prompt `per-depth pN:` compact
lines, the phase split. Nothing the tool reads is edited: the `[ Ns]` elapsed stamps, the `[live joint …]` brackets
and the `match pN:` headers are exactly as the tickets printed them, and the pool still has to reproduce the recorded
counts from them (that is what makes it a known answer rather than a fixture written to agree with the code).

`tests/mtp_depth_pool_test.py` runs `--selftest` over these three and fails if the pooled cells drift by one count.
15 changes: 15 additions & 0 deletions tests/fixtures/mtp_clause55/p0.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
== ticket 'clause55 p0' 20261105-115131 hold=16min argv: --live --match --prompts 1 --prompt-base 0 --tokens 64 --k 4 --tau 0
[ 169s] forward ready: vocab 248320, max_batch_rows 16, hidden source post-final-norm
[ 252s] prompt 0: TEXT IDENTICAL, 64 tokens, accepted 44/80, passes/token 0.625, stream 72 rows, 0 mismatch
[ 258s] match p0: 20 offline chains at this loop's own positions (stream 72 rows)
[ 258s] depth 1: live 90.00 % (18/20, reached) offline conditional 90.00 % (18/20) delta +0.00 pp [live joint 90.00 %, offline marginal 90.00 %]
[ 258s] depth 2: live 72.22 % (13/18, reached) offline conditional 72.22 % (13/18) delta +0.00 pp [live joint 65.00 %, offline marginal 65.00 %]
[ 258s] depth 3: live 53.85 % (7/13, reached) offline conditional 53.85 % (7/13) delta +0.00 pp [live joint 35.00 %, offline marginal 35.00 %]
[ 258s] depth 4: live 85.71 % (6/7, reached) offline conditional 85.71 % (6/7) delta +0.00 pp [live joint 30.00 %, offline marginal 35.00 %]
[ 258s] first verdict difference on the same token: stream position 13, depth 4: live reject, offline accept
[ 258s] == live vs offline at the SAME positions, per-depth acceptance CONDITIONAL on reaching the depth (the 2 pp bar):
[ 258s] depth 1: live 90.00 % (18/20) offline 90.00 % (18/20) delta +0.00 pp PASS
[ 258s] depth 2: live 72.22 % (13/18) offline 72.22 % (13/18) delta +0.00 pp PASS
[ 258s] depth 3: live 53.85 % (7/13) offline 53.85 % (7/13) delta +0.00 pp PASS
[ 258s] depth 4: live 85.71 % (6/7) offline 85.71 % (6/7) delta +0.00 pp PASS
== ticket 'clause55 p0' exited 0; log: /home/bcloud/wt/mtp-window/ticket-clause55-p0-20261105-115131.log
35 changes: 35 additions & 0 deletions tests/fixtures/mtp_clause55/p1-4.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
== ticket 'clause55 p1-4' 20261105-115551 hold=16min argv: --live --match --prompts 4 --prompt-base 1 --tokens 64 --k 4 --tau 0
[ 66s] forward ready: vocab 248320, max_batch_rows 16, hidden source post-final-norm
[ 177s] prompt 1: TEXT IDENTICAL, 64 tokens, accepted 28/140, passes/token 1.109, stream 72 rows, 0 mismatch
[ 190s] match p1: 35 offline chains at this loop's own positions (stream 72 rows)
[ 190s] depth 1: live 60.00 % (21/35, reached) offline conditional 60.00 % (21/35) delta +0.00 pp [live joint 60.00 %, offline marginal 60.00 %]
[ 190s] depth 2: live 19.05 % (4/21, reached) offline conditional 19.05 % (4/21) delta +0.00 pp [live joint 11.43 %, offline marginal 22.86 %]
[ 191s] depth 3: live 75.00 % (3/4, reached) offline conditional 75.00 % (3/4) delta +0.00 pp [live joint 8.57 %, offline marginal 26.47 %]
[ 191s] depth 4: live 0.00 % (0/3, reached) offline conditional 0.00 % (0/3) delta +0.00 pp [live joint 0.00 %, offline marginal 8.82 %]
[ 191s] first verdict difference on the same token: stream position 9, depth 2: live reject, offline accept
[ 302s] prompt 2: TEXT IDENTICAL, 64 tokens, accepted 36/112, passes/token 0.875, stream 72 rows, 0 mismatch
[ 313s] match p2: 28 offline chains at this loop's own positions (stream 72 rows)
[ 313s] depth 1: live 60.71 % (17/28, reached) offline conditional 60.71 % (17/28) delta +0.00 pp [live joint 60.71 %, offline marginal 60.71 %]
[ 313s] depth 2: live 64.71 % (11/17, reached) offline conditional 64.71 % (11/17) delta +0.00 pp [live joint 39.29 %, offline marginal 42.86 %]
[ 313s] depth 3: live 45.45 % (5/11, reached) offline conditional 50.00 % (5/10) delta -4.55 pp > 2 pp [live joint 17.86 %, offline marginal 25.93 %]
[ 313s] depth 4: live 60.00 % (3/5, reached) offline conditional 60.00 % (3/5) delta +0.00 pp [live joint 10.71 %, offline marginal 19.23 %]
[ 313s] first verdict difference on the same token: stream position 13, depth 2: live reject, offline accept
[ 407s] prompt 3: TEXT IDENTICAL, 64 tokens, accepted 50/64, passes/token 0.500, stream 72 rows, 0 mismatch
[ 414s] match p3: 16 offline chains at this loop's own positions (stream 72 rows)
[ 414s] depth 1: live 87.50 % (14/16, reached) offline conditional 87.50 % (14/16) delta +0.00 pp [live joint 87.50 %, offline marginal 87.50 %]
[ 414s] depth 2: live 85.71 % (12/14, reached) offline conditional 85.71 % (12/14) delta +0.00 pp [live joint 75.00 %, offline marginal 75.00 %]
[ 414s] depth 3: live 100.00 % (12/12, reached) offline conditional 100.00 % (11/11) delta +0.00 pp [live joint 75.00 %, offline marginal 73.33 %]
[ 414s] depth 4: live 100.00 % (12/12, reached) offline conditional 100.00 % (11/11) delta +0.00 pp [live joint 75.00 %, offline marginal 80.00 %]
[ 414s] first verdict difference on the same token: stream position 9, depth 4: live reject, offline accept
[ 494s] prompt 4: TEXT IDENTICAL, 64 tokens, accepted 41/104, passes/token 0.812, stream 72 rows, 0 mismatch
[ 503s] match p4: 26 offline chains at this loop's own positions (stream 72 rows)
[ 503s] depth 1: live 76.92 % (20/26, reached) offline conditional 76.92 % (20/26) delta +0.00 pp [live joint 76.92 %, offline marginal 76.92 %]
[ 503s] depth 2: live 55.00 % (11/20, reached) offline conditional 52.63 % (10/19) delta +2.37 pp > 2 pp [live joint 42.31 %, offline marginal 40.00 %]
[ 503s] depth 3: live 54.55 % (6/11, reached) offline conditional 50.00 % (5/10) delta +4.55 pp > 2 pp [live joint 23.08 %, offline marginal 20.00 %]
[ 503s] depth 4: live 66.67 % (4/6, reached) offline conditional 60.00 % (3/5) delta +6.67 pp > 2 pp [live joint 15.38 %, offline marginal 12.50 %]
[ 503s] == live vs offline at the SAME positions, per-depth acceptance CONDITIONAL on reaching the depth (the 2 pp bar):
[ 503s] depth 1: live 68.57 % (72/105) offline 68.57 % (72/105) delta +0.00 pp PASS
[ 503s] depth 2: live 52.78 % (38/72) offline 52.11 % (37/71) delta +0.67 pp PASS
[ 503s] depth 3: live 68.42 % (26/38) offline 68.57 % (24/35) delta -0.15 pp PASS
[ 503s] depth 4: live 73.08 % (19/26) offline 70.83 % (17/24) delta +2.24 pp FAIL
== ticket 'clause55 p1-4' exited 0; log: /home/bcloud/wt/mtp-window/ticket-clause55-p1-4-20261105-115551.log
28 changes: 28 additions & 0 deletions tests/fixtures/mtp_clause55/p5-7.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
== ticket 'clause55 p5-7' 20261105-120713 hold=16min argv: --live --match --prompts 3 --prompt-base 5 --tokens 64 --k 4 --tau 0
[ 49s] forward ready: vocab 248320, max_batch_rows 16, hidden source post-final-norm
[ 110s] prompt 5: TEXT IDENTICAL, 64 tokens, accepted 48/60, passes/token 0.484, stream 72 rows, 0 mismatch
[ 114s] match p5: 15 offline chains at this loop's own positions (stream 72 rows)
[ 114s] depth 1: live 86.67 % (13/15, reached) offline conditional 86.67 % (13/15) delta +0.00 pp [live joint 86.67 %, offline marginal 86.67 %]
[ 114s] depth 2: live 92.31 % (12/13, reached) offline conditional 92.31 % (12/13) delta +0.00 pp [live joint 80.00 %, offline marginal 80.00 %]
[ 114s] depth 3: live 100.00 % (12/12, reached) offline conditional 100.00 % (12/12) delta +0.00 pp [live joint 80.00 %, offline marginal 80.00 %]
[ 114s] depth 4: live 91.67 % (11/12, reached) offline conditional 91.67 % (11/12) delta +0.00 pp [live joint 73.33 %, offline marginal 80.00 %]
[ 114s] first verdict difference on the same token: stream position 9, depth 4: live reject, offline accept
[ 165s] prompt 6: TEXT IDENTICAL, 64 tokens, accepted 50/60, passes/token 0.469, stream 72 rows, 0 mismatch
[ 169s] match p6: 15 offline chains at this loop's own positions (stream 72 rows)
[ 169s] depth 1: live 93.33 % (14/15, reached) offline conditional 93.33 % (14/15) delta +0.00 pp [live joint 93.33 %, offline marginal 93.33 %]
[ 169s] depth 2: live 85.71 % (12/14, reached) offline conditional 85.71 % (12/14) delta +0.00 pp [live joint 80.00 %, offline marginal 80.00 %]
[ 169s] depth 3: live 100.00 % (12/12, reached) offline conditional 100.00 % (12/12) delta +0.00 pp [live joint 80.00 %, offline marginal 80.00 %]
[ 169s] depth 4: live 100.00 % (12/12, reached) offline conditional 100.00 % (11/11) delta +0.00 pp [live joint 80.00 %, offline marginal 78.57 %]
[ 231s] prompt 7: TEXT IDENTICAL, 64 tokens, accepted 41/104, passes/token 0.812, stream 72 rows, 0 mismatch
[ 237s] match p7: 26 offline chains at this loop's own positions (stream 72 rows)
[ 237s] depth 1: live 73.08 % (19/26, reached) offline conditional 73.08 % (19/26) delta +0.00 pp [live joint 73.08 %, offline marginal 73.08 %]
[ 237s] depth 2: live 57.89 % (11/19, reached) offline conditional 55.56 % (10/18) delta +2.34 pp > 2 pp [live joint 42.31 %, offline marginal 44.00 %]
[ 237s] depth 3: live 54.55 % (6/11, reached) offline conditional 50.00 % (5/10) delta +4.55 pp > 2 pp [live joint 23.08 %, offline marginal 28.00 %]
[ 237s] depth 4: live 83.33 % (5/6, reached) offline conditional 80.00 % (4/5) delta +3.33 pp > 2 pp [live joint 19.23 %, offline marginal 20.00 %]
[ 237s] first verdict difference on the same token: stream position 11, depth 2: live reject, offline accept
[ 237s] == live vs offline at the SAME positions, per-depth acceptance CONDITIONAL on reaching the depth (the 2 pp bar):
[ 237s] depth 1: live 82.14 % (46/56) offline 82.14 % (46/56) delta +0.00 pp PASS
[ 237s] depth 2: live 76.09 % (35/46) offline 75.56 % (34/45) delta +0.53 pp PASS
[ 237s] depth 3: live 85.71 % (30/35) offline 85.29 % (29/34) delta +0.42 pp PASS
[ 237s] depth 4: live 93.33 % (28/30) offline 92.86 % (26/28) delta +0.48 pp PASS
== ticket 'clause55 p5-7' exited 0; log: /home/bcloud/wt/mtp-window/ticket-clause55-p5-7-20261105-120713.log
Loading
Loading