Repository navigation
docs: status after steps 1-3, HRX re-verified at the first daily bump - #16
Conversation
- README, PORTING, npu.md, lemonade.md: steps 1-3 landed (3a-3c; the layer kernel is still not built from source), the mlx recipe, the XDNA pin; Laya (step 4) is next. Drops stale "this PR" rows and "Not yet" items that steps 2 and 3 closed. - hrx.md: re-verified at llama.cpp f1a0aca / hrx-system 51b1739 (#13): e2e and ctest pass; fresh llama-bench numbers including BF16; new "Not yet": ggml-hrx accepts IQ3_XXS matmuls at scheduling time but cannot run them, so sub-4-bit UD files fail on HRX0 with no CPU fallback. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
3a227c8 to
2d35ea8
Compare
…l with a pre-fault count
Two findings from investigating a faulting run-7 arm, both bigger than the check I was doing.
1. THE FAULT IS TIMESTAMPED. server.log stamps the fault line in
minutes.seconds.milliseconds.microseconds: '0.07.279.709 E common_sampler_sample: HRX returned NaN
logits'. Format confirmed against that arm's own req=1 (7.28s is where a 1-request arm dies), not
assumed.
This removes a confound in MY OWN H2 test: a whole-window total cannot distinguish 'more evictions
caused the fault' from 'the fault caused more evictions', because a faulting arm crashes, may be
retried, and outlives its arm - so its window can simply be longer. Counting only up to the fault
instant is a causal test: a fault cannot cause evictions that precede it. 315-evict-per-arm.py now
prints pre-fault (causal) and window (continuity) columns, and refuses a causal conclusion when no
fault arm has a recoverable time.
2. TWO OF RUN 7'S THREE FAULTS WERE FACE A FOLLOWED BY A FULL SERVER CRASH. p7c and p7l logged the
NaN line then died - logs end in a gdb backtrace ('#16 _start', '[Inferior 1 detached]'). That is why
their detectors recorded transport_failures=1/RemoteDisconnected: the transport failure is DOWNSTREAM
of the NaN, not a separate failure mode. The console called both FaceA nan=1 and on the server side
that is right.
So run 7's faults are two Face-A NaN crashes (7.3s, 57.8s into their arms) and one Face-B parse
failure (81.2s, req=20).
VALIDATION: both faces matched, so a Face-B arm is not mistaken for one with no fault time. p3l
81.181s (Face B, via 'unparsed Content-only'), p7c 57.838s, p7l 7.280s. False-positive check across
ALL run-7 arms: exactly those 3 yield a time and 0 of the 8 clean arms do. A parser that found faults
in clean arms would fabricate the H2 effect, so that check is the one that matters.
To run three probes back to back I wrote three scripts each shaped as 'while ps | grep <predecessor>; do sleep; done; clean; launch the next one'. Three disc sessions then appeared at once, including two copies of 315lout4, and 315chain-run.log was empty - stage 1 never started - while stage 2 launched anyway, because the guard is 'wait WHILE the predecessor is running', which is satisfied immediately when the predecessor does not exist at all. The guard answers 'is it running now?' rather than 'did it ever run and finish?'. Two sessions then composed the regime simultaneously, so neither arm's cell means what its label says and every arm in that window is unusable. Why this is the same defect one layer up: it is defect #14 with the sign flipped. There, absence of output was read as a clean result; here, absence of a process is read as completion. Both are a predicate on a proxy standing in for the event of interest, and both silently proceed. The campaign now has four instances of this shape - #13, where an EXTRA_ENV named a feature the build does not implement and absence of effect was read as absence of an effect; #14; #16; and the pkill-on-name traps of #11/#12 where killing the wrapper was read as stopping the run. Records the fix and what I did instead: the correct guard records evidence of the predecessor's completion via a sentinel line written after its last arm, and additionally requires that it started, but chaining is not worth that machinery here, so I abandoned it and ran the stages directly one at a time, verifying each launch by its own log header before starting the next - slower, but it cannot silently skip a stage, which is the failure mode that cost two composed regimes and about twenty minutes of box time this session is already over budget on. Cost: two simultaneous compositions whose arms are discarded, the section-40 chain probe never ran and must be re-issued, and one duplicate 315lout4 session, with no conclusions drawn from any arm in that window.
…nt-backend carries the warm-up (#495) Lemonade's standard benchmark had Lemonade's own Qwen3.8-27B entry drafting with the model's MTP head (`--spec-type draft-mtp`, label "mtp") at 24-28 tok/s while the onebit recipe ran the same file without a drafter at 11-12. And on the one-hop route lemond starts llama-server itself, so the warm-up request serve sends a ready backend (#492) never ran. - launch_for: a .gguf with <architecture>.nextn_predict_layers > 0 on hrx gets `--spec-type draft-mtp` (no -md: llama-server loads the head from -m), as Lemonade's llamacpp recipe does; --mtp-max / --mtp-p-min apply, recipes see drafter=mtp. Not with --mtp / --dflash, --parallel > 1, --embedding / --reranking, on cpu, or with --no-mtp / ONEBIT_NO_MTP=1. Both serve and the one-hop plan. - --print-backend: an hrx chat plan carries "warmup": {path, body}, serve's own warm-up request (shared code now), for Lemonade's onebit recipe to send before it marks the model loaded (lemonade fork #16). ONEBIT_NO_WARMUP=1 leaves it out. - tests/print_backend.sh: the default drafter, its opt-outs and the warmup key. - docs/serve.md: both, with the Lemonade bench numbers (27B chat/code 11.1/11.9 -> 26.2/29.5 tok/s). Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
…ed verdict does not
Review of this tool found both halves, and both are right:
1. **The comments channel was missing.** Only `pulls/{n}/reviews` was fetched, so a `VERDICT:` line posted as an
issue comment read as no verdict at all -- and that is where most lanes put theirs. Now both channels are merged
into the one list `reviewed_at` asks about, each paginated.
2. **The token alone is not a verdict.** `no VERDICT at abc1234 yet` contains both the word and the sha and means the
opposite; the old `'VERDICT' in body` accepted it. A verdict now has to START a line (`VERDICT_LINE`, allowing
markdown decoration before it), which is how the lanes actually write them.
The arm this tool shipped with was too weak to catch (2) -- *"I read abc1234 carefully"* has no VERDICT token at all,
so it passes even with the broken matcher. Two arms now pin the real shapes: the disclaiming sentence is **not** a
verdict, and `**VERDICT: READY** `abc1234`` still is. Reverting the matcher fails 2 arms.
**Honest about the evidence:** the review named four PRs as comment-only verdicts (llama.cpp#280, site#17/#16/#15).
I re-measured all four: each has reviews **and** comment verdicts, so the reviews channel already answered there, and
sweeping all 15 open PRs across the six repos finds **0** where the two channels disagree today. So this is a
robustness fix, not a live miscount -- the tool was right by accident on today`'s queue, and would have been wrong on
the next lane that only comments. Self-test ok, ctest pr_review_queue_selftest Passed, copyright rc 0.
…heir CURRENT head (#2322) * tools/ci/pr_review_queue.py: which open PRs still need a verdict at their CURRENT head I have hand-rolled this query in bash roughly sixty times tonight, and it was wrong at least twice in ways that matter: * `gh api` defaults to `per_page=30`. `engine#2183` has **30 reviews**, so the newest ones were off the page: my scan reported it unverdicted at a head I had already verdicted, and I nearly said on the record that the verdict was never given. (`engine#333` was the same story from the other side: verdicted while I was looking.) * A review is a claim about a HEAD. The question is not "has this PR been reviewed" but "does any review name the head it has now", and a PR that moved since its verdict is back in the queue -- which is what the 7-hex head in a verdict line is for. pr_review_queue.py the fleet's six repos pr_review_queue.py --repos engine a subset pr_review_queue.py --min-age-hours 1 only past the 1 h gate Both list endpoints page until exhausted. Merged and draft PRs are out of the queue by definition: a draft is not asking for a verdict, and a merged PR cannot take one. Split so it can be tested without the network: `reviewed_at(reviews, head)` is a pure function over the JSON, and six arms pin the cases that decide the answer -- a verdict naming the head counts; a longer sha of the same head counts; a verdict at a DIFFERENT head does not (the case the queue exists for); a review with no verdict line does not; a null body does not raise; and **CHANGES is a verdict** (the PR was reviewed, the answer was no). Live run on this tree, which disagreed with my hand scan and was right: gpu-kernels #71 0.40h head=8bf40b9 hip-hrx: keep dense prefill and small-batch K-quant ... pr_review_queue: 1 open PR(s) with no verdict at their current head Only the self-test is registered as a ctest row: the live mode needs `gh` and the network, which a row must not. `ctest -R pr_review_queue` -> Passed 0.06 s; `copyright --check` rc 0; `ci_targets_check --check` rc 0. * pr_review_queue: arm the VERDICT guard, which the first six arms did not pin Removing `if 'VERDICT' in body and head in body:` to plain `if head in body:` left `--self-test` at **rc 0**: every arm passed a body that contained BOTH the word and the head, so none of them could tell the guard from the comparison. The shape that separates them is the common one — a reviewer comment that quotes the sha it looked at: 'I read abc1234 carefully and it looks fine to me' -> not a verdict That is now an arm, and with it the same mutation is caught: FAIL a comment that NAMES the head but is not a verdict does not count: want False, got True Seven arms. This is the same defect I have been flagging in other lanes' PRs all night (an arm that tests the easy direction), found in my own tool by mutating it rather than trusting it. * pr_review_queue: a verdict posted as a COMMENT counts, and a disclaimed verdict does not Review of this tool found both halves, and both are right: 1. **The comments channel was missing.** Only `pulls/{n}/reviews` was fetched, so a `VERDICT:` line posted as an issue comment read as no verdict at all -- and that is where most lanes put theirs. Now both channels are merged into the one list `reviewed_at` asks about, each paginated. 2. **The token alone is not a verdict.** `no VERDICT at abc1234 yet` contains both the word and the sha and means the opposite; the old `'VERDICT' in body` accepted it. A verdict now has to START a line (`VERDICT_LINE`, allowing markdown decoration before it), which is how the lanes actually write them. The arm this tool shipped with was too weak to catch (2) -- *"I read abc1234 carefully"* has no VERDICT token at all, so it passes even with the broken matcher. Two arms now pin the real shapes: the disclaiming sentence is **not** a verdict, and `**VERDICT: READY** `abc1234`` still is. Reverting the matcher fails 2 arms. **Honest about the evidence:** the review named four PRs as comment-only verdicts (llama.cpp#280, site#17/#16/#15). I re-measured all four: each has reviews **and** comment verdicts, so the reviews channel already answered there, and sweeping all 15 open PRs across the six repos finds **0** where the two channels disagree today. So this is a robustness fix, not a live miscount -- the tool was right by accident on today`'s queue, and would have been wrong on the next lane that only comments. Self-test ok, ctest pr_review_queue_selftest Passed, copyright rc 0. --------- Co-authored-by: lane <lane@local> Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
…ork (#2425) Three pairs in one session on 2026-11-14, and nothing in the fleet noticed any of them until a reviewer had read both sides: engine#2418/#2419 both ADDED tools/ci/hip_f16_audit_pinned.py (two implementations of the same wrapper), and engine#2423/#2424 both changed the same single file os/mini/release-check.sh for the same issue. Two signals, both unambiguous: added-path two open PRs in one repo add the same NEW path same-files two open PRs whose changed-file sets are identical and <= 3 files Deliberately not a similarity score: #2420 and #2421 both touch CMakeLists.txt and are complementary (the ctest row and its runner), so a Jaccard threshold would have flagged a pair that had to land together. Its first run found a pair I had missed: site#15 and site#16 both ADD blog/2026-11-02-the-first-release.md -- main does not have that path, and the two versions differ by 67 lines. Reported on #16. Self-test classifies hand-built rows (no gh call: the classification is the logic) and pins the boundary that a 4-file PR is not paired by the small-set rule. Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Docs catch-up after #9-#13 (and #15).
f1a0aca, hrx-system51b1739) on Strix Halo:tests/hrx_lemonade_e2e.shPASS (Paris on Vulkan0 and HRX0),ctest3/3. New llama-bench numbers (pp512/tg128): Q4_K_M Vulkan0 14097/349, HRX0 21990/314.unsupported HRX node ... iq3_xxs. So UD-Q2_K_XL, UD-IQ2_M and UD-IQ1_S fail on HRX0 with no CPU fallback. They run on Vulkan0.ZINC is not covered here: it waits for #14.
🤖 Generated with Claude Code