test(#1356): assert the rendered label, not where identifiers sit in map.js - #1933
Conversation
|
Reviewing this properly shortly; flagging the CI result first so you are not left guessing. It failed on something unrelated to what this PR changes: That is a customizer persistence check, not one of the three a11y assertions you are replacing. Two things worth knowing before you go looking:
Worth one rerun before treating it as real. If it fails the same way twice, it is worth a look and I will help chase it. On the PR itself: replacing a source grep with a check on what the function renders is the right direction, and the mutation table showing what each version catches is exactly the evidence I would want. Full review to follow. |
|
Recycling both of your PRs, and it is worth saying why, because it is a trap that has cost this queue a lot of time today. Both of your runs are That fully explains #1912. Its re-run at 11:44 still tests a base from before #1932, which is the fix for the master breakage I caused this morning by merging #1929 and #1930 together. So it keeps failing on For #1933 it is less clear-cut. Its run started at 09:31, which is the same minute #1932 merged, so its base may or may not have included the fix. The failure there was I said I would look properly if it failed twice rather than send you chasing it, so: I read your Closing and reopening both, which forces a new merge ref against current master. Nothing needed from you. |
Second instance of the same race, found by re-running the master pipeline for 859173f. That re-run failed on: ✗ setting persists across reload: multibyte toggle should restore checked=true from localStorage with none of the PR code that first surfaced it (Kpa-clawbot#1933) present, which settles that it is a master-level flake and not that PR's doing. The step waited for the element to exist and then read .checked on the next tick: await persistPage.waitForFunction(() => !!document.getElementById('liveMultibyteToggle')); const checked = await persistPage.evaluate(() => ...checked); The element existing does not mean the code that reads localStorage and applies it has run. Waiting on existence is a proxy for the thing under test — the same mistake Kpa-clawbot#1317 documented in test-channel-color-picker-e2e.js. Now waits for the toggle to exist AND be checked, with the same fall-through-to-the-assertion shape as the swatch fix in this PR, and the read reports "(toggle absent)" rather than throwing if the element is genuinely missing. No product code changed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wzwr3eXseyNM7Xj598djjE
|
Settled, and in your favour: your PR was not the cause. I re-ran the master pipeline for The same assertion that failed on your PR, on plain master. That is as clean a control as we are going to get. I said earlier that I had read your The cause is in the test: it waits for the toggle element to exist and then reads Your #1933 stands on its own merits and I still like it: replacing a source grep with a check on what the function renders is the right direction, and it is what would have spared us both the 312-versus-277-character detour this morning. |
…ers sit in map.js
V3.e, V3.f and V3.g all describe what `makeRepeaterLabelIcon` produces, but all
three asserted it by grepping `public/map.js`. V3.e went further and bounded the
distance between two identifiers to 200 characters. Three problems with that:
1. It fails on edits that change nothing. Any statement inserted in the
neighbourhood pushes the two identifiers apart and the build goes red while
the markup is byte-identical.
2. It cannot tell code from prose about code. A comment mentioning both
identifiers satisfies the grep, so the assertion reports green no matter what
the function does. Reproduced: with such a comment present, moving the hash
assignment away from the glyph still passed.
3. It does not assert the thing it is named after. On master the match is not
the declaration order at all -- it is `MB_GLYPHS[...]` reaching the later
`shortHash` inside `ariaStatus`, 212 characters downstream. The ordering the
assertion claims to check is incidental to whether it passes.
This loads map.js in the same DOM-less vm sandbox test-map-clustering.js already
uses, exposes `makeRepeaterLabelIcon` through the existing
`window.__meshcoreMapInternals` hook, and asserts the emitted markup:
* glyph, U+2009 thin space, hash, in that order and adjacent;
* no glyph and no thin space when there is no multi-byte status;
* aria-label exactly "multi-byte <status>, hash <ID>", and "repeater hash
<ID>" without one;
* the visible span carries aria-hidden.
Still no browser, so it stays in the JS-unit-tests CI step.
Mutation-tested rather than eyeballed. Old assertions vs new, same mutations:
mutation old V3.e/f/g new
glyph moved after the hash all silent caught
plain space instead of U+2009 all silent caught
span loses aria-hidden V3.g caught caught
aria-label loses its comma all silent caught
200 chars inserted between the
two identifiers (no behaviour
change) V3.e FAILS passes
The sandbox loader deliberately has no try/catch that warns and continues: if
map.js stops loading, the suite must fail rather than quietly skip every
assertion below it.
Full JS unit list from .github/workflows/deploy.yml: 65/65.
a3ef2bf to
281eac6
Compare
|
Merging. I reproduced both halves of the claim rather than reading the diff, because "this test is fragile" and "this test is worthless" are different verdicts and only the first one justifies the change. The false positive is real, and it is one assertion. On master I inserted a comment between
The failure is The teeth survive. The thing I actually wanted to know is whether asserting on rendered output loses coverage. It does not. On the merged tree:
So the new assertions catch more per real regression than the single grep did, and stop firing on edits that change no output. That is the whole trade and it lands on the right side of it. Against the master that now includes #1912, which touched the same test file: the merge is textually clean and the full file goes 44 passed, 0 failed. I ran it on the merged tree, not on your branch, since your branch predates that merge. Exposing Worth saying plainly: this PR exists because our review process on #1912 cost you a round trip over an assertion that was never testing the thing it claimed to test. Fixing the gate instead of routing around it was the right response, and the one that took more work. |
The audit table printed DECLARED, NOT OBSERVED and UNDECLARED OBSERVED side by side. The first two were never independent: notObserved is a strict subset of declaredRegions, so the page showed the same set twice, once whole and once filtered, and left the reader to diff them. Measured on a live 197-row response before changing anything: notObserved is a subset of declaredRegions : 197 of 197 rows, no exceptions all declared regions observed : 7 rows none observed : 78 rows mixed : 44 rows no declared regions at all : 68 rows The 44 mixed rows are the ones that cost the reader time. They declare 8.3 regions on average of which 6.5 are unobserved, so usually one or two are green. BE-TUR-REP1_ON3FNZ declares 20 regions and lists 19 under NOT OBSERVED; the only difference is that "be" is missing from the second list. Finding that today means comparing two lists of twenty. Now it is one green chip among nineteen red. UNDECLARED OBSERVED is dropped. A repeater does not forward a scope it has not been configured with, so the column cannot fill except transiently when an owner REMOVES scopes while older messages are still inside the window, which is an artefact of the window rather than a fault worth a column. Confirmed empty on all 197 rows at both 24h and 7d. Nothing is lost: the STATUS column already renders an "N undeclared" badge for that case, and undeclaredObserved still feeds statusScore and the search index, both untouched. Sorting keys off notObserved, which is what the page ranks by anyway. Declared order is preserved rather than regrouped by colour, so a region stays in the position an operator is used to scanning. Verification: 8 new cases in test-frontend-helpers.js asserting the RENDERED markup through a new window.__meshcoreScopeAuditInternals handle, following map.js (Kpa-clawbot#1356/Kpa-clawbot#1933) rather than grepping source. Suite 662 to 670 passed, 0 failed. Then every one of the 197 live rows was pushed through the real function: 7 all-green, 78 all-red, 44 mixed, 68 empty, reproducing the independently computed figures exactly, and 793 chips against 793 declared regions so no chip is invented or dropped. Colours use existing variables (--status-green, --status-green-text) alongside the --status-red the missing chip already used, so the two read as one scale. Dead code removed: scopeChips and undeclaredChips had no remaining callers. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PQS3XLoPD98yu9pxdRujqg
Follow-up to the review on #1912, where this assertion cost a round trip. Independent of that PR — this branch is off current
masterand touches no code path it changes.The problem
#1356 V3.e,V3.fandV3.gall describe whatmakeRepeaterLabelIconproduces, but all three assert it by greppingpublic/map.js. V3.e also bounds the distance between two identifiers:Three separate failure modes, all observed:
1. It fails on edits that change nothing. #1912 inserts one variable declaration in that function; the markup is byte-identical and the build went red.
2. It cannot tell code from prose about code. My first attempt at fixing #1912 added a comment explaining the constraint — and the comment mentioned both identifiers, so it satisfied the grep by itself. With that comment present I moved the hash assignment away from the glyph, reintroducing the exact defect, and the test still reported green. A check that a comment can satisfy is worse than one that is merely brittle.
3. It does not assert the thing it is named after. On
masterthe match is not the declaration order at all. It isMB_GLYPHS[...]reaching the latershortHashinsideariaStatus, 212 characters downstream. Whether the glyph is actually prepended to the hash is incidental to whether this passes.That third point also corrects something I said on #1912, and it corrects it against myself: both the 312 you quoted and the 299 I "corrected" it to are the distance between the two declarations, which is not the distance the regex uses. Measuring the one it does use:
MB_GLYPHS[→ nextshortHashmasterunknownWidthbelow the glyphshortHashbelow the glyphSo moving
unknownWidthdown does not merely fall short — it makes the gap worse, because it lands between the glyph andariaStatus. My earlier "233, still 33 over" was the wrong metric on the wrong pair. Apologies; the conclusion happened to hold but the reasoning did not.What this does
Loads
map.jsin the same DOM-lessvmsandboxtest-map-clustering.jsalready uses, exposesmakeRepeaterLabelIconthrough the existingwindow.__meshcoreMapInternalshook, and asserts the emitted markup:U+2009thin space, hash — in that order and adjacent;aria-labelexactlymulti-byte <status>, hash <ID>, andrepeater hash <ID>without one;aria-hidden.No browser, so it stays in the JS-unit-tests step rather than moving to Playwright.
Mutation-tested, not eyeballed
U+2009aria-hiddenaria-labelloses its commaFull JS unit list from
.github/workflows/deploy.yml: 65/65.Notes for review
try/catchthat warns and continues. Ifmap.jsstops loading, the suite must fail rather than silently skip every assertion below it.